Adobe’s first big Apollo app is an Internet video application codenamed Philo. The pervasiveness of Flash Player created multi-billion dollar Internet video startups powered by the Flash video format. The Philo team hopes to expand the display size and quality of distributed videos and get publishers encoding using the latest Flash video encoders. Publishers can brand the entire video player, delivering MTV content in what looks like a MTV video player, or a Rocketboom-themed player…

Great!! thats just what I want – an MTV branded video player to play MTV videos and a Rocketboom branded video player to play Rocketboom videos. Awesome!

This is not the future of the web people, and it certainly isn’t the future of the desktop. Please try to resist.

If, as it seems, the iPhone’s software will all come from Apple then all the development fun will happen in the browser, in particular in whatever version of Safari the shiny little device ships with. And it will be fun if Apple give us, as developers, access the right information. I’m talking here about that information coming from the user’s fingers. After all, who wouldn’t want to create the kind of effortlessly intuitive interfaces the Jeff Han showed off in his TED talk?

Will they let us though? There are a few directions Apple could go here.

1. Be scrooges and filter out multiple touches before they get to us, only letting the user interact with web pages one finger at a time, the others controlling scrolling probably. Web Dev Chorus: Booooo!
2. Give us the raw events from all touches but leave it up to us to do the hard lifting of working out which mousedown/mousemove/mouseup events came from the same finger. Web Dev Chorus: Grumble, Grumble, even huger javascript libraries, Grumble
3. Generously preprocess the input data for us and label mouse* Events with unique “device” identifier. Web Dev Chorus: Yaaay!! Go Apple!

The iPhone definitely has javascript (and may have flash), but that’s not enough. Javascript’s current way of getting information from pointing devices like a mouse, what’s called the DOM API, has no way of distinguishing between clicks from multiple sources (nor does Flash for that matter). Neither DOM 3 nor Web Apps 1.0 has added any properties to the event object to deal with multiple pointing devices. At a minimum it would be nice to standardize on a new field of the event object that uniquely identifies the input device for the duration of a mousedown/mousemove/mouseup cycle.

I would imagine Apple will not be that pushed to let javascript know about multi-touch. It will only detract from their own installed apps and introduce potentially low standard interfaces that would nonetheless be associated with their brand. However the problem will certainly come to the fore as other companies follow Apple into the multi-touch field. We might as well think about what we’d like now.

Update: Dave Hyatt from the Safari team replies…

I’ll have much more to say about this in the coming months.

…which makes me cautiously optimistic. It must be pretty exciting to be him right now.

In fact Writely’s right-click is hilarious. If I click ‘copy’ or ‘cut’ or ‘paste’ in their menu i get an error popup saying “We don’t have access to your clipboard from javascript so you can’t use these commands from our very pretty right-click menu”. But if you hadn’t messed with my right-click menu in the first place, it would work just fine!

Discoverability

Novice users are surprised when you tell them you can right click things to get extra options.

Added complexity

After discovering that their interface to the computer has another variable, they can become confused as to which button they should use in which situation.

Too many options

Because GUIs usually have a lot of containment metaphors, right-clicking on a point means you could be directing your attention to any of several nested entities. As a result, context menus don’t have enough context and tend to be very long and hence difficult to find things in.

Discoverable, Entity Specific, Left-Click Context Menus

There is an alternative

• When the user makes some gesture towards an object such as hovering over it, a discreet target is displayed close outside the border of the object.
• If the user makes a gesture towards the target, hovering or clicking, it expands to show possible contextual actions for that object.
• Otherwise, if the user moves in a different direction, the target disappears.

IMHO, this goes some way to solving the problems above. Newbies can discover contextual functinoality easily. They don’t need to worry about an extra mouse button. And the menus can be entity-specific, so the functionality can be simplified. Plus, this interface is an evolution: it can be complimentary to the standard right-click menus.

Issues

Small Entities

Sometimes the user may want to direct attention to a small part of the interface, for example a character in a text editor or a pixel in a graphics package. These are valid conceptual entities and as such should have associated actions and options. However, it seems impossible to display a context menu target for every mouse over of such entities – The smaller the entity, the harder it is to avoid the menu target.

Overlapping Entities

Sometimes different conceptual entities take up exactly the same screen real estate. This increases the complexity of the contextual menu, but no more than that of a right-click menu.

Microsoft Office 2007

This pattern has popped up as one of the changes in the new MS Office interface overhaul. Here’s what happens when you select a piece of text. Look closely, there’s a hint of a contextual toolbar there:

As you move your mouse towards the faint target it gets solid and you can directly change the properties of the text:

If you move your mouse in the opposite direction it disappears and never comes back.

It seems the Office team have only used this on text selections and only for text formatting properties so far. It works in Excel and Power point as well. If you want to try out the new interface for yourself, Microsoft have set up vanilla installs that you can interact with through a website without installing anything. You have to have Internet Explorer on Windows, and be willing to install some ActiveX remote desktop plugin.

Elsewhere

Here’s another example from Flickr. Right-click menus are even less discoverable on the web, since not even experienced users expect them. Hence, Flickr does this when you mouse over a person:

Seen any other examples?