In Fernando Pereira’s latest take on Powerset, he eludes to a similar nonmonotonic function for natural language search. Conversing efficiently in natural language requires that you can predict, to some degree of accuracy, what will be understood by your conversation partner. Systems that get uncannily close to human understanding yet fail in cases that are only predictable to the system designers may be frustrating to use. How will the tradeoff between increasing utility and increasing inscrutability pan out? Where is Powerset going to be on this hypothetical graph?

As Fernando says, only testing on the final system will tell.

I had never shopped at CD Baby before. I had a fun time. Now they get a blog mention and a link out of it. Site designers take note.

If, as it seems, the iPhone’s software will all come from Apple then all the development fun will happen in the browser, in particular in whatever version of Safari the shiny little device ships with. And it will be fun if Apple give us, as developers, access the right information. I’m talking here about that information coming from the user’s fingers. After all, who wouldn’t want to create the kind of effortlessly intuitive interfaces the Jeff Han showed off in his TED talk?

Will they let us though? There are a few directions Apple could go here.

1. Be scrooges and filter out multiple touches before they get to us, only letting the user interact with web pages one finger at a time, the others controlling scrolling probably. Web Dev Chorus: Booooo!
2. Give us the raw events from all touches but leave it up to us to do the hard lifting of working out which mousedown/mousemove/mouseup events came from the same finger. Web Dev Chorus: Grumble, Grumble, even huger javascript libraries, Grumble
3. Generously preprocess the input data for us and label mouse* Events with unique “device” identifier. Web Dev Chorus: Yaaay!! Go Apple!

The iPhone definitely has javascript (and may have flash), but that’s not enough. Javascript’s current way of getting information from pointing devices like a mouse, what’s called the DOM API, has no way of distinguishing between clicks from multiple sources (nor does Flash for that matter). Neither DOM 3 nor Web Apps 1.0 has added any properties to the event object to deal with multiple pointing devices. At a minimum it would be nice to standardize on a new field of the event object that uniquely identifies the input device for the duration of a mousedown/mousemove/mouseup cycle.

I would imagine Apple will not be that pushed to let javascript know about multi-touch. It will only detract from their own installed apps and introduce potentially low standard interfaces that would nonetheless be associated with their brand. However the problem will certainly come to the fore as other companies follow Apple into the multi-touch field. We might as well think about what we’d like now.

Update: Dave Hyatt from the Safari team replies…

I’ll have much more to say about this in the coming months.

…which makes me cautiously optimistic. It must be pretty exciting to be him right now.

In fact Writely’s right-click is hilarious. If I click ‘copy’ or ‘cut’ or ‘paste’ in their menu i get an error popup saying “We don’t have access to your clipboard from javascript so you can’t use these commands from our very pretty right-click menu”. But if you hadn’t messed with my right-click menu in the first place, it would work just fine!

Discoverability

Novice users are surprised when you tell them you can right click things to get extra options.

Added complexity

After discovering that their interface to the computer has another variable, they can become confused as to which button they should use in which situation.

Too many options

Because GUIs usually have a lot of containment metaphors, right-clicking on a point means you could be directing your attention to any of several nested entities. As a result, context menus don’t have enough context and tend to be very long and hence difficult to find things in.

Discoverable, Entity Specific, Left-Click Context Menus

There is an alternative

• When the user makes some gesture towards an object such as hovering over it, a discreet target is displayed close outside the border of the object.
• If the user makes a gesture towards the target, hovering or clicking, it expands to show possible contextual actions for that object.
• Otherwise, if the user moves in a different direction, the target disappears.

IMHO, this goes some way to solving the problems above. Newbies can discover contextual functinoality easily. They don’t need to worry about an extra mouse button. And the menus can be entity-specific, so the functionality can be simplified. Plus, this interface is an evolution: it can be complimentary to the standard right-click menus.

Issues

Small Entities

Sometimes the user may want to direct attention to a small part of the interface, for example a character in a text editor or a pixel in a graphics package. These are valid conceptual entities and as such should have associated actions and options. However, it seems impossible to display a context menu target for every mouse over of such entities – The smaller the entity, the harder it is to avoid the menu target.

Overlapping Entities

Sometimes different conceptual entities take up exactly the same screen real estate. This increases the complexity of the contextual menu, but no more than that of a right-click menu.

Microsoft Office 2007

This pattern has popped up as one of the changes in the new MS Office interface overhaul. Here’s what happens when you select a piece of text. Look closely, there’s a hint of a contextual toolbar there:

As you move your mouse towards the faint target it gets solid and you can directly change the properties of the text:

If you move your mouse in the opposite direction it disappears and never comes back.

It seems the Office team have only used this on text selections and only for text formatting properties so far. It works in Excel and Power point as well. If you want to try out the new interface for yourself, Microsoft have set up vanilla installs that you can interact with through a website without installing anything. You have to have Internet Explorer on Windows, and be willing to install some ActiveX remote desktop plugin.

Elsewhere

Here’s another example from Flickr. Right-click menus are even less discoverable on the web, since not even experienced users expect them. Hence, Flickr does this when you mouse over a person:

Seen any other examples?

Status:

Looking for a guide …

Status:

Connected to guide: SomeGuide

SomeGuide:

Hi there. I will be helping with your search.

SomeGuide:

Hey, there. How are you today?

SomeGuide:

So you’re looking for the ‘answer to everything’?

You:

i am good

You:

and you?

You:

yes

SomeGuide:

I’m great. Thanks for asking

You:

SomeGuide, I am really curious what might be a good answer for everything

SomeGuide:

You mean in conversation?

SomeGuide:

Or ‘everything’ in an all-encompassing philosophical scope.

You:

yes, philosophical

You:

It sounds like a hard task, but I am looking for that to put into a slideshow for my next presentation

You:

as an opening quote

You:

it will be a technical talk about research opportunities

You:

for the future

SomeGuide:

Hmm, that’s quite a question…

…….

Go read the rest at Matthias’ excuse for a permalink. It has a happy ending!