There is a hypothesis pertaining to robotics and computer generated, human-like characters called the Uncanny Valley. It states that as robots become more and more human-like, the comfort of humans interacting with them takes a sharp dip before it rises again towards fully human interaction.
In Fernando Pereira’s latest take on Powerset, he eludes to a similar nonmonotonic function for natural language search. Conversing efficiently in natural language requires that you can predict, to some degree of accuracy, what will be understood by your conversation partner. Systems that get uncannily close to human understanding yet fail in cases that are only predictable to the system designers may be frustrating to use. How will the tradeoff between increasing utility and increasing inscrutability pan out? Where is Powerset going to be on this hypothetical graph?
As Fernando says, only testing on the final system will tell.