What’s Holding Augmented Reality Back?

AR has been eagerly awaited for decades, but so far, it has failed to truly deliver

Augmented reality demo featured in M. W. Krueger’s book ‘Artificial Reality II,’ 1991.

What you’re looking at above is the state of augmented reality nearly two decades ago. It’s also a clue as to why, today, Magic Leap is reportedly looking for more venture funding after having already raised more than $2.6 billion from Google and other Silicon Valley giants but has little to show for it beyond an expensive AR headset that’s rumored to have unimpressive sales. It’s also a cautionary case study for Apple, which is reportedly planning a launch of its own AR headset line in 2022.

This ’90s demo has a strikingly similar interaction model to Magic Leap’s user interface. But while visually compelling, this approach to interaction will always suffer from the problems associated with ambiguous input from hand gestures. It’s a key challenge that’s continually held AR back as a technology.

The author presenting at Innovem Fest 19 in Spain.

The unacknowledged complexity of gesture control

AR advocates often assume gesture control will be the next iteration in user interfaces since it seems so intuitive and natural to human expression. (And as discussed in my last Modus essay, it has the seeming inevitability of sci-fi.) But hand-gesture models and libraries are not uniform. The ambiguous input produced by humans forces the computer to process far more information than a controller with comparatively limited function, like a touchscreen. This makes the interaction prone to a wide range of errors. The user’s background could be too dark, they could be wearing gloves, or they could have hands smaller than those that the device was tested with. This interaction model also likely requires having to train someone to use gestures they’re not yet familiar with, and not everyone will make the gestures in the same way.

By contrast, physical buttons are incredibly practical. A computer can always interpret the push of a button as a one-to-one interaction. Button-based interfaces are usually colorful and in the right places for your hands. You can quickly pick up the muscle…