spatialcomputing.info

A newly published patent application from Apple highlights the company’s internal exploration of “micro-gestures” for AR input, which involves using the thumb against the index finger as a sort of virtual joystick or selector. Another newly granted patent covers the use of head-worn haptics as a means of directing the user’s attention toward virtual objects which are out of sight.

Alongside a string of acquisitions and job hiring focused on XR, Apple has long been exploring the space through internal R&D. While much of that work is secret, the details that emerge through the company’s patents shed light on some of the technologies and concepts Apple has been exploring in VR and AR.

In a patent application filed September 2020 and published this month, Apple covers the use of “micro-gestures” as a means of controlling XR devices. One version of this micro-gesture input envisions the user using their thumb, pressed against the side of their index finger, as a sort of virtual joystick for making directional selections.

The patent application indicates that Apple thinks this approach would allow for a hands-free, highly motor efficient input which is also unobtrusive:

“Using the cameras to capture the micro-gestures to interact with the three-dimensional environment allow the user to freely move about the physical environment without be encumbered by physical input equipment, which allows the user to explore the three-dimensional environment more naturally and efficiently,” the application states. “In addition, micro-gestures are discrete and unobtrusive, and are suitable for interactions that may occur in public and/or require decorum.”

The patent application shows several examples beyond the ‘virtual thumbstick’, including gestures which treat each segment of each finger as its own ‘button’ for input (ie: pressing your thumb to the middle section of your ring finger).

The patent also throws some serious shade at existing input modalities for XR devices:

“For example, systems that provide insufficient feedback for performing actions associated with virtual objects, systems that require a series of inputs to achieve a desired outcome in an augmented reality environment, and systems in which manipulation of virtual objects are complex, tedious and error-prone, create a significant cognitive burden on a user, and detract from the experience with the virtual/augmented reality environment. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices.”

Interestingly, Facebook has shown essentially the same exact ‘virtual thumbstick’ as an input concept used with its prototype wrist-worn XR input device:

This particular patent application hasn’t yet been granted to Apple yet, but something tells me the two companies are likely to be butting heads in the near future over XR patents.

Another Apple patent, this one filed November 2018 and granted last month, envisions head-worn haptic devices which would be used to help direct the user’s attention to objects that are outside of their field of view.

The patent envisions three different levels of head-worn haptics, ranging from those implanted in earbuds or glasses all the way to a headband forming a complete circle of haptic modules.

The haptics would be used, in some way, to turn the user’s attention to one side or the other in order to draw their gaze toward something. Use-cases spelled out in the patent include helping the user locate an out of view speaker during a VR or AR teleconference, or directing the user’s attention to an important object inside of a virtual reality environment.

Exactly how haptics would be superior for guiding the user’s attention over simply using spatial audio isn’t entirely clear from the patent.

We’ve seen head-worn haptics experimented with previously, though for an entirely different use-case; researchers found they could significantly reduce discomfort related to smooth locomotion in VR with haptics mounted on a headset.