spatialcomputing.info

Facebook announced today that an upcoming update to the Quest development SDK will include experimental support for a Passthrough API which will allow Unity developers to build AR experiences and features into apps on Quest 2.

Although Quest wasn’t launched as an AR headset initially, its impressive passthrough camera capability showed that it could handle AR functions surprisingly well. In fact, if you use the passthrough view as your default background, the Quest boundary and main menu floats against the backdrop of your playspace—effectively making it an AR experience that’s been available on the headset for some time.

Until now Facebook kept the tools for building AR apps on the headset to itself, but today it announced it’s unlocking the same capabilities for third-party developers too.

Initially rolling out as an “experimental” feature, the so called ‘Passthrough API’ will become available in the v31 update to the Quest development SDK. The Passthrough API is only available on Quest 2 for now; it isn’t clear if Facebook plans to extent it to the original Quest.

Facebook expects the new capabilities will enable a brand new class of applications on Quest 2 that take advantage of AR in interesting ways. Here’s some examples and explanations of the features developers can use:

Composition: You can composite Passthrough layers with other VR layers via existing blending techniques like hole punching and alpha blending.

Styling: You’ll be able to apply styles and tint to layers from a predefined list, including applying a color overlay to the feed, rendering edges, customizing opacity, and posterizing.

Custom Geometry: You can render Passthrough images to a custom mesh instead of relying on the default style mesh—for example, to project Passthrough on a planar surface.

Although transparent AR headsets—like HoloLens and Magic Leap—give a much clearer (and color) view of the real-world, passthrough AR headsets like Varjo XR-3 (and now Quest 2), tend to offer a much more immersive field-of-view and more convincing virtual imagery thanks to complete opacity control and the potential for perfect latency between  real world imagery and virtual imagery. Though Quest 2’s AR view is still fairly low resolution and black & white, it’s expected that future headsets from Facebook will focus on improving the passthrough AR view.

While developers will be able to start experimenting with AR on Quest 2 with the v31 SDK via Unity, support for other engines—and the ability to actually publish AR apps to users—is expected later this year.

With a device like Quest 2 loaded full of cameras, it’s definitely worth thinking about privacy, especially now that third-party apps can make use of AR capabilities. On that front, Facebook says that “apps that use Passthrough API cannot access, view, or store images or videos of your physical environment from the Oculus Quest 2 sensors,” and that “raw images from device sensors are processed on-device,” meaning that the images the camera sees don’t get sent to Facebook or to third-party developers.