spatialcomputing.info

One of the biggest gripes developers have with creating mixed reality content for Quest has been the lack of access to the headset’s cameras. That’s about to change.

At Connect this week, Meta VP of VR/AR Mark Rabkin announced that developers will soon be able to access a new camera Passthrough API, which is slated to come “early next year.”

The release is expected to enable “all kinds of cutting-edge MR experiences,” Rabkin said on stage, which is set to include tracked objects, AI applications, “fancy” overlays, scene understanding and “so much more.”

Testing is said to arrive “very soon,” which could mean select developers will be able to opt-in to use a beta version of Passthrough API in their games.

As it is today, in lieu of raw camera access, developers rely on things such as geometric representations of the environment, body and hand-tracking data, and user-defined furniture placement.

Gaining greater access to cameras will allow developers more granularity in digesting the user’s individual space, something that so far has been lacking as studios have been forced to go for ‘low-hanging fruit’, such as optional MR modes to VR games which mainly rely on the wow-factor of seeing your own room. There are a few great examples though of studios working within these limitations to create clever room-scale MR games, such as Creature’s recently released mixed reality game Starship Home (2024).

It also means Meta will need to tighten down on camera privacy concerns, access has always been the sole responsibility of the company. Granted, Meta hasn’t disclosed specific limitations for its Passthrough API yet, or whether this includes raw access, so it remains to be seen what level of anonymity that additional camera data will provide.

Additionally, the company announced in a blog post this week it was updating Horizon OS alongside the launch of Quest 3S on October 15th to include “simpler, more transparent, and easier to use Privacy Settings,” such as activity status, privacy indicators when apps are using Quest’s microphone, location, and spatial data, as well as a new sensor lock feature, which cuts power to your headset’s external cameras and microphones “in certain situations”, like when the headset goes to sleep.