spatialcomputing.info

While Meta’s Quest has always relied heavily on cameras for tracking location of the headset, controllers, and the world around the user, developers haven’t had the same privileged access to the headset’s cameras. Earlier this year Meta gave developers the ability to experiment with direct access to the headset’s cameras in private projects; starting this week developers can now publicly release apps that make use of the new feature.

This week’s update of the Passthrough Camera API for Quest means that developers can now publish apps to the Horizon store that directly access the front-facing cameras of Quest 3 and 3S. This opens the door to third-party applications which can scan the world around the user to understand more about it. For instance, developers could add computer-vision capabilities to track objects or people in the scene, or to build a map of the environment for analysis and interaction.

For a long time this was impossible due to limitations Meta placed on what developers could and couldn’t do with the headset’s hardware. Despite computer-vision capabilities being widely available to developers on smartphones, Meta was hesitant to allow the same on its headsets, apparently due to privacy concerns (and surely amplified by the many privacy controversies the company has faced in the past).

Previously, third-party apps could learn some information about the world around the user—like the shape of the room and objects within it—but this information was provided by the system in a way that prevented apps from directly seeing what the cameras could see. This made it possible for developers to build mixed reality applications that were, to some extent, aware of the space around the user. But it made some use-cases difficult or even impossible; for example, tracking a specific object held by the user.

Last year Meta announced it would finally unlock direct access to the headset’s cameras. In March, it began offering an experimental version of the capability to developers, allowing them to build apps that accessed the headset’s cameras. But they weren’t allowed to publish those apps to the public, until now.

The company has also specified the technical capabilities and performance of the cameras that the developers can access on Quest 3 and 3S:

• Image capture latency: 40-60ms
• GPU overhead: ~1-2% per streamed camera
• Memory overhead: ~45MB
• Data rate: 30Hz
• Max resolution: 1280×960
• Internal data format YUV420

Meta says that a developer’s use of camera data on Quest is covered under its Developer Data Use Policy, including a section on “Prohibited Uses of User Data,” which prohibits certain uses of data, including to “perform, facilitate, or provide tools for surveillance,” and “uniquely identifying a device or user, except as permitted [in the policy].”