spatialcomputing.info

Meta today announced plans to update Quest 3 with AI vision features similar to what’s on the company’s latest Meta Ray-Ban glasses. Meanwhile, Apple still hasn’t confirmed if Apple Intelligence features will hit Vision Pro at the same time as other Apple devices this Fall.

Meta has pointed to AI investments as its other major area of R&D alongside XR, and now the company is starting to bring the two together with consumer-facing AI features.

Sometime later this Summer in the US and Canada, Meta says it will roll out an update to Quest 3 enabling the ‘Meta AI with Vision’ feature. The feature will give the headset AI voice chat capabilities and also allow the headset to ‘see’ what’s in the user’s real-world field-of-view. Users can ask general questions but also inquire about things in front of them. Meta gives some examples in its announcement:

Let’s say you’re watching YouTube videos of some breathtaking hikes in mixed reality while packing for your upcoming trip to Joshua Tree. You can ask Meta AI for advice on how to best dress for the summer weather. Or you could hold up a pair of shorts and say, “Look and tell me what kind of top would complete this outfit.” You can get the forecast so you can prep for the weather ahead and even ask for local restaurant recommendations to indulge your inner foodie.

Or say you’re in-headset and listening to music while working on a paper for school on a massive virtual monitor. You could ask Meta AI to identify some of the most memorable quotes from Shakespeare’s Hamlet and explain the significance of the “to be or not to be” soliloquy or the play within a play.

You might be playing Assassin’s Creed® Nexus VR, parkouring across rooftops when your curiosity is piqued. Why not ask Meta AI whether or not there were actual assassins in colonial Boston? The answer may surprise you…

For the time being, the camera can only see what’s in the real world, but won’t have awareness of virtual content shown by the headset. Meta is alluding that the Meta AI with Vision feature may eventually include both real world and virtual world awareness.

Unfortunately Meta confirms the feature won’t be coming to Quest 2 or older devices (and it’s likely Quest Pro won’t get it either).

Meta didn’t talk about whether requests for this feature are being processed on-device or in the cloud, nor did it address things like encryption, though it did confirm the feature is based on Bing AI. The company hasn’t yet responded to our request for more info on privacy architecture.

While Meta is rapidly deploying AI capabilities to its devices, Apple has yet to confirm if its so-called ‘Apple Intelligence’ features are coming to Vision Pro.

Earlier this year Apple announced a range of Apple Intelligence features coming to iPhones, iPads, and Macs in beta this fall. Despite announcing visionOS 2 at the same time, no Apple Intelligence features have been confirmed for Vision Pro. That leaves it up in the air whether the headset will get any Apple Intelligence features at the same time as other Apple devices, or if users will need to wait until later versions of VisionOS.

Apple says many Apple Intelligence features are processed on-device, but some requests—including those which lean on ChatGPT—will request off-device processing. Apple claims off-device requests “never store your data,” “are used only for your requests,” and that Apple will make the behind-the-scenes code available for privacy auditing.