spatialcomputing.info

If you watched the WWDC  2024 keynote, you’ll have seen the announcement of VisionOS 2 and some highlights of what’s coming to the Vision Pro headset. But there’s lots more that Apple didn’t actually show on stage. Here’s everything we know so far.

Beyond what Apple showed of VisionOS 2 in the WWDC keynote, there’s a lot more coming when the update launches this Fall.

We’ve scoured everything Apple has released about the forthcoming Vision Pro update to get a comprehensive picture of everything that’s coming. Let’s jump right in.

Photos App Upgrades

The Photos app is getting additional enhancements in VisionOS 2. Although the iPhone can already capture spatial photos and videos, you need to switch to a special mode during capture. The Photos app will be able to take existing photos and convert them into 3D photos to view on Vision Pro. As far as we know, this will not be an option for videos.

We’ve seen similar 2D to 3D conversion tech from other companies in the past, but with mixed results. It will be interesting to see how well Apple’s approach works.

The Photos app is also getting proper SharePlay support, including Spatial Personas. While it was possible previously to stream the Photos app as a flat window with friends over FaceTime, now the app will support sharing full spatial content (photos and videos) and panoramas with other Vision Pro users.

Videos in the Photos app can also now be trimmed just like on iPhone and iPad.

Mac Virtual Display Upgrades

At launch, one of Vision Pro’s most useful features was Mac Virtual Display, which made it effortless to stream your Mac screen right into the headset. And while you could get a nice big monitor, it was stuck in one aspect ratio.

While many asked for support for multiple virtual monitors, in VisionOS 2 Apple opted to add new ‘wide’ and ‘ultrawide’ monitor options instead.

From a screen real estate standpoint, Apple says, “it’s the equivalent of having two 4K displays sitting side by side,” about the ultrawide mode.

New Gestures for Easier Navigation

With how often you open the app menu and Control Center while using Vision Pro, we were surprised how clunky these actions were on the headset.

Apple heard that feedback loud and clear, and has added two new gestures to make this easier in VisionOS 2.

To open the home screen users can turn their hand palm-up and pinch their fingers. A palm-down pinch will open the clock and make it easy to get to notifications and the Control Center where many helpful tools (like screen casting and recording) are found.

Customizable App Menu

A simple feature missing at launch; VisionOS 2 will allow users to rearrange their app menu, just like they would expect on iPhone and iPad.

You’ll also be able to pull apps out of the ‘Compatible Apps’ folder to put them in the main app menu for easier access.

Immersive Background Improvements

VisionOS shipped with a handful of beautiful immersive background environments, but also several tantalizing “coming soon” bubbles that we’ve been waiting patiently for.

VisionOS 2 will bring a new Bora Bora environment, taking users to an island in the South Pacific.

‘Cinema mode’ puts movies on Vision Pro onto a massive screen that naturally merges with the selected immersive background. But at launch this was restricted to apps like Apple TV.

With VisionOS 2, web videos in Safari will have the option of being presented in the same way. Apple says this will work for YouTube, Netflix, and Amazon videos out of the gate, and presumably web developers will be able to enable this functionality in other players too.

Users will also now be able to tilt videos upwards, while in cinema mode, to make reclined viewing more comfortable.

Keyboard Visible Through Immersive Backgrounds

Speaking of Immersive Backgrounds, at launch Vision Pro had excellent hand occlusions which brings your own hands into immersive environments. Unfortunately if you were using a Magic Keyboard, they keyboard itself would disappear when in an immersive environment, even though your hands would show on top of where it should be.

VisionOS 2 makes the natural improvement of showing the full keyboard even while an immersive background is visible.

Support for Third-party Mice

At launch Vision Pro could only be paired with Apple’s Magic Trackpad. But regular mice weren’t supported (not even the company’s own Magic Mouse).

VisionOS 2 will add support for the Magic Mouse and third-party mice too. Presumably this will cover third-party mice and trackpads.

Guest User Enhancements

It’s always fun to show people Vision Pro for the first time. And the built-in Guest User feature makes this easy. Unfortunately it takes a minute or two for the Guest User to calibrate every time they don the headset.

In VisionOS 2, Guest User can remember the last guest’s calibration data for up to 30 days. So if they need to take a break or you want to grab the headset for a minute to navigate them to a new experience, they can put it back on and skip the calibration step.

Watch Multiple Sports Games in Apple TV

In VisionOS 2, Apple TV will allow users to watch up to five different games at the same time—between MLS and MLB.

Persona Enhancements

Apples Persona avatars have been steadily getting better since the launch of Vision Pro, and VisionOS 2 promises more improvements still.

Apple says users can expect “more accurate skin tones and vibrant clothing colors.” It also promises more natural hand movements.

And if you’re using your Persona as a ‘webcam’ (ie: a windowed view, not a Spatial Persona) you’ll no longer be stuck using the default background that every Vision Pro user is stuck with. With VisionOS 2 Apple says you’ll be able to choose “a variety of backgrounds,” hopefully including choosing your own photos.

Look-to-Dictate in Messages

Look-to-Dictate is a convenient part of Safari, where you just look at the microphone icon in the URL bar, and it prompts you for voice input (skipping the keyboard).

VisionOS 2 will bring the same feature to Messages to make it easier to send texts hands free.

AirPlay to Vision Pro

It’s always been possible to cast content from Vision Pro, but not to cast content to Vision Pro from other AirPlay devices.

VisionOS 2 will make this small but obvious improvement, meaning you’ll be able to cast from your iPhone, Mac, and other AirPlay compatible devices, and see the video pop up in your headset.

Mindfulness App Breathing Detection

Mindfulness is Apple’s first-party meditation app for Vision Pro. In VisionOS 2 the app will now be able to detect your breathing rate and use it to match the app’s visualizations. This is presumably done using the headset’s microphones.

Train Support for Travel Mode

Travel Mode in VisionOS 2 will explicitly support use in trains. The mode changes the headset’s tracking behavior to ensure it tracking works accurately even while being in a moving vehicle.

Panorama Viewing on the Web

In the Photos app, panoramic photos can be stretched around you to experience them in an immersive way. That same feature will now work with in Safari for panoramic photos hosted on the web. We don’t know at this point if the headset will automatically detect panoramic photos or if webpages will need to declare them as such in order to use this feature.

Apple Music Karaoke

The Apple Music app in VisionOS 2 will get support for collaborative playlists between friends, and Apple Music’s ‘Sing’ feature which works like karaoke. It sounds as if friends will even be able to use Sing together through FaceTime.

Live Captions

The Live Captions feature is also coming to VisionOS 2. This acts like ‘real time subtitles’ by showing in the headset a written version of the words being spoken around you, or being played through the headset in audio or videos. Primarily intended as an accessibility feature for those with hearing disabilities, the feature may one day support real-time translation too.

Improved Quick Look

Quick Look is the easiest way to show realistic 3D models in augmented reality at scale—even directly from a web page. VisionOS 2 will bring improvements to Quick Look, like being able to anchor objects to surfaces and change colors and materials without loading additional models. Apple envisions this as being especially useful for previewing how a piece of furniture might fit in your room.

Apps Will Remember Their Position Between Reboots

It’s impressive that you can ‘leave’ and app sitting on one side of your house, walk away, then come back and find it still hanging out right there. But unfortunately if you power down your headset the app will forget its positions.

In VisionOS 2, apps can remember the position even between reboots. However, our understanding is that recenting the headset will still recall all apps to appear in front of the user, ruining any large-scale app arrangements you may have set up.

Communication Safety in VisionOS Apps

VisionOS 2 will bring Apple’s Communication Safety feature to the headset. The feature, intended primarily for children,” gives you the option to blur sensitive photos and videos you receive before you choose whether to view them.”

---

There’s a bunch of other changes and improvements coming for developers too, stay tuned for more coverage on that front.