spatialcomputing.info

Apple is bringing a huge visual upgrade to its Persona avatars on Vision Pro in VisionOS 26. After seeing the new system first-hand, it’s hard not to be impressed. But a major question remains—how will Apple overcome the challenge of bringing this level of fidelity to smaller headsets with even less room for the cameras that are essential for this kind of tech?

Personas in VisionOS 26 Raise the Bar Higher Still

The existing Persona system on VisionOS 2 was already the most lifelike real-time virtual avatar system available on the market. But Apple is raising its own bar with a Persona update that’s coming to VisionOS 26. In fact, the company is so happy with the results that they’ll be removing the “beta” tag from the Persona feature.

At WWDC last week I got to try out the new Persona tech myself, and I have to say it looks as good as they showed off in their initial reveal footage.

Even though it uses the same capture procedure, the same cameras on the headset, and still processes everything on-device, the results are clearly improved. Skin looks much more detailed; I was particularly impressed with how it captured my stubble. Hair on the head is more detailed too.

But maybe even more than that, Apple’s Persona system captures the motion of the face with impressive detail. You can see me moving my face in uncommon and asymmetrical ways, but the results still look nuanced and realistic. It isn’t actually clear if the motion-mapping was updated with this new version of Personas, or if it simply looks more realistic because the underlying scan is now more detailed.

Apple also confirmed to Road to VR that these improvements carry over to the version of the Persona that’s used on the external ‘EyeSight’ display. And while brightness and resolution of the external display is largely the limiting factor right now, the Persona that’s displayed on the outside of the headset should look a little more detailed and realistic.

Overall, the sensation of Personas looking ‘ghostly’ is greatly reduced. However, hands still look ghostly (and maybe even more than they would otherwise, since there is now a greater contrast between the blurriness of the hands and the solidity of the face).

How Will This Scale to Smaller Headsets?

This is an obvious leap in the visual quality of Personas, but a big question that’s now on my mind is: how will Apple be able to maintain this quality bar on smaller headsets in the future?

It’s not just that a more compact headset will need to be more power-efficient in order to do the same amount of computing in a smaller package. Nor is it simply that a smaller headset means less room to fit cameras.

The key thing that makes Personas possible in the first place is that cameras on the headset have line-of-sight to the user’s mouth, cheeks, and eyes. This is the raw ‘ground truth’ view that needs to be interpreted to accurately figure out how to map the motion of the face onto the virtual avatar.

This isn’t too hard if you have a complete front-facing image of someone’s face. But it becomes more and more challenging as the angle of the view becomes more extreme. That’s why early face-tracking tech usually had a camera that hung way out in front of the user (so it could have a clear, undistorted view).

Even some modern face-tracking headset add-ons still hang the camera quite far away from the face for a clearer view.

If you want to make a smaller headset, the cameras end up moving closer to the face. This means the ‘ground truth’ data coming from the cameras is from an extremely sharp angle. The sharper that angle, the harder it is to map the motion to the user’s face.

But companies have gotten clever. For headsets like Quest Pro and Vision Pro, one option to tackle this ‘sharp angle’ ground truth issue is to train an algorithm by letting it see both a clear view of the user’s face and a sharp angle view of the face at the same time. This allows the algorithm to better predict how the clear view maps to the sharp angle view.

This kind of approach works for headsets like Quest Pro and Vision Pro, which still stick out far enough that downward facing cameras can see enough to do the job with some extra training.

But the future direction of headsets is pointing toward goggles-sized and even glasses-sized devices. We can already see this in PC VR headsets like Bigscreen Beyond, where it’s clear that even mounting a camera on the furthest edge of the headset wouldn’t make for a particularly clear view for the mouth. And as we go even smaller, the view will become completely occluded.

The one upside here is that eye-tracking alone is probably safe for a long time to come. Since XR is primarily mediated through the eyes, there will almost always be a good-enough angle for eye-tracking cameras to view the user’s eye movements.

But lifelike avatars are clearly something people want for communicating remotely in XR.  Making that happen will require full face-tracking, not just eye-tracking.

Continue on Page 2: One Possible Solution »