spatialcomputing.info

The Marvel What If…? experience is set to launch tomorrow exclusively on Vision Pro. We got an early look at the experience and found a polished presentation that foreshadows the possibilities and pitfalls of immersive VR games on the headset.

Marvel’s What If…? on Vision Pro is being described as an “interactive Disney+ Original story.” And while it isn’t meant to be a full-blown VR game, it’s the headset’s biggest fully immersive production to date—most other native ‘games’ on the headset feel a lot like board games that float in front of you.

In What If…? players experience a narrative blended with casual gameplay driven by Vision Pro’s hand-tracking.

Made by Marvel and ILM Immersive, the production quality meets the high bar you’d expect from this kind of experience. Though scarcely an hour long, the experience is polished with strong sound, visuals, and voice acting throughout.

Fittingly for the emphasis on the Marvel multiverse, the narrative plays out across several distinct mediums. Players will see Marvel characters between their own living room (thanks to augmented reality), in fully immersive virtual spaces (thanks to virtual reality), and in 3D animated videos that play on floating screens in front of you.

The jump between these distinct contexts is actually handled very smoothly (aside from an awkward lack of environmental occlusion in AR), whether taking a portal from your living room into a fully immersive space, or seeing a 3D video playing back on a crystal shard floating in front of you (as a representation of some vignette from another timeline).

From a gameplay standpoint, What If…? shows us both the potential and likely pitfalls of fully immersive experiences on Vision Pro.

On the one hand (pun intended), the game makes use of hand-tracking to detect a range of gestures. You’ll use different gestures to summon an arm shield, grab distant objects with a force-like power, or shoot lasers from your hands. Some more complex gesture interactions are used effectively too, like a spell where you pinch your fingers together on both hands then pull them apart to draw a line, rotate your hands to twist the line, then finalize the spell by opening your hands and pushing them together, palm-to-palm.

But even though all of these gestures work reasonably well for this casual experience, they ultimately feel just like that—gestures: simple, prescribed hand motions. You’re never actually touching anything in the experience… just making a gesture and watching it cause things to happen.

I’ve said it before: the least interesting interactions in VR are those that happen far away from you. In VR, it’s not the distant bad guy falling over after you’ve shot them with an arrow that makes VR unique. What’s unique about VR is the fact that you can literally reach over your shoulder to pull out an arrow, nock it to your bow, stretch the string back, aim with one eye, and release—like you had an actual bow in your hands.

Now those reading along carefully should stop to think “isn’t the bow scenario you just described really just a series of ‘gestures’?” Sure. If you want to use the broadest definition of that word, you could dump it into that category. But it’s the fidelity of the gestures that we’re concerned about. It’s the difference between wiggling a Wiimote to make the character on screen swing their sword, versus actually swinging the sword yourself—dictating with precision exactly where it should land on the enemy.

There’s no doubt that Vision Pro as a headset is capable of doing the latter. But it has to be said… it’s damn tough without the precision of tracked controllers.

Falling back to hand gestures—and using those hand gestures to cause distant things to happen—isn’t highly engaging. Not being able to reach out and touch the virtual world around you—even to just pick up a rock and throw it with your own hand—diminishes much of the magic of VR.

And that’s where we find What If…?. It’s doing more than I even would have expected with Vision Pro’s hand-tracking. The gestures are varied and detected fairly consistently, but what you do with them feels kind of… detached. I never managed to cross the chasm from I’m ‘doing a hand gesture’ to ‘I’m casting a spell’… and further still from ‘I’m actually interacting with this world’.

Now to be fair, that’s partly due to the experience’s gameplay itself. It’s very casual by the standards of any ‘gamer’; easy enough that most people will be able to figure it out. That’s clearly on purpose; this is a supposed to be a narrative as much as an interactive experience. Marvel and ILM Immersive could have made the gameplay deeper and squeezed a bit more engagement out of the hand-tracking. It’s just hard to see it crossing that chasm I talked about above.

But let’s concede that this casual gameplay is ultimately in service of telling a story. Is the narrative any good? Well, my opinion will surely be colored by the fact that I’m not terribly attuned to the Marvel Cinematic Universe (even if I’ve caught the rough strokes over the years). Dropping me into a What If…? experience—that’s expressly about throwaway alternate timelines—certainly doesn’t lend itself to meaningful character development or real stakes. The narrative has the distinct feeling of being a framework for this experimentation in this new medium, rather than a story unto itself.

So in the end, is What If…? worth checking out on Vision Pro? Sure. I can’t say it was a terrible amount of fun, but hey, it’s free and well produced. And if not for the entertainment value, it’s a clear lesson on what can and probably can’t be done with the headset’s hand-tracking input. In the end this will go down as an early experiment, but certainly not a killer app.