spatialcomputing.info

Varjo’s XR-3 headset has perhaps the best passthrough view of any MR headset on the market thanks to color cameras that offer a fairly high resolution and a wide field-of-view. But rather than just using the passthrough view for AR (bringing virtual objects into the real world) Varjo has developed a new tool to do the reverse (bringing real objects into the virtual world).

At AWE 2021 this week I got my first glimpse at ‘Varjo Lab Tools’, a soon-to-be released software suite that will work with the company’s XR-3 mixed reality headset. The tool allows users to trace arbitrary shapes that then become windows into the real world, while the rest of the view remains virtual.

This makes it possible to bring parts of the real world into virtual reality. For instance, as the company shows, you can trace the outline of a physical steering wheel peripheral so that you can see the real wheel inside of the virtual world.

The Varjo Lab Tools software supports three core masking modes: static, depth, and marker-based. With the static mode you can draw a passthrough window that will essentially be tracked to your head. With the depth mode you can have the system automatically pull in anything that’s within a certain distant to the headset (judged by the lidar sensor). You can imagine this being used to automatically show your real hands holding something when you raise them up in front of you, without pulling in the background of your room.

In the marker-based mode you can create a window that’s tracked to a simple fiducial marker and moves with the marker. This would be great for something like the steering wheel example above, or allowing your real keyboard to persistently appear in the virtual world. The markers can be easily printed and mounted anywhere… and you can track up to 1,000 of them.

At AWE 2021 I got to try the system for myself, and I was impressed with the user-friendliness of the tool. To draw a marker-tracked mask, for instance, you just use a VR controller to select the marker you want to track, and then trace out the mask with a simple point-and-click system. For example, I used the tool to mask out an area around a gamepad that had one of the markers attached to it. Then when I jumped into VR I could see a window around the gamepad, allowing me to look down and see my real hands on the gamepad and easily reference the position of my fingers with regard to the buttons.

And what’s particularly cool is that all of this passthrough magic is happening completely independent of the VR application, which means the passthrough windows that you draw can function inside of any app without a special integration by the developer.

It’s a very cool system that makes passthrough fundamentally more useful, but it isn’t yet flawless. One of the most obvious limitations right now is the update rate of the masked area. While the view through the passthrough window (ie: the view of the real world) updates at a perfectly fast rate, the shape of the window itself (when attached to a marker) only seems to update a few times per second. So if you’re moving the marker even slightly fast you’ll see the passthrough window lag behind as it tries to keep up.

I’m not sure what the limiting factor is on the update rate for the passthrough window, but I take it this is something that will be improved in the future. A much faster update rate would make real-world objects appear much more seamlessly within the virtual world.

– – — – –

Varjo says that the passthrough masking capabilities can also be used with the chroma key functionality it has introduced previously, making the XR-3’s passthrough system very functional and flexible compared to anything else on the market today. Indeed, just as the impressive resolution of Varjo’s flagship headsets has long felt like a glimpse of the eventual future of consumer VR headsets, we’d also hope to see this kind of advanced passthrough functionality come to consumer headsets, eventually.

For now the capability is only on Varjo’s high-end enterprise XR-3 headset (as it’s the only one with all the passthrough hardware necessary to make it all happen). The company says the Varjo Lab Tools software, which will enable these features, will launch alongside the next major update to its core ‘Varjo Base’ software.