spatialcomputing.info

I’m still trying to wrap my mind around what it takes to build an AR app. RealityKit seems to be Apple’s new game/physics engine (replacing SceneKit) for AR experiences. Would one choose between Unity and RealityKit? How do you make that decision?

​

Also, if you’re feeling generous when responding, what is Metal and how does that fit into the picture? Thanks.

Hey!

Disclaimer, I have no previous experience with VR / AR technology. I do have experience in architecture, 3D modelling, rendering, photography, Rhino 3D, 3DMax etc. But I have never even tried on a VR headset once! (I’m ashamed to say)

I was inspired by Jon Rafman and his exhibit in Berlin near the Brandenburg Tor.

So I have a balcony at my apartment and have taken many photos and videos from here over the course of 5 years. The idea is to set up a camera / headset to achieve a live view from the balcony within the headset but to superimpose over this live view all of the content I have photographed from this point. The viewer can look around / off and scroll through the various content and then hide all other to see that frame of content in place with the live view (hopefully this makes sense).

I think I could manage setting up a very simple 3D representation of the balcony itself through modelling.. and could take a panoramic photograph from a fixed point which I would then map all my existing content to (using Photoshop image match / stitch ). This I can imagine how to do and the tools required. But I can imagine there is an additional program to bring this all together into an experience.

What would this program be called? And are there any simple online resources / tutorials you believe I could follow to achieve the desired effect?

i have some 5 spare months to work on something before starting my Tonmeister studies. would love some feedback for my multiplayer pokemon-go-y spatial audio AR app idea:

1. user chooses single or multiplayer mode depending on whether he’s alone or with friends
2. then chooses a scene that contains a set of pre-configured sounds
3. each user is assigned a unique colour/visual marker that represents and distinguish themselves in the scene
4. once user enters AR scene, he can start place sound markers that are associated with audio clips wherever they want in the environment based on what the sound they observe from other users
5. depending on where they place the markers, the audio clip will sound different
6. each marker animates to the dynamics of its audio track
7. users walk around and experience the immersive soundscape, the intensity of each audio clip will rise and fade as user walk closer to or away from the sound marker

so essentially an interactive sonic landscape in AR generated collectively by multiple users in real-time. each user gets to place audio tracks anywhere in AR space in reaction to other users’ actions. this collaborative effort allows the creators as well as listeners to experience an orchestra of sounds while nestled among individual sound sources coming from all directions in virtual space. basically i want it to be a group activity with individual’s decision influenced by one another’s that contribute to the whole sound sculpture.

tools: i was discouraged to use Swift as my tool for my original idea, so i decided to go with Unity which perhaps some of you guys could lead me to some tutorials of where to start. i would be very thankful! any good tutorials to start off to get going prototyping some simple interactions? also, any more beginner tutorials on manipulating sound in AR in Unity? any beginner tutorial on Unity’s DSP capabilities? open to feedback / ideas