User Experience Considerations for VR & AR Audio

Spatial Audio is a specific concept to VR and AR environments, because it places sounds in a specific location in the physical environment around you.

As an avid proponent for UX considerations in the VR world, I often come across articles or videos on necessary visual considerations for people within virtual reality. This is of course incredibly important to anyone looking into developing a VR product. However, because virtual reality is so holistic in it’s experience, many forget the importance of audio.  In an ever-growing combination of games and theatre, VR represents some of the most prominent sound technology of the modern day.

Spatial audio in particular is particular to 3D games, but is also a newly emerging field of VR. It places sounds in physical locations around you in order to represent the virtual environment you’re experiencing. For example, if a bird is on the right side of the screen, then a song or chirping can be heard predominantly in the right headphone and a soft reverberation in the left. This is more natural to a person’s actual experience in reality as sounds echo, grow, lessen, and reverberate depending on location and proximity to the user.

vr-visual-header

Image Source: http://vr.amp.amsterdam/what-is-spatial-audio/

So what considerations should UX Designers take when designing an audio/visual experience?  I asked my friend and co-founder of Noctvrnal, Anna Wozniewicz, who specializes in Immersive 360-Degree Sound Design. In my interview with her, she allowed me to understand the intricacies of spatial audio and how she creates worlds of sound.

What are the three most important things you consider when designing sound for an environment?

“Considerations include the environment itself (where are we?), what the significance of it is towards the story (how does being here tie into the story? how can we use the environment to further the story/plot?), and emotion (how does this place feel – to the audience? to the character?).”

How do you design for spacial audio? What is your process?

“Designing for spatial audio starts much the same as traditional 2D audio – you start with getting to know the story, characters, etc. and think about the considerations mentioned above. Then, you design the environment with those ideas in mind, and inevitably new ideas come up as you go. For 360 video, after creating a full stereo mix of the piece, it’s ready to spatialize, so we pan each sound individually around the space to make it as realistic as possible. Certain design elements that affect emotion (rather than directly tie into a visual cue on screen) are also spatialized accordingly to create a sense of uneasiness, or glee, or whatever tone the piece calls for.”

What considerations do you take into account from the environment?

“What kind of space we are in – is it reflective, like a cave or a hall, or is it open and not as reflective, like a field or a space that feels “cushioned?” Whether or not there are people or other living things around, both nearby or in the distance. How does the air sound – Windy? Dense? Eerie?”

Are there many differences between speech audio and atmoshperic audio? If so, what are they?

“The primary difference is that atmospheres are felt, whereas speech should be heard. If a piece has dialogue, we clean it up as best as possible to make it as clear and easy to hear as possible. Dialogue generally moves the story or plot forward. Environments serve more to convey the tone of a piece, and tend to get a lot less realistic, which also gives us a lot more room to play.”

Do you believe sound to be a good cue to the audience within VR? How have you observed users respond to audio cues? Do you believe certain scenarios lean towards users listening for cues?

“I think sound is even more important in VR than in traditional film or video games – even in a VR headset you only have a fraction of the 360 degree field of view at any time, but you’re still hearing the full “360 degrees” at any given moment, much like in real life. I don’t think nearly enough experiences are using audio cues to drive story yet – if it’s a fundamental part of developing the story within the experience, then you can direct viewers in a way that no other medium can. Integrating sound cues in a 360 degree environment separates VR  from other media, separate and unique from traditional film or gaming – it’s not really either of those things, so we should be thinking about how to use the unique aspects of VR, such as spatial sound, to enhance story. Perhaps games lend themselves better to utilizing sound cues, since they’re more interactive than a 360 video, and thus more reliant on the viewer making the correct decisions to move the story forward.
Users tend to respond to audio cues better if they are continuous. That is, if, say, a glass is dropped behind you in a VR experience and you only hear it break, then the sound is gone, it’ll be difficult for you as the viewer to locate where exactly that sound came from, thus making it a less effective cue. However, if you hear the glass drop and then you hear the sounds of shattered glass tinkling around in that same spot long after the initial break, then you have enough time to turn around, precisely locate the source of the sound, and focus your attention on that area (where theoretically the next part of the story would then take place).”

Are there specific atmospheric sounds that you have to include in an outdoor versus an indoor environment?

“I don’t think there’s anything you *have* to include – it’s more a matter of what’s going to sell to the viewer as being in that environment, whether it’s accurate to the real world or not. If it’s a forest, you’ll probably want some forest animals and winds and whatnot. If it’s a creepy demonic forest, you might have just enough forest sounds to inform the subconscious that you’re in a forest, but they might be dominated by the “demonic” sounds, maybe distorted animal or wind noises and other drone-y elements. If the viewer believes that they are where you’re telling them they are, then it doesn’t necessarily have to be “real”.”

What specific sounds do you believe lend most to a user’s immersion? Why?

“In VR, at bare minimum, you need sounds that match what’s being seen. If a person is walking around the viewer at a distance of two feet, but you can’t hear a single sound coming from them, whether that’s footsteps or even a cloth rustle, it’s going to be very disorienting, since they’re so close. These “realistic” sounds help ground is in the real elements of whatever world we’re seeing, since we’re used to hearing sounds accompanying pretty much everything in real life, too.”

Do you normally address direction or reverberation of a sound? Do you believe this to be important in a VR environment?

“Directionality and reverb are key to full immersion, with directionality being (in my opinion) more important. If something’s going on behind you, you want to hear that it’s coming from behind you, and hear it move as you turn your head to suddenly be coming from in front of you. Head-locked stereo audio isn’t nearly as immersive as spatial audio for that reason. Real life sounds don’t stay locked to your head as you turn to look at their source, so it’s a big turn off to immersion if your VR experience doesn’t have that element or realism. Reverb is important too (if you’re in a cave, it’ll sound different than an open field), but the sound has got to be coming from the right direction in the first place. Then, you can add the right kind of reverb, based on how it’d bounce off of walls or trees or whatever, to sound most accurate.”

How do you account for speech or sound in different kinds of spaces (outdoor versus indoor versus concert hall)?

“The software we use gives us options for pretty much anything we need, whether it’s a plugin for a DAW for 360 video or something that’s game engine based. There’s a lot you can change about the voice depending on how you manipulate the reverb, and then you can match the environmental sounds accordingly. Basically it’s just adding different types of reverb based on different room sizes and attenuation patterns.”

Is there anything you specifically utilize in the physics of audio to recreate it in a virtual world (not much of a science buff but I’d like to learn more about it)?

“If we want to get really realistic with the sound, then beyond placing it in the right direction and proper distance with the right reverb, there’s attenuation, which refers to how much the sound decays as it gets further away (as it would in real life). We can set the attenuation higher or lower depending on if the sound would be getting more or less absorbed by the environment.
There’s also the Doppler effect, which adds a bit of realism. If a car with a police siren is driving towards you, the siren has a specific frequency (pitch) it’s emitting. As the car drives toward you, the perceived frequency changes from higher to lower, creating that passing-car-siren effect that we’re used to hearing. And this applies to any sound as it’s passing by, not just car sirens! If you want to know more of the specifics, the Wikipedia page has a good explanation of the Doppler effect.”

Is there anything else that you believe to be important to your audio work?

“Directors/project leads that understand the importance of sound in VR storytelling, and aren’t limited to what picture alone can convey to viewers.”

 

Noctvrnal is an LLC co-founded by Anna Wozniewicz and Helena McGill, based in the Greater Los Angeles Area. They are bringing immersive, spatialized sound design to the forefront of the virtual and augmented reality movement. They provide complete post-production audio services for virtual, augmented, and mixed realities, including Foley and additional field recording.

They can be found at their twitter @Noctvrnal_VR, and their LinkedIn Noctvrnal LLC.

Posted in: UX & UI, V/A/M Reality

Tagged as:

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s