How the creator of the MP3 is building the future of VR sound

samsung vr million cingo gear
LG and Samsung don’t agree on much. Whether its TVs and smartphones, or washers and dryers, each of the warring Korean tech giants is constantly looking for an edge over its rival. But one thing they do agree on is a little-known German research firm called Fraunhofer IIS. Fraunhofer is responsible for creating 3D surround sound audio tech for both Samsung’s Gear VR (which won our product of the year for 2015), and LG’s own new mobile VR solution, the 360 VR, which made its debut at MWC 2016.

It’s actually not all that surprising that Samsung and LG both trust Fraunhofer to help build the aural foundation of their new VR creations. Fraunhofer’s audio wing has created audio codecs licensed by over 1,000 companies in eight billion devices after creating the MP3, and co-creating the AAC codec family. Virtually every music and video streaming company in the business, from YouTube to Spotify, uses the company’s stuff. And with VR on the horizon, Fraunhofer wants to dominate that, too.

We sat down with Fraunhofer at MWC to find out more about its plans to help shape the impending VR revolution, which will rely heavily on realistic sights and sounds to take flight.

“We were the first ones to do Internet audio, and we’re also now the first ones to do 3D sound for VR,” says Head of Marketing for Fraunhofer’s Audio and Multimedia division, Matthias Rose. “We keep on with our tradition of pioneering new markets for sound now in this new field of virtual reality.”

Whatever your thoughts about compressed audio, it transformed the way we hear the world of entertainment forever. Without it, there never would have been an iPod, much less Spotify, YouTube or Netflix. Now, Fraunhofer is hoping to do the same with VR, thanks in part, to two new highly mobile file codecs: HE-AAC, and what the company calls the “next generation” codec, MPEG-H.

But before we get into that, a bit on the emergence of 3D surround sound and its place in the VR landscape.

Making your ears hear in 3D

If you follow surround sound even a little, you’ve probably heard of Dolby Atmos. Dolby (along with and its equally-invested rival, DTS) has been a leader in creating a new way to mix and master audio tracks, based around what the pros call “sound objects.” While previous surround sound mixing was limited by recording and mixing audio into specific channels, the sound-object method allows for individual objects on screen — like a helicopter, or a buzzing bee — to move freely throughout sonic space, essentially constrained only by the imagination of the audio engineer.

Acoustic-Lab-Fraunhofer-IIS

As anyone who’s been to a Dolby Atmos-powered movie theater can attest, the effect can be dazzling. Rain drops seem to come from on-high thanks to speakers mounted on the ceiling, laser beams and rockets strafe from front to back through the theater, and so on. The more creative the audio engineers get, the more realistic the cinematic experience becomes.

As an interactive medium that tricks us into believing we’re in another world, VR arguably offers an even more impactful and visceral medium for sound objects to play in. Moreover, Fraunhofer says realistic surround sound is more important in VR than cinema because humans rely heavily on sound cues to inform us of our environment. For instance, the reflections of sounds in a specific space tell our brains if we’re in a tiny room, or a big echoey cave. Movements of objects in VR environments — such as a character running down a hallway, or a car driving down the road — need to sound believable in space — otherwise, the illusion of VR quickly falls apart.

“We were the first ones to do Internet audio, and we’re also now the first ones to do 3D sound for VR.”

That’s especially important in the nascent world of VR cinema, where the sound objects themselves can help dictate the plot.

“When you make cinematic content where the user can look anywhere he wants, you can use audio to help tell the story, making sure that the audience actually follows (the director’s) path of action and (doesn’t) wander off somewhere else,”  explains Jan Nordmann, Fraunhoffer Senior Director of Business Development for New Media. “So, if you have an explosion happening behind you, you will most likely look in that direction. The director can pick it up from there and give additional audio cues to make sure you’re following along.”

Along with sound objects, Fraunhofer’s audio tech also, of course, incorporates head tracking cues: No matter where you move your head, the sound stays in the proper place within the game or movie.

VR for all

Dolby, DTS, and others also want to sculpt the sound that makes VR seem real. In fact, Dolby showed us some of its applications in that department at MWC as well, powering the VR film, Collision, on a Samsung Gear VR.

However, unlike most object-based surround systems, Fraunhofer’s entire infrastructure — from its mixing and mastering plug-ins, to its MPEG-H and AAC-HE files — is totally open for anyone to use. Partners for its tech include a vast array of developers, from device makers, to broadcasters and content creators, to streaming services. That may well give Fraunhofer a leg up on competing proprietary systems.

“That’s one of the main differences between us as an audio company like Dolby (or) DTS: We make open standards together with partners, and they make proprietary products,” says Nordmann.

Acorn to oak

Fraunhofer’s system begins with the company’s 3D surround sound plug-in, which works with virtually any digital audio recording software, from ProTools to Nuendo. With the company’s intuitive plug-in, it’s easy to move sound objects around in a virtual 3D environment — with very little processing power, according to Fraunhofer. In a quick demo, I was able to easily move around a sound object — in this case, the voice of a hero in a VR film — and listen as the sound appeared to move closer and further away, as well as all around my head, in real time. And that’s just one piece of Fraunhofer’s VR audio architecture.

Cingo-VR

At its advanced audio lab (which the company claims is one of the three most sophisticated labs of its kind in the world), Fraunhofer is also developing hardware like 3D microphones to better capture 3D sound in a live environment. As such, the company has its hands on the 3D sound process from start to finish, beginning with recording, and ending with rendering the sound exactly as you hear it in the studio via the compressed MPEG-H and AAC-HE file formats.

Going mobile

Standardization is all well in good, but how do you create believable 3D sound for mobile devices and plain old stereo headphones?

“Fraunhofer audio powers, not all, but most of the mobile VR experiences that are on the market.”

“We are really a mobile audio company because of our AAC and MP3 developments,” says Nordmann. And that extends far beyond stereo sound. As users increasingly began taking their video with them in the streaming age via smartphones and tablets, they still wanted a full surround sound experience. That doesn’t work very well over those tiny speakers in your mobile device, so in 2013, Fraunhofer created its own infrastructure for virtual surround sound over headphones.

“We developed a product called Fraunhofer Cingo, which is basically a post-processing library that allows you to have virtual surround sound over regular stereo headphones. And that’s what Google incorporated into Google Nexus devices, in addition to the Android operating system,” Norman says. “That allowed Google Play to start selling movies with 5.1 surround sound encoding with our HE-AAC codec, and then rendered for headphones with Cingo. And that has been in every Nexus device since 2013.”

“At that point, that was a surround rendering technology for headphones, but we kept developing it, and now we have a full 3D rendering engine,” Norman explains. “And that is what Samsung licensed from us for the Gear VR.”

LG’s 360 VR is using it too, along with Google Cardboard devices (via Android), and others. “Fraunhofer audio powers, not all, but most of the mobile VR experiences that are on the market,” Nordmann says. The system is also native to iOS devices for deployment if and when Apple gets into the VR game.

Fraunhofer’s goal is to allow audio pros to easily render high-quality 3D surround sound files in an open-source environment which, in turn, may help allow the brilliance of the best and brightest innovators in VR to shine. Thanks to Fraunhofer, and its partners, the future of audio for mobile VR is already here.

Now all we have to do is wait for the new generation of virtual reality content creators, device makers, and other innovators to take it and run.

Cars

How Grammy-winner Elliot Scheiner helps Acura make one of the best in-car sound systems

We sit down with eight-time Grammy-winning recording studio legend, Elliot Scheiner, to learn a how he works with Acura to make one of the best in-car sound systems. The latest version launched with the all-new 2019 Acura RDX.
Home Theater

Still listening on tinny TV speakers? Try one of our favorite soundbars

You no longer have to sacrifice sound for size when selecting home audio equipment. Check out our picks for the best soundbars, whether you're looking for budget options, pure power, or tons of features.
Home Theater

The best noise-canceling headphones paint your music on a cleaner canvas

Drowning out the sound of babies, jet engines, and the outside world isn't as hard as it seems. Here are the best noise-canceling headphones, whether you're concerned with style, comfort, or sound.
Home Theater

From game consoles to high-grade spinners, these are the best Blu-ray players

Streaming may be popular, but the disc isn't dead yet! To get the very best picture and sound quality from your system, you need to be watching Blu-ray or 4K Ultra HD Blu-ray discs. Here are the best Blu-ray players you can buy right now.
Computing

Reality shines into your ‘Minecraft’ escape via Windows Mixed Reality Flashlight

What’s great about VR is that you can get the full building experience in Minecraft. The drawback is that you don’t have visual access to the real world without lifting or removing the headset. Microsoft has a fix for Windows.
Photography

Adobe’s next big software secret? An augmented reality tool called Project Aero

Itching to create an augmented reality object? Adobe will soon have software to help. Adobe recently teased Project Aero, an upcoming software that helps design AR objects, including support for Apple's ARKit.
Virtual Reality

HTC says ‘it takes time to launch a new technology,’ claims lead in VR revenue

HTC posted a response to a Digital Trends editorial charting VR headset sales on Amazon. The company said "it takes time to launch a new technology," and posted data showing it makes the most revenue among its peers.
Virtual Reality

Daydream VR users can browse with Google Chrome in virtual space

Tired of browsing the internet in boring regular reality? If you're a Daydream VR user, you no longer have to, as Google has just announced the ability to use Chrome from within your Daydream VR headset.
Computing

Magic Leap’s augmented reality interface floats wherever you want it

New screenshots of a potential user interface for the Magic Leap augmented reality headset give us a firmer grasp of what the AR world inside the goggles might actually look like during day-to-day use.
Mobile

Google Home too boring? You want Gatebox’s cute virtual character in your life

Want to share your life with a virtual character? Then your dream is closer to reality thanks to Gatebox. It will control your smart home, and the character that lives inside the Gatebox can recognize and interact with you.
Deals

The Google Daydream View, one of the best VR headsets, is now on sale for $30

If you own a compatible smartphone, the Google Daydream View is one of the best vVR headsets money can buy. If you've been looking to dip your toes into the world of VR, Verizon has slashed the price of the headset down to just $30.
Emerging Tech

IrisVision uses VR to help people with fading eyesight to see again

A smartphone-based VR system can now help people with severe macular degeneration see better. While it doesn’t cure blindness, it does enable users with this vision disorder to carry out tasks they would otherwise find impossible.
Mobile

Facebook Messenger now includes AR games you can play with your friends

If sending messages wasn't already enough with Facebook Messenger, users can now challenge friends to augmented reality games. With the new feature, you can play with up to six people at a time from anywhere in the world.
Photography

Google enables virtual tour creation via Street View or a smartphone

Google wants anyone to be able to create a virtual tour -- Google Tour Creator is an online program for creating virtual tours with multiple points of interest. The program is designed to make advanced tours easy to create.