How the creator of the MP3 is building the future of VR sound

samsung vr million cingo gear
LG and Samsung don’t agree on much. Whether its TVs and smartphones, or washers and dryers, each of the warring Korean tech giants is constantly looking for an edge over its rival. But one thing they do agree on is a little-known German research firm called Fraunhofer IIS. Fraunhofer is responsible for creating 3D surround sound audio tech for both Samsung’s Gear VR (which won our product of the year for 2015), and LG’s own new mobile VR solution, the 360 VR, which made its debut at MWC 2016.

It’s actually not all that surprising that Samsung and LG both trust Fraunhofer to help build the aural foundation of their new VR creations. Fraunhofer’s audio wing has created audio codecs licensed by over 1,000 companies in eight billion devices after creating the MP3, and co-creating the AAC codec family. Virtually every music and video streaming company in the business, from YouTube to Spotify, uses the company’s stuff. And with VR on the horizon, Fraunhofer wants to dominate that, too.

We sat down with Fraunhofer at MWC to find out more about its plans to help shape the impending VR revolution, which will rely heavily on realistic sights and sounds to take flight.

“We were the first ones to do Internet audio, and we’re also now the first ones to do 3D sound for VR,” says Head of Marketing for Fraunhofer’s Audio and Multimedia division, Matthias Rose. “We keep on with our tradition of pioneering new markets for sound now in this new field of virtual reality.”

Whatever your thoughts about compressed audio, it transformed the way we hear the world of entertainment forever. Without it, there never would have been an iPod, much less Spotify, YouTube or Netflix. Now, Fraunhofer is hoping to do the same with VR, thanks in part, to two new highly mobile file codecs: HE-AAC, and what the company calls the “next generation” codec, MPEG-H.

But before we get into that, a bit on the emergence of 3D surround sound and its place in the VR landscape.

Making your ears hear in 3D

If you follow surround sound even a little, you’ve probably heard of Dolby Atmos. Dolby (along with and its equally-invested rival, DTS) has been a leader in creating a new way to mix and master audio tracks, based around what the pros call “sound objects.” While previous surround sound mixing was limited by recording and mixing audio into specific channels, the sound-object method allows for individual objects on screen — like a helicopter, or a buzzing bee — to move freely throughout sonic space, essentially constrained only by the imagination of the audio engineer.

Acoustic-Lab-Fraunhofer-IIS

As anyone who’s been to a Dolby Atmos-powered movie theater can attest, the effect can be dazzling. Rain drops seem to come from on-high thanks to speakers mounted on the ceiling, laser beams and rockets strafe from front to back through the theater, and so on. The more creative the audio engineers get, the more realistic the cinematic experience becomes.

As an interactive medium that tricks us into believing we’re in another world, VR arguably offers an even more impactful and visceral medium for sound objects to play in. Moreover, Fraunhofer says realistic surround sound is more important in VR than cinema because humans rely heavily on sound cues to inform us of our environment. For instance, the reflections of sounds in a specific space tell our brains if we’re in a tiny room, or a big echoey cave. Movements of objects in VR environments — such as a character running down a hallway, or a car driving down the road — need to sound believable in space — otherwise, the illusion of VR quickly falls apart.

“We were the first ones to do Internet audio, and we’re also now the first ones to do 3D sound for VR.”

That’s especially important in the nascent world of VR cinema, where the sound objects themselves can help dictate the plot.

“When you make cinematic content where the user can look anywhere he wants, you can use audio to help tell the story, making sure that the audience actually follows (the director’s) path of action and (doesn’t) wander off somewhere else,”  explains Jan Nordmann, Fraunhoffer Senior Director of Business Development for New Media. “So, if you have an explosion happening behind you, you will most likely look in that direction. The director can pick it up from there and give additional audio cues to make sure you’re following along.”

Along with sound objects, Fraunhofer’s audio tech also, of course, incorporates head tracking cues: No matter where you move your head, the sound stays in the proper place within the game or movie.

VR for all

Dolby, DTS, and others also want to sculpt the sound that makes VR seem real. In fact, Dolby showed us some of its applications in that department at MWC as well, powering the VR film, Collision, on a Samsung Gear VR.

However, unlike most object-based surround systems, Fraunhofer’s entire infrastructure — from its mixing and mastering plug-ins, to its MPEG-H and AAC-HE files — is totally open for anyone to use. Partners for its tech include a vast array of developers, from device makers, to broadcasters and content creators, to streaming services. That may well give Fraunhofer a leg up on competing proprietary systems.

“That’s one of the main differences between us as an audio company like Dolby (or) DTS: We make open standards together with partners, and they make proprietary products,” says Nordmann.

Acorn to oak

Fraunhofer’s system begins with the company’s 3D surround sound plug-in, which works with virtually any digital audio recording software, from ProTools to Nuendo. With the company’s intuitive plug-in, it’s easy to move sound objects around in a virtual 3D environment — with very little processing power, according to Fraunhofer. In a quick demo, I was able to easily move around a sound object — in this case, the voice of a hero in a VR film — and listen as the sound appeared to move closer and further away, as well as all around my head, in real time. And that’s just one piece of Fraunhofer’s VR audio architecture.

Cingo-VR

At its advanced audio lab (which the company claims is one of the three most sophisticated labs of its kind in the world), Fraunhofer is also developing hardware like 3D microphones to better capture 3D sound in a live environment. As such, the company has its hands on the 3D sound process from start to finish, beginning with recording, and ending with rendering the sound exactly as you hear it in the studio via the compressed MPEG-H and AAC-HE file formats.

Going mobile

Standardization is all well in good, but how do you create believable 3D sound for mobile devices and plain old stereo headphones?

“Fraunhofer audio powers, not all, but most of the mobile VR experiences that are on the market.”

“We are really a mobile audio company because of our AAC and MP3 developments,” says Nordmann. And that extends far beyond stereo sound. As users increasingly began taking their video with them in the streaming age via smartphones and tablets, they still wanted a full surround sound experience. That doesn’t work very well over those tiny speakers in your mobile device, so in 2013, Fraunhofer created its own infrastructure for virtual surround sound over headphones.

“We developed a product called Fraunhofer Cingo, which is basically a post-processing library that allows you to have virtual surround sound over regular stereo headphones. And that’s what Google incorporated into Google Nexus devices, in addition to the Android operating system,” Norman says. “That allowed Google Play to start selling movies with 5.1 surround sound encoding with our HE-AAC codec, and then rendered for headphones with Cingo. And that has been in every Nexus device since 2013.”

“At that point, that was a surround rendering technology for headphones, but we kept developing it, and now we have a full 3D rendering engine,” Norman explains. “And that is what Samsung licensed from us for the Gear VR.”

LG’s 360 VR is using it too, along with Google Cardboard devices (via Android), and others. “Fraunhofer audio powers, not all, but most of the mobile VR experiences that are on the market,” Nordmann says. The system is also native to iOS devices for deployment if and when Apple gets into the VR game.

Fraunhofer’s goal is to allow audio pros to easily render high-quality 3D surround sound files in an open-source environment which, in turn, may help allow the brilliance of the best and brightest innovators in VR to shine. Thanks to Fraunhofer, and its partners, the future of audio for mobile VR is already here.

Now all we have to do is wait for the new generation of virtual reality content creators, device makers, and other innovators to take it and run.

Mobile

Sonarax harnesses ultrasonic sound for improved security, indoor navigation

Navigating the mall, finding your car, and authenticating your identity for mobile payments can all be a pain, but that's not all they have in common -- they could also potentially be made much easier with high-frequency ultrasonic sound.
Computing

Make your games sound amazing with one of these sets of awesome gaming speakers

If you want to take your gaming sound to the next level, you need to find the best gaming speakers for your set up. Whether you're on a tight budget, want some fancy RGB lighting, or just need the best speaker set money can buy, these great…
Home Theater

What is Dolby Atmos Music, and how can you experience it?

Dolby Atmos is a surround sound format that creates immersive, 3D soundtracks to make movies come alive. But the same technology is also being adopted for music creation, and it's a game-changer. Here's everything you need to know.
Home Theater

The best noise-canceling headphones paint your music on a cleaner canvas

Drowning out the sound of babies, jet engines, and the outside world isn't as difficult as it seems. Here are the best noise-canceling headphones, whether you're concerned with style, comfort, or sound.
Gaming

Minecraft AR teased during Microsoft Build 2019, full reveal arrives on May 17

At the Vision Keynote during the first day of Microsoft Build 2019, we saw a short trailer of what looks to be Minecraft AR, an upcoming experience coming to mobile devices. Watch the video here and join us for the full reveal on May 17.
Photography

As AR heads to Google search, Lens learns to translate, add tips, and more

Google Lens can soon help you decide what to eat by scanning the menu, then calculate the tip with one look at the receipt. Translation options are also expanding for Lens, while augmented reality tools will be built into search results.
Gaming

Minecraft Earth hands-on preview: The next big AR craze Is coming

Minecraft Earth is a new mobile augmented reality game from Microsoft that has Pokémon Go dead in its sites. The cute monster collect-them-all started the genre, but Minecraft Earth hopes to redefine it.
Mobile

Google’s ARCore is getting better at tracking moving images

Google has announced a series of updates to ARCore that should make the augmented reality experience a whole lot more fluid. For example, ARCore is getting better at tracking moving images through Augmented Images.
Mobile

Nike Fit aims to help you slip into your new sneakers more easily

Nike says a new augmented reality feature for its app should increase the chances of a snug fit for customers buying its shoes online, or even in-store. Nike Fit launches in the U.S. in July and the following month in Europe.
Computing

Dive headfirst into the best VR experiences available now on the Oculus Rift

The Oculus Rift brought back virtual reality and put a modern twist on it. Grab your Touch Controllers, put on your VR headset, and jump into the fun with some of the best Oculus Rift games available now.
Virtual Reality

Your first augmented reality work helmet might be made by Lenovo

A new augmented reality headset from Lenovo could challenge Microsoft's dominance in the commercial AR industry, with the ThinkReality A6 offering an AR experience akin to something between the first and second-generation Hololens headsets.
Gaming

New Sony patents could be PlayStation’s answer to Google Stadia

Google Stadia will be attempting to cut into the home gaming console pie but one of its competitors may already have an answer for the game streaming platform. A new Sony patent shows what could be the PlayStation spin on streaming.
Gaming

Immerse yourself in a new universe with these incredible PSVR games

The PSVR has surpassed expectations and along with it comes an incredible catalog of games. There's plenty of amazing experiences to be had so we've put together a list of the best PSVR games available today.
Mobile

Google’s new $999 augmented reality smartglasses are ready for business

Google unveiled Google Glass Enterprise Edition 2, a new version of its business-focused augmented reality wearable. The company's smartglasses ship with a faster processor, an updated camera, and safety frames from Smith Optics.