LG and Samsung don’t agree on much. Whether its TVs and smartphones, or washers and dryers, each of the warring Korean tech giants is constantly looking for an edge over its rival. But one thing they do agree on is a little-known German research firm called Fraunhofer IIS. Fraunhofer is responsible for creating 3D surround sound audio tech for both Samsung’s Gear VR (which won our product of the year for 2015), and LG’s own new mobile VR solution, the 360 VR, which made its debut at MWC 2016.
It’s actually not all that surprising that Samsung and LG both trust Fraunhofer to help build the aural foundation of their new VR creations. Fraunhofer’s audio wing has created audio codecs licensed by over 1,000 companies in eight billion devices after creating the MP3, and co-creating the AAC codec family. Virtually every music and video streaming company in the business, from YouTube to Spotify, uses the company’s stuff. And with VR on the horizon, Fraunhofer wants to dominate that, too.
We sat down with Fraunhofer at MWC to find out more about its plans to help shape the impending VR revolution, which will rely heavily on realistic sights and sounds to take flight.
“We were the first ones to do Internet audio, and we’re also now the first ones to do 3D sound for VR,” says Head of Marketing for Fraunhofer’s Audio and Multimedia division, Matthias Rose. “We keep on with our tradition of pioneering new markets for sound now in this new field of virtual reality.”
Whatever your thoughts about compressed audio, it transformed the way we hear the world of entertainment forever. Without it, there never would have been an iPod, much less Spotify, YouTube or Netflix. Now, Fraunhofer is hoping to do the same with VR, thanks in part, to two new highly mobile file codecs: HE-AAC, and what the company calls the “next generation” codec, MPEG-H.
But before we get into that, a bit on the emergence of 3D surround sound and its place in the VR landscape.
Making your ears hear in 3D
If you follow surround sound even a little, you’ve probably heard of Dolby Atmos. Dolby (along with and its equally-invested rival, DTS) has been a leader in creating a new way to mix and master audio tracks, based around what the pros call “sound objects.” While previous surround sound mixing was limited by recording and mixing audio into specific channels, the sound-object method allows for individual objects on screen — like a helicopter, or a buzzing bee — to move freely throughout sonic space, essentially constrained only by the imagination of the audio engineer.
As anyone who’s been to a Dolby Atmos-powered movie theater can attest, the effect can be dazzling. Rain drops seem to come from on-high thanks to speakers mounted on the ceiling, laser beams and rockets strafe from front to back through the theater, and so on. The more creative the audio engineers get, the more realistic the cinematic experience becomes.
As an interactive medium that tricks us into believing we’re in another world, VR arguably offers an even more impactful and visceral medium for sound objects to play in. Moreover, Fraunhofer says realistic surround sound is more important in VR than cinema because humans rely heavily on sound cues to inform us of our environment. For instance, the reflections of sounds in a specific space tell our brains if we’re in a tiny room, or a big echoey cave. Movements of objects in VR environments — such as a character running down a hallway, or a car driving down the road — need to sound believable in space — otherwise, the illusion of VR quickly falls apart.
“We were the first ones to do Internet audio, and we’re also now the first ones to do 3D sound for VR.”
“When you make cinematic content where the user can look anywhere he wants, you can use audio to help tell the story, making sure that the audience actually follows (the director’s) path of action and (doesn’t) wander off somewhere else,” explains Jan Nordmann, Fraunhoffer Senior Director of Business Development for New Media. “So, if you have an explosion happening behind you, you will most likely look in that direction. The director can pick it up from there and give additional audio cues to make sure you’re following along.”
Along with sound objects, Fraunhofer’s audio tech also, of course, incorporates head tracking cues: No matter where you move your head, the sound stays in the proper place within the game or movie.
VR for all
Dolby, DTS, and others also want to sculpt the sound that makes VR seem real. In fact, Dolby showed us some of its applications in that department at MWC as well, powering the VR film, Collision, on a Samsung Gear VR.
However, unlike most object-based surround systems, Fraunhofer’s entire infrastructure — from its mixing and mastering plug-ins, to its MPEG-H and AAC-HE files — is totally open for anyone to use. Partners for its tech include a vast array of developers, from device makers, to broadcasters and content creators, to streaming services. That may well give Fraunhofer a leg up on competing proprietary systems.
“That’s one of the main differences between us as an audio company like Dolby (or) DTS: We make open standards together with partners, and they make proprietary products,” says Nordmann.
Acorn to oak
Fraunhofer’s system begins with the company’s 3D surround sound plug-in, which works with virtually any digital audio recording software, from ProTools to Nuendo. With the company’s intuitive plug-in, it’s easy to move sound objects around in a virtual 3D environment — with very little processing power, according to Fraunhofer. In a quick demo, I was able to easily move around a sound object — in this case, the voice of a hero in a VR film — and listen as the sound appeared to move closer and further away, as well as all around my head, in real time. And that’s just one piece of Fraunhofer’s VR audio architecture.
At its advanced audio lab (which the company claims is one of the three most sophisticated labs of its kind in the world), Fraunhofer is also developing hardware like 3D microphones to better capture 3D sound in a live environment. As such, the company has its hands on the 3D sound process from start to finish, beginning with recording, and ending with rendering the sound exactly as you hear it in the studio via the compressed MPEG-H and AAC-HE file formats.
Going mobile
Standardization is all well in good, but how do you create believable 3D sound for mobile devices and plain old stereo headphones?
“Fraunhofer audio powers, not all, but most of the mobile VR experiences that are on the market.”
“We developed a product called Fraunhofer Cingo, which is basically a post-processing library that allows you to have virtual surround sound over regular stereo headphones. And that’s what Google incorporated into Google Nexus devices, in addition to the Android operating system,” Norman says. “That allowed Google Play to start selling movies with 5.1 surround sound encoding with our HE-AAC codec, and then rendered for headphones with Cingo. And that has been in every Nexus device since 2013.”
“At that point, that was a surround rendering technology for headphones, but we kept developing it, and now we have a full 3D rendering engine,” Norman explains. “And that is what Samsung licensed from us for the Gear VR.”
LG’s 360 VR is using it too, along with Google Cardboard devices (via Android), and others. “Fraunhofer audio powers, not all, but most of the mobile VR experiences that are on the market,” Nordmann says. The system is also native to iOS devices for deployment if and when Apple gets into the VR game.
Fraunhofer’s goal is to allow audio pros to easily render high-quality 3D surround sound files in an open-source environment which, in turn, may help allow the brilliance of the best and brightest innovators in VR to shine. Thanks to Fraunhofer, and its partners, the future of audio for mobile VR is already here.
Now all we have to do is wait for the new generation of virtual reality content creators, device makers, and other innovators to take it and run.
