Bringing realism to VR is complex, but these developers found a way in holograms

Read any extensive coverage of virtual reality and undoubtedly you’ll come across the word “immersion” at least once or twice. That’s a relatively vague term for the feeling of presence within the virtual world, but it’s as close as we’ll get to providing a singular point of reference for gauging the success of a VR developer in crafting a believable experience.

Most developers utilize similar technique to achieve immersion. They work within the bounds of existing virtual reality headsets and craft small-scale, high-fidelity scenes, with binaural audio. If they do a good enough job, you may find yourself wandering into the uncanny valley. It’s rare to climb the rise on the other side — yet that’s exactly what Awake from virtual reality production house Start VR is attempting to do.

Its roughly 20-minute first episode is a lightly interactive cinematic VR experience that delivers a captivating story around the theme of lucid dreaming. But as intriguing as the narrative is, it’s the volumetric capture technique used to craft it that is Awake‘s most striking element.

The end result is characters with incredibly realistic facial expressions, motions, and gestures. This is no ping-pong-ball motion capture. It’s something else entirely.

**Mild spoilers for Awake’s story may be present in the following interview.**

Behind the scenes of the filming of Awake VR motion capture
Start VR

So long ping pong balls, hello holograms

“The technique that Awake was specifically written around is volumetric video,” explained creator and director Martin Taylor. “So it’s a new type of performance capture that takes the full actor’s performance including all of the wardrobe and the full expression and captures them as a filmic hologram.”

Volumetric capture wasn’t something Start VR could perform alone. It needed Microsoft’s help.

This is very different from the motion capture techniques used to craft most in-game characters for both virtual reality experiences and more traditional 2D games. While those use samples of movements to craft the 3D models that are skinned to bring virtual characters to life, volumetric capture records the entire person. Inside and out, so to speak.

“It’s using a special rig with 106 cameras with depth sensors all pointing towards a cylindrical volume in the center and up to two actors can stand in that volume and deliver their performances,” Taylor told Digital Trends. “All of these cameras and sensors capture that, and lots of processing and software stitches it together into a completely believable and solid digital copy of the performance.”

That gives the developers not only an accurate 3D representation of where those actors are and how they’re posed at any particular time, but a near photo-realistic video impression of them too.

Start VR's 106 camera recording rig let developers model actors with a kind of accuracy and photo realism not typically seen in VR games.

“The raw version of the captures are a huge series of OBJ models which are all uniquely different from one another and then a video texture that’s wrapped over the top,” Taylor said.

Early pioneers in its usage, volumetric capture wasn’t something Start VR could perform alone. It needed Microsoft’s help.

The right dance partner

In one of the more uplifting segments from Awake: Episode One, our protagonist, Harry, shares a dance with his partner with the image splashed across the in-game wall as if from some unseen projector. Just as those characters found each other within the game’s universe, Start VR required a specific partner to help create the volumetric performances that form the centerpiece of Awake’s experience.

“We did some very crude early tests with this technology with a Microsoft Kinect and a Canon 7D camera joined together and the results were comically crude,” Taylor explained. “But there was just something in the experience of being inside VR and knowing you’re looking at a real person […] It took you that bit closer to getting across the uncanny valley to give you a very connected experience. It was a really exciting moment and very early on in our business.”

Initially Start VR looked into developing its own volumetric capture rig, but the complexities of managing such a facility, as well as the up front cost involved in acquiring and configuring so many cameras and depth sensors was too much to consider.

“I thought there was no way that we would be able to pull this off without this level of grunt.”

The team spent a couple of years looking at local suppliers and studios with their own volumetric capture set ups. None of them could do what Start VR wanted, which was to not only record performances, but be able to re-light them after recording. One of the most unique aspects of Awake is that the player, although limited in interactive possibilities, can release small motes of light from their virtual hands, which can enlighten shadowed areas of the experience.

The one company who would make this possible, as well as deliver the kind of motion capture system Start VR needed, was Microsoft.

“It wasn’t until we turned up at their studio that we realized what was required to make this work,” Taylor continued. “The amount of computers, raw processing power, hardware and infrastructure made us think. Immediately I thought there was no way that we would be able to pull this off without this level of grunt.”

Behind the scenes of the filming of Awake VR motion capture
Start VR

From stars to screens

Once the actors’ performances have been captured, bringing them into the game environment was no simple feat. Start VR blazed a trail in software as well as hardware.

“The other big challenge was [building] the experience in Unreal engine,” Taylor explained. “The plugin didn’t exist to accept dynamic lighting. We had to do basic functions like create an in point and an out point. If two characters were filmed in the same volume we had to split them apart and put them on different sides of the room.”

“We’re definitely pushing the engine to do things it doesn’t natively do. We’re working on it daily.”

Once that was solved and the experience was starting to come together, the next big problem was optimization. Start VR worked closely with Nvidia to get the experience running smoothly on a GTX 1080 Ti graphics card and then began to work its way down to see what kind of cuts could be made to allow the experience to run on weaker hardware.

Awake currently offers two modes of play. The high fidelity option is by far the more enjoyable experience, though Start VR suggests that a GTX 1080 as a minimum required specification. We managed to get it working on an AMD Radeon Fury X, but frame rates did stutter at times. The standard option is much more forgiving, but the visuals do take a significant hit.

This may not always be the case with this sort of technology, though. Start VR CEO, Kain Tietzel told Digital Trends that he believes Nvidia may leverage bespoke hardware (perhaps akin to the Tensor and RT cores found in its RTX graphics cards) to offer improved performance for volumetric capture video.

awake game start vr volumetric holograms awak episode one bird
Awake: Episode One Start VR

“No one has really written anything specific to handle this kind of rendering,” he said. “As demand increases and the production starts to become more common place I imagine we’ll start seeing Nvidia having dedicated hardware acceleration to make the performances greater and more realistic.”

Install size was also a major concern for Start VR.

“The raw capture is 10GB per second of raw volumetric capture so there’s a lot of work that goes into squeezing 15 minutes into a 10GB [download],” Taylor said. “It’s using the latest and greatest of all the plugins within Unreal Engine. We’re definitely pushing the engine to do things it doesn’t natively do. We’re working on it daily.”

You’re not a participant in this world. You’re a voyeur. For Start VR, that was the plan all along.

Start VR could opt for even higher fidelity models than it uses now. Earlier implementations of volumetric capture like Factory 42’s “Hold the World educational experience hosted by David Attenborough, look better than Awake in some senses. That’s because they are delivering a mere three degrees of motion virtual reality experience. They’re also focusing all of the attention on the face and upper body.

“You could push the polygon count we allow [in Awake] just into the face and then it becomes super high resolution,” Taylor explained. “There are some tradeoffs we’re choosing there. We’ve chosen to go for entire body captures because we want people to be able to see each part of the character, walk around them. I’ve seen people rolling around on the floor to see if the characters are persistent.”

Playing with holograms

After all the effort and special consideration needed to craft Awake’s volumetric experience, the end result is something very different from what most gamers will be used to. The “texture” of the characters vary from extremely detailed on their faces, to roughshod and blurred in harder to reach areas. The mix of extremely realistic motion and somewhat blocky models reminded us of Rockstar’s 2011 release, LA Noire with its unique visual style. Or as if someone had wrapped a UHD video over an original Xbox character’s shell.

Awake: Episode One | Start VR
Awake: Episode One Start VR

It’s far from perfect, with the technology struggling in particular with gaps between fingers and hairs, and a slightly blocky outline of the characters. But it’s captivating in a manner that is rare to find in virtual or any other kind of simulated reality experience, and it holds huge potential for a different kind of interactive experience.

While screenshots of Awake may have an uncanny valley vibe to them, when you’re actually in Awake’s world, there’s something about the volumetric capture that hurdles that depression entirely. The actors don’t look perfect, but they look real, and that’s something that cannot be said for even the best motion captured CGI models.

“Even understanding how we light and direct performances will change as well.”

That immersion in the virtual world is also somehow enhanced by forcing you to take a step back from it. You’re not a participant in this world. You’re a voyeur. For Start VR, that was the plan all along.

“[Other filmmakers] always feel compelled to have the performers in the scene look at you, the camera, and I don’t know if that necessarily works all the time,” Tietzel told Digital Trends. “If you’re constantly treated like you have to be looked at I think it changes the nature of the performance and I think it’s rare that you actually see believable performances acting to a camera and you’re meant to react to them.”

Adding final touches to our upcoming VR masterpiece AWAKE. Helps to have the right tools though in the @wacom #cintiq #cintiqpro !! ⭐️???????? #digitalart

Posted by Start VR on Monday, November 26, 2018

That’s not to say that you never feel involved in the scene in Awake. There are moments, specifically crafted ones, where you end up in the line of sight of a character, giving you jolts of wonder as to whether they can see you or not. It doesn’t pull you out of the experience, but reminds you of your voyeuristic roll in the scene. If anything, it enhances how personal the story is, despite its hints at a more expansive narrative beyond this first episode.

(Lucid) dreaming of the future

Start VR has eight episodes planned out for the Awake series already, with Taylor having already spent years crafting a story that is both personal and expansive. As those episodes roll out we can expect lots of improvements, we’re told.

“The technology, the accuracy, the frame rate is going to change dramatically,” Tietzel told Digital Trends. “Even understanding how we light and direct performances – what’s the best way to get performances out of the actors in that space will change as well. This is a learning experience.”

“So there’s moments where the character reaches out to give you something. Do you feel compelled to try and take that off them?”

As well as expanding their own expertise with directing actors and VR game development, Start VR is also working with Microsoft to continue enhancing the volumetric capture technique to bring new wrinkles to the virtual canvas.

“Having the character notice you is one way to tell a story and already I think there are variations coming up where we can add in eyeline,” Taylor highlighted. “We’re working with Microsoft to control the character’s head movement to be able to turn and lock on to you up to about 30 degrees either side of the recorded position […] Also compelling people to do reactive things. So there’s moments where the character reaches out to give you something. Do you feel compelled to try and take that off them? It’s those different ways of connecting with a character rather than just looking at them that we’re trying.”

Awake: Episode One | Start VR
Awake: Episode One Start VR

Start VR also plans to experiment with multiple environments (Awake: Episode One takes place almost exclusively in one. Maybe one and a half), but it’s beyond the scope of virtual reality which Taylor and Tietzel feel their experiments with volumetric capture will be most readily felt.

“We’re already exploring what it might mean to bring this kind of capture into augmented reality, and real life performances in the real world are the next step,” Tietzel said. “Early tests in this area have delivered some fascinating results.”

These experiences and Start VR’s first episode of Awake highlight the exciting potential of volumetric capture. They sidestep the performance hiccups of technologies like ray tracing for realistic in-game lighting and deliver something that is believable up close, even if it still has a few blemishes.

Awake: Episode One is an exciting first step in a narrative journey, but it’s an even greater leap for the technological enhancement that helped birth it into the world. We’re excited for what Awake’s story and the capture technologies it champions may evolve into, and imagine we’ll be seeing a lot more of them both in the future.

Awake: Episode One is available on Steam and Viveport with a price of $8.

Movies & TV

Why First Man’s Oscar-nominated visual effects are a giant leap for filmmaking

Paul Lambert, the award-winning visual effects supervisor on First Man, reveals the innovative techniques that blended old footage with modern movie magic to make the Apollo 11 mission to the moon resonate with audiences 50 years later and…
Movies & TV

How Avengers: Infinity War’s Oscar-nominated VFX team made Thanos a movie star

The purple-skinned Thanos proved to be a breakout character in Avengers: Infinity War, thanks to the work of actor Josh Brolin and visual effects studios Digital Domain and Weta. Here's how they brought him to life and earned the film an…
Virtual Reality

Getting into VR is spendy. Which headset is truly worth your hard-earned cash?

Virtual reality has finally gone mainstream, but how do you find the best VR headset for you? Check out a few of our favorites, whether you want the best of the best or a budget alternative for your mobile device.
Movies & TV

Eye-popping Alita: Battle Angel delivers a beautifully hollow cyberpunk spectacle

With Alita: Battle Angel, Robert Rodriguez and James Cameron finally deliver a cyberpunk adventure that keeps the action high with groundbreaking visual effects, even when the story falls short.
Gaming

Bayonetta 3 still in the works, set to launch on Nintendo Switch this year

Bayonetta 3 was notably absent from the latest Nintendo Direct presentation, raising concerns that the game will be delayed. A Nintendo executive clarified that the Nintendo Switch exclusive remains on track for a 2019 release.
Gaming

Over 16,000 Apex Legends cheaters have already been banned by Respawn

Respawn Entertainment revealed that it has already banned over 16,000 cheaters in Apex Legends in the less than two weeks since its release. The developer may be working on an in-game feature to report cheaters.
Gaming

Get the most life out of your Xbox One controller with these tips

The Xbox One controller has fantastic battery life, but eventually your batteries will still die. To make it even better and ensure you never have to stop gaming, follow these tips.
Gaming

How you can share your best gaming moments with friends on the PS4

Check out Digital Trends' quick guide to everything you need to know to save your outstanding PlayStation 4 gameplay moments, share them online, and transfer them to your computer.
Gaming

A beginner’s guide to Far Cry New Dawn’s postapocalyptic Hope County

Far Cry New Dawn is yet another entry in Ubisoft's popular open world shooter franchise. But while it may feel quite similar in motion, the progression system is changed in a lot of ways. Our beginner's guide is here to help.
Gaming

Learn to scavenge like a pro with our Far Cry New Dawn crafting guide

Far Cry New Dawn revolves around the hunt -- the hunt for materials to craft new weapons, vehicles, and other useful items. It's easy to miss out on great opportunities to find new materials if you don't go off the beaten path.
Gaming

Here's our take on the best game-streaming services currently out there

You can still get your physical video game discs or cartridges at your local store or download games digitally, but another option is even more convenient: Game streaming. Here are the best game streaming services.
Gaming

PS4 vs. Switch: After weighing the pros and cons, which one comes out on top?

Nintendo Switch versus PlayStation 4: Which one has better overall value? We break down the pros and cons of each platform to tell you which of these consoles is truly worth the money.
Gaming

Xbox's app lets you access your console while away from home. Here's how

Microsoft's Xbox allows you to access your profile information and launch media content directly from your mobile device. Check out our quick guide on how to connect your smartphone to an Xbox One.
Gaming

Here’s how to set up a virtual private network (VPN) on your Xbox One

Online privacy is more important now than it's ever been, and gaming is happening online more than ever before. Here's a quick guide on how to set up a VPN for your Xbox One so you game in safe anonymity.