Robbie Cooper, co-founder of Metapixel, has a vision. “I want fully animated, believable characters in a VR game or environment that react in a totally natural manner. To have that feeling of wanting to reach out and touch them because they’re so real. That’s how a VR world should be.”
He’s not just dreaming about it: He and the Metapixel team are actually working towards making it possible. Using an incredible camera rig and some custom-made software, Metapixel creates astonishingly lifelike avatars with photo-realistic features, based on high-resolution images of an actual person, ready for use in games, movies, and beyond. We visited Metapixel’s office in Brighton, U.K. to see the technology in action, and have our own avatar created.
Photo-real avatar
The camera rig looks like a portal to another dimension; a single click of the shutter is all it will take to zap you across galaxies or through time. There are 102 Nikon D810 cameras mounted on it, all equipped with one of three different lenses — a 50mm, an 85mm, or a 72-300mm zoom.
There’s no galaxy zapping involved, of course, but once that shutter is released, each camera takes a 36-megapixel RAW-format picture of the subject from almost every conceivable angle. Those shots are then transferred to Metapixel’s computer system where they are processed in a matter of seconds. Over the course of the next hour or so, the software creates an amazing photo-real avatar on screen, looking exactly like the subject. It’s actually a technique called photogrammetry, where photos taken from different viewpoints are stitched together to form a single, 360-degree render similar to a 3D body scan.
And when I say exactly like the subject, I mean it. The result is uncanny. To test it out, I was invited to stand in the middle of the rig, and have my image immortalized forever. It’s likely the first and only time 102 cameras will be pointed in my direction at the same time. Told to stare at a single point and keep completely still, with my arms out in a slightly uncomfortable manner, I waited as the cameras were primed — then I was blinded by a flash as my photo was taken.
The more realistic the avatar looks and acts, the more you’re engaged with what’s going on.
Each picture is carefully checked for errors, odd lighting, and shadows (which can confuse the computer and ruin the end image), and for any missing images where a camera for some reason misfired. Metapixel’s own software, called Capturing Reality, then does all the heavy lifting. It pulls together each photo and maps the image on a mesh, building up the final avatar. It took 10 to 15 minutes to build up the first stage, then another 20 to add the texture and color.
You end up with the picture seen here, an avatar so real it’s frightening. It’s also considered only “normal” quality. There are other stages, which increase the level of detail even further. The whole process took less than an hour, including all the camera setup and the processing.
The possibilities
What can be done with your picture-perfect avatar? There are masses of exciting possibilities, but for Cooper, it’s all about virtual reality. “Combine this with really good artificial intelligence, and the world becomes real,” he said. For example as AI improves, he told me, avatars could understand where you’re “looking” in a VR game and respond accordingly. The more realistic the avatar looks and acts, the more you’re engaged with what’s going on.
He pointed out the work of animation studio 3Lateral, where by using face-tracking technology, photo-real avatars mirror the facial expressions and reactions of an actor, as proof of what’s already being worked on. To come face-to-face with what’s almost a living, breathing person in a VR game or environment has the potential to change the way we interact and empathize with in-game characters.
There’s also exciting potential in movies. An actor who is scanned this way becomes essentially immortal. Imagine actors continuing to work even after their death, or calling up a scan from years ago to act out a scene in which the actor is required to look younger — all this is possible with a digitally reproduced, identical-looking avatar. It goes a step beyond 3D animation, and can use the techniques seen in motion capture.
Combine the scan with a 3D printer and you’ve suddenly got absolutely perfect looking action figures, too. Bring the technology into museums and teaching environments, and you return to Cooper’s vision of characters you believe in. You may someday see and hear a historical person tell the tale of their lives, with true emotion registering on a face that appears to be completely human. It’s exciting stuff, and the possibilities keep occurring the more you think about it.
Almost reality
We’re still a while away from seeing all this in reality. Metapixel has come a long way very quickly; the company only built its first camera rig, with 30 cameras at the time, at the end of 2015. While it has expanded, the custom software is still being worked on, and there are upgrades to make the photo-taking process more efficient coming soon. Besides, VR itself — the key driving tech for Metapixel — is still in its infancy.
Metapixel is attracting plenty of attention for such a young company, and it is working with a movie production company, VR developers, and even conference organizers to supply personal, full-body scans on a USB stick to all participants. Imagine having the chance to use that avatar at a conference held in virtual reality in the future. The results are impressive and exciting for what’s really an early-stage technology. We can’t wait to see it evolve.
