CGI faces will soon be indistinguishable from real ones. Here’s how

cubic motion scanning technology mocap siren live

Not long ago, Andy Wood, the president of a British computer graphics company called Cubic Motion, was in a meeting helping pitch its technology to a major games studio in Los Angeles. To showcase what they were capable of, Cubic Motion’s team had created an animated cutscene using one of the game studio’s trademark properties: a stylized female half-human, half-animal character.

The demo had gone well, the pre-rendered cutscene convincingly delivering the studio’s script. Then the Cubic Motion team executed its agreed-upon master stroke; the stunt they hoped would earn them the contract.

Having delivered its pre-written lines, the character in the cutscene broke the fourth wall. It turned to the camera, seeming to peer out at the business executives in the room, and began to address them individually. This wasn’t a pre-programmed trick, either. The pre-rendered cutscene, it transpired, was not pre-rendered at all. It was acted out live by a performer in Manchester, on the other side of the world, who was being transformed in real-time into the game studio’s character. When the character went “off-script” and started interacting with the people in the room, the effect was staggering.

cubic motion scanning technology mocap siren

“I’ve never seen a reaction like that,” Wood told Digital Trends. “There was an audible gasp from everyone in the room. Someone grabbed my arm because they just couldn’t believe what was happening.” Cubic Motion won the contract.

To appreciate why this is so significant, you need to know a little bit about the way that motion capture characters are typically, well, captured. Motion capture actors are normally recorded far in advance. They wear form-fitting body suits that are covered in dozens of tiny dots, either LEDs or things that resemble miniature ping-pong balls. These are called markers, and are used to provide animators with reference points for tracking a body’s motion in three-dimensional time and space. Once this process has been carried out, animators can set to work fleshing out the finished graphics: quite literally joining the dots until they have something that resembles whatever character they are hoping to create.

cubic motion scanning technology mocap siren close up

Cubic Motion’s approach is different. The company has developed marker-less technology for real-time tracking of models; designed to instantly transform them into robust completed renders, running at a silky smooth 60 frames per second. This involves some impressive machine learning algorithms, which are able to take an image of a face and separate its component parts, digitally mark up the different elements, and allow it to be captured in a near-unprecedented amount of detail.

“As far as we’re aware, there’s nobody in the world able to achieve facial capture through computer vision in the way that we do it to the level of real-time fidelity that Cubic Motion do,” Wood continued.

While there’s a good chance you haven’t heard the name Cubic Motion before, you’ll almost certainly know its work. Its world-beating facial animation technology has been used in games such as God of War, Insomniac’s Spider-Man, Anthem, and Blood & Truth. And there’s a whole lot more to come.

The trouble with Gollum

For many people reading this, Gollum from Peter Jackson’s Academy Award-winning Lord of the Rings movies trilogy remains a high-point for motion-captured performances. It’s one of the first times that audiences saw a CGI character who delivered a performance every bit as emotionally charged as any of the more recognizably human characters in the cinematic trilogy.

“I often use Gollum [from the Lord of the Rings trilogy] as a benchmark of a credible motion capture-delivered character,” Wood said. “But when Andy Serkis played Gollum, he was tracked using motion capture that required him to wear multiple markers on his face. The team did a stunning job, but there are limitations. You can’t put dots for markers on people’s eyes, for instance. You can’t track tongues, and so on. To achieve the finished effect, a huge team of artists were needed to do things like hand animate his eye movements because these couldn’t be tracked. With our marker-less tracking, every expression from an eye blink to a pupil dilation can all be recorded — and, furthermore, in real-time.”

Cubic Motion doesn’t do all of this in isolation, of course. It works with other industry-leading companies such as Epic Games, Tencent, 3Lateral, and Vicon, who have developed everything from high-definition scanners to the Unreal game engine. But Cubic Motion’s contribution is profound. Its technology provides the enabling underpinnings that brings together all the other parts. No amount of high-definition dimples, perfectly shaded skin or beautifully rendered peach fuzz on a human face could make up for movement that looks rigid and unnatural.

“If you can capture the essence of a human being, and digitally transfer that onto a digital character or double, it’s like a transference of the soul,” Wood continued. “It allows you to take all the characteristics of a human and transfer it onto something non-human in a way that is compelling and utterly recognizable.”

cubic motion scanning technology mocap persona 2

Soul is a concept that comes up a surprising amount when it comes to technological reproduction. There are reports from anthropologists about indigenous tribes, first encountering western explorers, who did not want to be photographed due to their belief that this photographic process would somehow steal their soul. They needn’t have worried, since hundreds of years later we’re still struggling with technology to do this. Cubic Motion believes that it is on the right path to help. The ability to instantly track the movement of a human face and immediately transfer it to a CGI character, photo-realistic or otherwise, is a game-changer. It makes it possible to capture the incredibly nuanced performances of an actor in a way that was unimaginable just a few years ago.

In 2018, the company showed off its technology by debuting a photorealistic reproduction of a young Chinese woman, called Siren, whose real facial features were tracked in more than 200 locations and reproduced digitally without any perceptible delay. You might stop short of calling it soul transference, but there’s no doubt that the results have a realistically lifelike energy to them. No eyeball-drawing animators required.

How will this be used?

This isn’t just about the cost benefits of saving animators time, however. It hints at a future in which all of us can be scanned and reproduced in astonishing fidelity and placed into digital worlds in unparalleled levels of detail. This could be used for game features allowing players to be scanned and inserted into a game in hitherto-unimaginable levels of photo accuracy.

But there are plenty of other scenarios in which this technology could find a home. Consider, for instance, being able to interact with a virtual avatar of your doctor, perhaps via augmented or virtual reality, for a virtual appointment. Or imagine that you have an 8-year-old kid who struggles with math. What if they could be given extra tutoring courtesy of an on-screen avatar resembling a photo-realistic version of themselves at 12, having buckled down and gotten to grips with the subject? Heck, us adults could access a fitness app presented by a realistic avatar resembling us in 18-months, if only we get serious about eating healthy and hitting the gym three days a week. “You could create a self-fulfilling prophecy to give you self-belief,” Wood said.

These sound like the stuff of science-fiction right now, but it’s not as unfeasibly futuristic as it might sound. For instance, a Japanese company called Gatebox markets devices not dissimilar to an Amazon Echo or Google Home — only featuring tiny animated assistants designed to interact with their owners. As it happens, the effects of having inspirational avatars modeled on users have been investigated as well. At New York University, a study into long-term decision-making found that users who interacted with a personalized avatar, artificially aged to look older, thought more carefully about decisions that would impact their future. They were more likely to save money for retirement than those who saw an avatar the same age as themselves.

Of course, such avatars needn’t remind us of the creeping hand of death on our shoulder. Technology such as this could just as easily transform us into a younger, better looking version of ourselves. Or into a person of another gender — or even another species, real or imagined. The sky’s the limit.

Cubic Motion’s real-time facial capture technology is, at present, far out of the hands of everyday consumers. Wood himself has a full 3D facial scan, captured using the company’s state-of-the-art 360 degree scanning technology, which he shows off at product demos. (Somewhat inconveniently, this means that he must re-shave his beard prior to each demo to ensure his human self perfectly matches the fidelity of his digital double. Such are the sacrifices that pioneers must make.) But for most of us this is not an option. The camera rig Cubic Motion uses, which is just one part of the overall package, would cost you or I hundreds of thousands of dollars.

cubic motion scanning technology mocap persona header

However, self-scanning technology, at a much lower fidelity, is found on today’s smartphones. The iPhone X-series models, for instance, boast something not dissimilar for making possible Animojis and Memojis based on the movement of user’s faces. Cubic Motion has also recently launched Persona, its proprietary capture and helmet-mounted system, which can now be licensed out by other companies. Persona features all the technology, such as in-built infrared lighting, to let others do this work for themselves.

As sci-fi writer William Gibson has pointed out: the future’s here, it’s just not very evenly distributed yet. But as components get cheaper and technology continues to advance, we should expect that to change. Probably faster than we expect.

Spanning the uncanny valley

Back in 1970, the Japanese robotics professor Masahiro Mori put forth the observation that we now refer to as the “uncanny valley.” It’s the idea that, as we build robots or develop 3D computer animations that closely resemble humans, a slight divergence from the real thing can cause feelings of eeriness or even revulsion in viewers. There is good reason for this. A large part of our brain is dedicated to encoding faces. Kids as young as three months learn to categorize faces, long before they can distinguish between other categories of object. As a result, a face that looks not quite like a face unnerves us more than, say, a computer-generated leaf that doesn’t look entirely accurate.

Does Cubic Motion’s impressively lifelike work mean that it has bridged this uncanny valley effect? Wood is optimistic, but he’s not ready to proclaim victory just yet. “Whether we’ve actually made it to the other side of the uncanny valley just yet is for the public to decide,” he said.

cubic motion scanning technology mocap animation by hand

To paraphrase a (possibly apocryphal) line from the American essayist Ralph Waldo Emerson, crossing the uncanny valley is more of a journey than a destination. Like the goal of artificial intelligence researchers of replicating “intelligence” in a computer, creating a totally lifelike digital human is a challenging, non-objective benchmark to achieve. “Anything we think of as digitally real now, we will probably look back on with different eyes in a few year’s time and be able to spot things we can improve on,” Wood explained. “If you go back to movies like Jason and the Argonauts or King Kong, when people first experienced them they looked pretty credible. People walked away thinking the special effects were remarkable.”

For all the charm of stop-motion animation, neither of these effects today look wholly convincing. Nor, we could argue, do many of the photographic methods that we have for recording human likenesses. The softened black-and-white portrait of an actor in a 1940s movie, with all the characterful flaws of celluloid, look positively abstract next to the unforgiving HD of a modern television broadcast. And modern HD will, as crazy as it sounds, one day look dated as well; most likely in the face of fully three-dimensional entertainment.

“But the journey is so advanced now that it’s very obvious that it will get there,” Wood said. And when we do, Cubic Motion very much hopes to be on the finish line. Presumably with our digitized human doppelgängers cheering it on.

News

Third-generation Snapchat Spectacles capture photos and video in 3D

Snap Inc. announced new Spectacles sunglasses that can capture your world in a 3D view. The Spectacles 3 glasses have two cameras on both sides of the frames to capture depth and dimension, similar to how our eyes see the world.
Movies & TV

Here are the best shows on Netflix right now (August 2019)

Looking for a new show to binge? Lucky for you, we've curated a list of the best shows on Netflix, whether you're a fan of outlandish anime, dramatic period pieces, or shows that leave you questioning what lies beyond.
Movies & TV

Who needs sunshine? Stay inside and watch the best movies on Netflix instead

Save yourself from hours wasted scrolling through Netflix's massive library by checking out our picks for the streamer's best movies available right now, whether you're into explosive action, witty humor, or anything else.
Gaming

These are the must-have games that every Xbox One owner needs

More than four years into its life span, Microsoft's latest console is finally coming into its own. From Cuphead to Halo 5, the best Xbox One games offer something for players of every type.
Emerging Tech

Astro the dog-inspired quadruped robot can sit, lie down, and… learn?

Move over Spot! Researchers from Florida Atlantic University have built a new dog robot called Astro. Thanks to deep learning technology, it promises to be able to learn just like a real dog.
Health & Fitness

We spit in a ton of test tubes to find the best and most unique DNA tests

DNA tests aren’t just limited to ancestry. You can test for your risks for certain diseases, the best workouts and diets for your health and fitness, and more.
Emerging Tech

Awesome Tech You Can’t Buy Yet: Racing drones and robotic ping pong trainers

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!
Emerging Tech

Artificial tree promises to suck up as much air pollution as a small forest

Startup Biomitech has developed an artificial tree that it claims is capable of sucking up as much air pollution as 368 real trees. It could be a game-changer for cities with limited free space.
Emerging Tech

Mars 2020 rover now has a rotating array of drill bits for sampling Martian rock

Most the key components in the Mars 2020 rover are installed and ready to go. The next phase of construction was to install the bit carousel, an important mechanism for the gathering and sorting of samples from the Martian surface.
Emerging Tech

NASA selects landing site candidates for OSIRIS-Rex to sample asteroid Bennu

Last year, the OSIRIS-REx craft arrived at asteroid Bennu, from which it will collect a sample from the asteroid to be brought back to Earth. Now, the NASA team has selected four potential sites to choose from for the sampling mission.
Emerging Tech

NASA wants to send two more missions to Mars to collect rock samples

With its Mars 2020 mission, NASA hopes to collect samples from the surface of the planet. The challenge is how to get those samples back to Earth. Now, NASA has revealed its plans for two followup missions to Mars.
Emerging Tech

Eric Geusz: Apple engineer by day, spaceship designer by night

An Apple software engineer by day, artist Eric Geusz spends his nights drawing everyday household objects as amazing, science fiction-style spaceships. Check out the impressive results.
Emerging Tech

The black hole at the center of our galaxy is flaring and no one knows why

At the heart of our galaxy lies a supermassive black hole, Sagittarius A*. Normally this giant monster is relatively docile, but recently it's been a hotbed of unexpected activity, rapidly glowing 75 times brighter than normal.
Emerging Tech

SpaceIL’s crashed lander may have sent thousands of tardigrades to the moon

When the SpaceIL craft Beresheet crashed into the moon earlier this year, it left more than just an impact mark. Thousands of micro-animals called tardigrades were along for the ride and may have survived the crash.