Samsung tapped Hollywood talent to bring AR Emoji to life

Embedded into the Samsung Galaxy S9’s camera app, AR Emoji allows you to use the front-facing camera to create and send digital cartoon avatars. Similar to Apple’s Animojis in concept, AR Emoji captures your facial movements and mirrors them in the form of a character or animal. But with its “Create My Emoji” feature, you can also create one of yourself by simply taking a selfie — allowing you to choose a skin tone, outfit, hairstyle, hair color, and glasses.

With AR Emoji, Samsung set out to solve what it sees as a problem — finding a way to replace existing emojis with ones that are interactive. The company wanted to do this in a way that was simple, by providing users with the right tools to capture emotion but still allowed them to easily share the result on a variety of messaging platforms.

So, how did Samsung create its AR Emojis? For starters, it went straight to the 3D animation experts who brought to life some of Hollywood’s most popular characters.

From big screen to smartphone camera

To power the new feature, Samsung collaborated with Loom.ai —  a San Francisco-based startup specializing in 3D animation. The company’s software allows users to create fully personalized and customizable 3D avatars from a single photograph, without custom hardware or depth sensors.Through a partnership and a software development kit (SDK), Samsung combined Loom.ai’s technology with the S9’s camera tech (augmented reality and facial recognition) to make custom AR Emoji possible.

The vision behind Loom.ai all started with movies — cartoons and superheros to be exact. Having previously worked at popular visual effects companies like DreamWorks and Lucasfilm, Loom.ai’s founders –Mahesh Ramasubramanian and Kiran Bhat — have helped bring well-known characters, such as Shrek and the Hulk, to the big screen.

To inject personality into those characters, they used performance-capture technology to incorporate the actors who played them — recording the movements of the actor, not their visual appearance. But after seeing how movie audiences reacted to their 3D creations, Ramasubramanian and Bhat decided to take that technology further — beyond Hollywood — by starting Loom.ai, which focuses on smartphones.

“People instantly react to having these characters come to life when it’s driven by human actors,” Ramasubramanian, who serves as CEO and co-founder, told Digital Trends. “We started Loom.ai with the goal that we need to be able to find an easy way for every individual to be able to see themselves in 3D, and for that we had to build a new technology.”

With feature films, creating a 3D avatar requires tons of equipment — expensive hardware and software, makeup, and a studio setting. But Loom.ai found a way to simplify the process of capturing a person’s features without the usual tools; instead, its patented technology uses a single photograph and deep learning to analyze an image. If a photograph is free of any facial expressions, the deep-learning algorithm looks at the pixels and automatically produces a face that encapsulates the correct muscle movement.

“When Samsung’s partnership rolled around we had a chance to implement that sophisticated technology on a device, which we feel is already a pretty compelling engineering task by itself,” Bhat, Loom.ai’s CTO and co-founder, told Digital Trends. “It’s a big technology jump in being able to take some of those ideas and aspirations from the big screen and bring it to mobile.”

Incorporating this technology into the Galaxy S9 hardware required Loom.ai to restructure its algorithms, in order to run on the device. But Loom.ai did not create AR Emoji alone: Both Ramasubramanian and Bhat credit Samsung in dedicating tremendous effort into optimizing the software. By providing the SDK to integrate into smartphones, the Loom.ai founders think Samsung can make the framework much richer by adding more content and customizability options in the future.

How the Galaxy S9 creates a digital avatar

So, how exactly does the S9 (and S9 Plus) create AR Emojis? While Samsung didn’t reveal its secret sauce, it gave Digital Trends a look under the hood.

Working off Loom.ai’s technology, the phone’s camera captures and processes a 2D image into 3D. To enhance this further, the S9 uses additional software — facial tracking and Samsung’s Blend Shape technology — to provide facial expressions for the AR Emoji.

You’re then left with a cartoon-like version of yourself that can emote 18 different expressions — which Samsung plans on slowly expanding to 54 — ranging from joy and pleasure, to sorrow and anger. All data is also local to the device, and doesn’t get stored anywhere else.

Both working together, facial tracking technology tracks over 100 facial features in real time (such as winking or smiling) to capture individual emotions, while Blend Shape helps to deliver a greater range of them — which blends expressions together based on 75 basic human facial features. Samsung says, for Blend Shape, it conducted at least 100 tests on a weekly basis within its labs, in order to make sure there is enough data that would entice its users. (Blend Shape is also used to create its Disney AR Emojis, like Mickey and Minnie Mouse.)

In our own AR Emoji test, a majority of us felt our characters looked extremely inaccurate. One writer felt there was a lack of skin-tone options available, with no dark-skinned tones to choose from. Others felt their AR Emoji didn’t look anything like them, which made editing the avatar seem like more of a chore than an enjoyable task.

According to Loom.ai, creating an accurate AR Emoji relies heavily on the way the selfie is shot. Users should go about taking the photo the same way they’d take a passport photo. “You want even lighting, nothing covering your face, and a neutral expression so that we can add the expressions later,” Ramasubramanian explained. “It’s a matter of people getting a little accustomed to it and then seeing the possibilities of animating it and using it.”

Ramasubramanian also said that users should treat it the way they would while taking any selfie — meaning, you can take tons of photos until you’re satisfied with the photo. Since AR Emoji allows you to create multiple avatars, you can compare them to find the one you like best. Each one you create also saves to your phone, so you can switch up the version you use depending on your mood.

The latest update to AR Emoji includes additional hairstyles, headwear such as a hijab, and other options. The update is currently rolling out to unlocked versions of the Samsung Galaxy S9 and S9 Plus in Korea, China, and Europe. European carrier devices and U.S. unlocked devices should have the update available for download this week. The update for U.S. carrier devices will be rolling out over the coming weeks. Samsung says AR Emoji will continue to evolve in the future, but for now, is staying tight lipped on exact details.