Samsung tapped Hollywood talent to bring AR Emoji to life

Embedded into the Samsung Galaxy S9’s camera app, AR Emoji allows you to use the front-facing camera to create and send digital cartoon avatars. Similar to Apple’s Animojis in concept, AR Emoji captures your facial movements and mirrors them in the form of a character or animal. But with its “Create My Emoji” feature, you can also create one of yourself by simply taking a selfie — allowing you to choose a skin tone, outfit, hairstyle, hair color, and glasses.

With AR Emoji, Samsung set out to solve what it sees as a problem — finding a way to replace existing emojis with ones that are interactive. The company wanted to do this in a way that was simple, by providing users with the right tools to capture emotion but still allowed them to easily share the result on a variety of messaging platforms.

So, how did Samsung create its AR Emojis? For starters, it went straight to the 3D animation experts who brought to life some of Hollywood’s most popular characters.

From big screen to smartphone camera

To power the new feature, Samsung collaborated with Loom.ai —  a San Francisco-based startup specializing in 3D animation. The company’s software allows users to create fully personalized and customizable 3D avatars from a single photograph, without custom hardware or depth sensors.Through a partnership and a software development kit (SDK), Samsung combined Loom.ai’s technology with the S9’s camera tech (augmented reality and facial recognition) to make custom AR Emoji possible.

Samsung wants to find a way to replace existing emojis with ones that are interactive.

The vision behind Loom.ai all started with movies — cartoons and superheros to be exact. Having previously worked at popular visual effects companies like DreamWorks and Lucasfilm, Loom.ai’s founders –Mahesh Ramasubramanian and Kiran Bhat — have helped bring well-known characters, such as Shrek and the Hulk, to the big screen.

To inject personality into those characters, they used performance-capture technology to incorporate the actors who played them — recording the movements of the actor, not their visual appearance. But after seeing how movie audiences reacted to their 3D creations, Ramasubramanian and Bhat decided to take that technology further — beyond Hollywood — by starting Loom.ai, which focuses on smartphones.

“People instantly react to having these characters come to life when it’s driven by human actors,” Ramasubramanian, who serves as CEO and co-founder, told Digital Trends. “We started Loom.ai with the goal that we need to be able to find an easy way for every individual to be able to see themselves in 3D, and for that we had to build a new technology.”

AR Emoji
Brenda Stolyar/Digital Trends

With feature films, creating a 3D avatar requires tons of equipment — expensive hardware and software, makeup, and a studio setting. But Loom.ai found a way to simplify the process of capturing a person’s features without the usual tools; instead, its patented technology uses a single photograph and deep learning to analyze an image. If a photograph is free of any facial expressions, the deep-learning algorithm looks at the pixels and automatically produces a face that encapsulates the correct muscle movement.

“When Samsung’s partnership rolled around we had a chance to implement that sophisticated technology on a device, which we feel is already a pretty compelling engineering task by itself,” Bhat, Loom.ai’s CTO and co-founder, told Digital Trends. “It’s a big technology jump in being able to take some of those ideas and aspirations from the big screen and bring it to mobile.”

To power AR Emoji, Samsung enlisted the help of Loom.ai —  a San Francisco-based startup specializing in 3D animation.

Incorporating this technology into the Galaxy S9 hardware required Loom.ai to restructure its algorithms, in order to run on the device. But Loom.ai did not create AR Emoji alone: Both Ramasubramanian and Bhat credit Samsung in dedicating tremendous effort into optimizing the software. By providing the SDK to integrate into smartphones, the Loom.ai founders think Samsung can make the framework much richer by adding more content and customizability options in the future.

How the Galaxy S9 creates a digital avatar

So, how exactly does the S9 (and S9 Plus) create AR Emojis? While Samsung didn’t reveal its secret sauce, it gave Digital Trends a look under the hood.

Working off Loom.ai’s technology, the phone’s camera captures and processes a 2D image into 3D. To enhance this further, the S9 uses additional software — facial tracking and Samsung’s Blend Shape technology — to provide facial expressions for the AR Emoji.

You’re then left with a cartoon-like version of yourself that can emote 18 different expressions — which Samsung plans on slowly expanding to 54 — ranging from joy and pleasure, to sorrow and anger. All data is also local to the device, and doesn’t get stored anywhere else.

Both working together, facial tracking technology tracks over 100 facial features in real time (such as winking or smiling) to capture individual emotions, while Blend Shape helps to deliver a greater range of them — which blends expressions together based on 75 basic human facial features. Samsung says, for Blend Shape, it conducted at least 100 tests on a weekly basis within its labs, in order to make sure there is enough data that would entice its users. (Blend Shape is also used to create its Disney AR Emojis, like Mickey and Minnie Mouse.)

In our own AR Emoji test, a majority of us felt our characters looked extremely inaccurate. One writer felt there was a lack of skin-tone options available, with no dark-skinned tones to choose from. Others felt their AR Emoji didn’t look anything like them, which made editing the avatar seem like more of a chore than an enjoyable task.

According to Loom.ai, creating an accurate AR Emoji relies heavily on the way the selfie is shot. Users should go about taking the photo the same way they’d take a passport photo. “You want even lighting, nothing covering your face, and a neutral expression so that we can add the expressions later,” Ramasubramanian explained. “It’s a matter of people getting a little accustomed to it and then seeing the possibilities of animating it and using it.”

Ramasubramanian also said that users should treat it the way they would while taking any selfie — meaning, you can take tons of photos until you’re satisfied with the photo. Since AR Emoji allows you to create multiple avatars, you can compare them to find the one you like best. Each one you create also saves to your phone, so you can switch up the version you use depending on your mood.

The latest update to AR Emoji includes additional hairstyles, headwear such as a hijab, and other options. The update is currently rolling out to unlocked versions of the Samsung Galaxy S9 and S9 Plus in Korea, China, and Europe. European carrier devices and U.S. unlocked devices should have the update available for download this week. The update for U.S. carrier devices will be rolling out over the coming weeks. Samsung says AR Emoji will continue to evolve in the future, but for now, is staying tight lipped on exact details.

Product Review

Mediocre battery and a big notch slight Google's otherwise perfect Pixel phone

Google’s Pixel 3 XL has two big flaws: The gigantic notch on the front, and mediocre battery life. That being said, this is the best Android experience you can find in a smartphone today.
Mobile

Samsung will reportedly announce its folding smartphone at MWC in February

Samsung has been showcasing bendable display tech for a few years and now a folding smartphone might finally arrive. The Galaxy X, or perhaps the Galaxy F, may be the company's first example. Here's everything we know about it.
Mobile

Upcoming Galaxy S10 may support 8K video and have Neural Processing Unit

While we still may be months away from an announcement, there's no doubt about it: Samsung is working hard on its successor to the Galaxy S9. Here's everything we know about the upcoming Samsung Galaxy S10.
Gaming

‘Diablo Immortal’ is just the beginning. Mobile games are the future

Diablo fans were furious about Diablo Immortal, but in truth, mobile games are the future. From Apple and Samsung to Bethesda and Blizzard, we’re seeing a new incentive for games that fit on your phone.
Cars

Ares turns the Tesla Model S into a two-door roadster with Italian flair

Italian coachbuilder Ares has done what Tesla won't. It's taking a Model S, chopping off the roof, and sending the rear doors back to the parts bin to create a roadster that's ready for the French Riviera.
Music

Here's our head-to-head comparison of Pandora and Spotify

Which music streaming platform is best for you? We pit Spotify versus Pandora, two mighty streaming services with on-demand music and massive catalogs, comparing every facet of the two services to help you decide which is best.
Mobile

Netflix is testing a cheap mobile-only subscription tier in some markets

Netflix is trying to reach potential subscribers in emerging markets. To that end, the company has begun rolling out a cheaper, mobile-only subscription tier, which comes at around $4 per month -- half the price of the Basic subscription…
Mobile

Check out 30 of the best iPhone games you need to be playing

The iPhone has some of our favorite games available for any mobile platform. Here are the best iPhone games for every big-name genre, whether you're into puzzles, strategy, or something else entirely.
Mobile

Text messages are disappearing on Pixel 3s, but a fix is on the way

The Google Pixel 3 and Pixel 3 XL are arguably the best Android phones out there right now -- but they aren't perfect. Users have reported a number of issues and problems related to the Google Pixel 3 and Pixel 3 XL. Here's how to fix them.
Mobile

Get ready for more V-series phones: LG trademarks names up to the V90

It looks like we should get ready for more LG V-series phones. LG has trademarked names for the LG V50 all the way up to the LG V90. We may have to wait some time before we see these phones; the next LG flagship is expected to be the LG G8.
Home Theater

What is MHL, exactly, and how does it work with your TV?

There are more ways to mirror your smartphone or tablet to your TV than you might think. Check out our rundown of MHL for everything you need to know about the wired protocol and its myriad uses.
Deals

All the best Amazon Black Friday deals for 2018

Amazon may be an online-only retailer, but that doesn’t mean its Black Friday sales are anything to sniff at. In fact, due to its online status, Amazon has huge flexibility with the range of products and deals it can offer. Here's our…
Deals

Black Friday 2018: The best deals so far

Black Friday is the biggest shopping holiday of the year, and it will be here before you know it. If you can't wait until November 23 to start formulating a shopping plan, we've got you covered.
Social Media

Facebook is rolling out a Messenger ‘unsend’ feature and here’s how to use it

Facebook is starting to roll out a "remove message" feature for its Messenger app. It lets you delete a message in a thread within 10 minutes of sending it, and replaces it with a note telling recipients that it's been removed.