ModiFace replaces makeup brushes with neural networks, and it’s coming to the S9

Modiface AR Makeup on the Samsung Galaxy S9
Brenda Stolyar/Digital Trends
Brenda Stolyar/Digital Trends

Samsung’s Galaxy S9 and S9 Plus come with a variety of new camera features including AR Emoji, Super Slow Motion, and a variable aperture. But there’s one more new trick that we’ve never seen built into a smartphone before – the augmented reality makeup tool in Bixby Vision.

Baked into the S9’s camera app, Makeup lets you apply different products to your face with augmented reality. Point the selfie camera at your face, and the makeup styles you pick are layered over your face, exactly like Snapchat filters. The benefit is you’re trying on makeup in the comfort of your own home, while also avoiding the need to clean up afterwards. It’s the first time we’ve seen this kind of feature built into a smartphone, but the technology is far from new. Samsung tapped ModiFace, a company with more than a decade of research in this field, to integrate its beauty AR technology into the camera.

From skincare to augmented reality makeup

If you’ve ever used the Sephora app to virtually try-on makeup, or Benefit’s Brow Try-On app to test out a new eyebrow shape, then you’ve used ModiFace’s AR technology. The company now powers more than 200 custom augmented reality apps for high-end beauty brands, but it all initially started with skincare.

Smartphone popularity inspired ModiFace to test its technology with augmented reality.

Before the smartphone revolution, ModiFace worked with dermatologists and certain brands to help people find solutions for skincare concerns.

People could upload a photo of themselves to a web app, and the company would process it to pinpoint problem areas and suggest improvements to skincare routines.

ModiFace originally used 2D images with the web app, but as technology progressed, the company was soon able to move to allowing users to upload video. When smartphones got popular, ModiFace began testing its technology with augmented reality, and added makeup and hair to its roster of supported features.

“With the smartphone boom, it was the perfect opportunity for us to expand because it allowed us to bring AR right to our users in the palm of [their] hand,” Jeff Houghton, ModiFace vice president of Technology, told Digital Trends.

Its current software development kits (SDKs) are the culmination of 10 years of engineering. By working closely with beauty brands, ModiFace is able to provide a lightning fast and easy-to-follow user experience, while also encouraging product discoverability.

ModiFace AR powered mirror.
ModiFace's technology powers more than 200 augmented reality apps for high-end beauty brands, including numerous smart mirrors, that allow customers virtually try on makeup in real-time. Photo Credit: ModiFace.
ModiFace’s technology powers more than 200 augmented reality apps for high-end beauty brands, including numerous smart mirrors, that allow customers virtually try on makeup in real-time. Photo Credit: ModiFace.

Its technology doesn’t stop at smartphone apps, extending to retail stores as well. Back in November, MAC Cosmetics debuted its MAC Virtual Try-On Mirror at certain locations. The mirror lets customers virtually try on makeup in real-time by swiping throughout the interface. It helps eliminate the need to test a lot of products at the store, narrowing it down to the styles you like the most.

Putting its tech on the Galaxy S9

The same technology and concept is what’s available on the new Galaxy S9. By collaborating with Samsung, ModiFace was able to optimize the experience.

The partnership
could also kick start a
new trend for phone manufacturers.

“Starting in house, we train our Neural Networks on thousands of images to create the base tracking and face analysis for our apps,” Houghton said. “This Neural Network is then embedded inside Samsung’s app. We worked with Samsung to tweak several parameters to make sure we were achieving the effects that brands and end users want to see.”

The partnership between the two companies could kick start a new trend for phone manufacturers. While beauty technology is still in its infancy, makeup has found its way on to our screens for years through social media. Whether it’s YouTube or Instagram tutorials, people are constantly looking to their smartphones to find new inspiration when it comes to makeup. With the Galaxy S9, all you need to do is swipe open the camera and you have a catalog of makeup at your disposal.

On the Galaxy S9, you can search through a variety of different cosmetic products available from both Sephora or Cover Girl, and you can purchase them on the spot. As you scroll through each product, it will apply itself to your face like Snapchat filters.

Using facial tracking and 3D video rendering, the makeup filters are mapped to the face at 30 frames per second. There’s no lag, and nothing is misaligned. It works as instantly as you to tap the next product you want to try on. You can try on complete looks – which includes lipstick, foundation, eyeshadow, blush, mascara and eyeliner – or you can try them all separately.

Trying on foundation can be extremely tricky, almost always requires being at the store and having an employee help you find your exact shade. There’s also different foundation types to take into account – matte, sheer, water-based, or ones that provide full coverage. To make sure the shades are as accurate as possible, the ModiFace team tediously went through each product and compared how it looked in real life versus how it looked on the smartphone. They worked with both Samsung and the brands directly to create a matching render.

“I myself was actually involved with that process and it’s kind of a lot of fun,” Houghton said. “You get to try on a lot of different things, and then you get to wipe them off and see how they look in the app. It involves a lot of screenshot comparison and manual work to make sure that the product we’ve included is just perfect.”

The team went through each product and compared how it looked in real life
versus how it looked
on the smartphone.

But there’s still always room for improvement. Using deep learning and image research, the team studies internal data — gathered from those who are testing the final product — to learn more about a user’s face in order to train its algorithms. This helps to produce better results when it comes to factors like how colors mix on to your lip, how light gets added into those colors, and how your face should look under different lighting if you have a specific foundation on. The data also helps to improve texture and overall coverage of the makeup.

With this kind of deep learning, the company is able to create new and more realistic effects. Recently, ModiFace partnered with L’Oreal to create 3D hair tracking that can recolor your hair in real time. The company has plans to work with Samsung to bring similar effects to its phones.

But for those with privacy concerns, Modiface doesn’t collect or monitor any data from its applications when in the hands of users. Any images you take with the apps will never leave your device.

Gaming

‘Diablo Immortal’ is just the beginning. Mobile games are the future

Diablo fans were furious about Diablo Immortal, but in truth, mobile games are the future. From Apple and Samsung to Bethesda and Blizzard, we’re seeing a new incentive for games that fit on your phone.
Mobile

Samsung patents show what Infinity-O display could look like on Galaxy S10

While we still may be months away from an announcement, there's no doubt about it: Samsung is working hard on its successor to the Galaxy S9. Here's everything we know about the upcoming Samsung Galaxy S10.
Mobile

Samsung will reportedly announce its folding smartphone at MWC in February

Samsung has been showcasing bendable display tech for a few years and now a folding smartphone might finally arrive. The Galaxy X, or perhaps the Galaxy F, may be the company's first example. Here's everything we know about it.
Mobile

Samsung made a $2,700 flip phone with the brains of a smartphone

The Samsung W2019 is a high-end flip phone that comes with flagship specifications such as 6GB of RAM and Qualcomm's Snapdragon 845. The device features a pair of S-AMOLED displays, a dual rear camera setup, and an eye-popping price tag.
Mobile

You can now message businesses straight through Google Maps

Google has been updating Maps with a ton of new features over the past few months, and now it's back with another one -- the ability for users to message businesses directly through the Maps app.
Mobile

No cash. No talking. What goes next? Welcome to your ‘app-tive’ digital life

Bank of America's 2018 Trends in Consumer Mobility Report found mobility has changed our lives. Mobile tech influences how most of us communicate, meet people, build relationships, and handle money as we move toward a cashless society.
Mobile

The Motiv smart ring is coming to 20 more countries and physical stores

Remember Motiv's activity tracking smart ring? It's back with a raft of new features that adds biometric identification and token authentication, all on a device that fits on your finger.
Wearables

Everything you need to know about Garmin’s GPS watches and trackers

Garmin jumped into the GPS smartwatch and fitness tracker market five years ago and has built a portfolio of devices that rivals competitor Fitbit. Here's your guide to the latest and greatest fitness devices that Garmin has to offer.
Mobile

Sharp doubles down on the notch trend with Aquos R2 Compact

As if one notch wasn't enough, Japanese manufacturer Sharp unveiled a new smartphone that has two -- one teardrop style notch at the top, and a bigger notch at the bottom. Here's what the world's first dual-notch smartphone looks like.
Mobile

Verizon has made its first 5G video call … with a phone that’s already out

Verizon has announced that it has successfully made its first video call, using a smartphone that's already available: The Motorola Moto Z3. To make the call, Verizon used the 5G Moto Mod.
Home Theater

Google Chromecast and Chromecast Ultra: Everything you need to know

Google's Chromecast plugs into your TV's HDMI port, allowing you to stream content from your tablet, laptop, or smartphone directly to your TV. Here's what you need to know about all iterations, including the 4K-ready Chromecast Ultra.
Mobile

The best protective iPhone cases to defend against dirt, dings, and drops

If you’re going off-road or work outdoors, it could be a good idea to invest in a tough case. These are our picks of the best protective iPhone cases for all iterations of the iPhone, from the iPhone XS to the 7.
Social Media

Build a wish list and shop videos with Instagram’s latest shopping update

Eyeing a product on Instagram? Now there are more ways to shop from the social network. Instagram just rolled out options to save products in a collection as users can also now shop from videos.
Product Review

With style and feature upgrades, Misfit's next-generation Vapor 2 gets it right

Misfit’s next-generation smartwatch, the Vapor 2, packs built-in GPS, a heart-rate sensor, and more, into a beautiful design that starts from $250. We take a closer look at the company's latest device.