'Stylized Facial Animation' project turns video into realistic-looking hand-drawn art

In November of 2016, we spoke with Adobe about Project StyLit, a research project that used artificial intelligence to apply an artistic style to a three-dimensional model. It was an intriguing technology, but it had many limitations and an apparently limited usefulness. Building off of what the team learned with Project StyLit, Adobe Research has now taken a big step forward with a new project that, for now, is known simply as “Stylized Facial Animation.” It can apply an input style (or “style exemplar”) to video with very natural results, doing a much better job of imitating hand-drawn animation than other techniques. While it is currently limited to faces, it shows promise for how animators will one day be able to automate the rotoscoping process, or at least aspects of it.

The technology will be publicly demonstrated on August 3 at SIGGRAPH, but Digital Trends had the chance to speak with Adobe’s David Simons about the project ahead of the presentation. Simons holds the role of Senior Principal Scientist within Adobe’s Advanced Product Development Group and is a 25-year veteran of After Effects, Adobe’s special effects program for video. After Effects seems a likely candidate for where the technology demonstrated in Stylized Facial Animation could one day end up. At this time, Adobe has not announced any plans for a commercial release of the tech, but it is the mission of Simons and the Advanced Product Development Group to turn research projects into marketable products, so it may not be that far off from finding its way into a future Creative Cloud update (it’s highly likely that it will be demonstrated at Adobe’s upcoming MAX conference).

Simons opened by explaining some key differences between Project StyLit and Stylized Facial Animation. StyLit relied on the information channels available in 3D models in order to appropriately map the style to the model, and styles were based off of simple “sphere study” sketches that clearly laid out tonality and color. “The idea was to make similar stylized synthesis, driven by your choice of style, but be able to apply it to a photograph or video,” Simons told Digital Trends.

In this case, rather than relying on sketches of spheres, the style exemplars are portraits of actual faces, and the qualities of the style — textures, lines, brush strokes — are appropriately redrawn in the shape of the subject’s face in the target video. In the resulting output animation, the human subject is clearly recognizable, yet appears to be hand drawn in the technique used in the style exemplar, even when the exemplar face is a different shape, age, or gender from the person in the video.

Neural networks to the rescue

As detailed in a research paper, Stylized Facial Animation is based on previous research on neural network computing, conducted by both Adobe and others. One of the earliest and most popular consumer applications of neural network technology was the Prisma app, which applied an artistic style to an image and, later, to video. An important distinction between the likes of Prisma and simple image filters is that apps based on neural networks actually redraw the image from scratch, rather than overlay the style on top of the photograph. But while Prisma could do this very quickly, it did a relatively poor job of maintaining details present in the source style.

While it still struggles in some specific areas, like forehead wrinkles, Adobe’s approach significantly increases the amount of artistic detail that makes it through to the final output. It accomplishes this by analyzing both the style exemplar and the target video, creating “guiding channels” that ensure the style is appropriately applied to the target. One channel is used to align the position of the exemplar and the target, another maintains the shading present in the target subject, and a third matches semantically similar regions (eyes to eyes, hair to hair, etc.).

The underlying technology is actually based on the Content-Aware Fill feature in Adobe Photoshop. “It looks at nearby areas and takes patches that fit together. It’s like trying to fill in a jigsaw puzzle with pieces from the rest of the area that match up,” Simons said.

The result is, essentially, just how you would expect such a technology to work, but it is considerably different from what we’ve seen in apps like Prisma as it maintains so much more detail from the source style. But when it comes to applying the effect to multiple frames for an animation, things get even trickier. With Project StyLit, animations could only be created by applying a style one frame at a time, then manually connecting the frames in Adobe Premiere Pro or another video-editing application.

“A lot of it is about avoiding the uncanny valley. You want a level of abstraction.”

One of the key components of Stylized Facial Animation is that it can apply a style to an entire video at once, but this can lead to the issue of temporal coherence. This is somewhat ironic, as temporal coherence is typically what such programs aim for — that is to say, where each frame logically matches up with the previous frame, leading to smooth motion — but this actually has the effect of making the resulting animation look less like it was done by hand. Hand drawn animations have a natural amount of decoherence between frames, as the pencil lines or brush strokes won’t perfectly match from one frame to the next. Existing methods inherently achieved very high temporal coherence, so Adobe researchers knew they had to develop their own way to reduce coherence in Stylized Facial Animation in order to get more natural results.

“We want it to look redrawn on every frame,” Simons explained, “But not so flickery that you can’t see what’s going on. A lot of it is about avoiding the uncanny valley. You want a level of abstraction.”

The solution came in allowing control over “temporal noise” by using a separate guiding channel to determine coherence. As each frame is synthesized from the preceding one, controlling the amount of blur in a frame’s temporal guiding channel will raise or lower coherence in the next frame, making it possible to find a balance between an animation that is too choppy and too smooth.

Applying the technology

While Simons was quick to reaffirm that the project is definitely “still research at this time,” there are two possible applications for it in the future. One would be for animating live-action footage, as demonstrated in the research paper, and the other would be for creating puppets to use in Adobe’s Character Animator, which automatically animates a character based on video of an actor.

Any portrait — a sketch, painting, or even a photograph — can be used to style the video.

Compared to Project StyLit, another advantage with Stylized Facial Animation is that any portrait can be used as a style exemplar. This doesn’t just give artists greater control over creating their own styles compared to a basic sphere study, but it also means non-artists can grab source images from virtually anywhere and use them as style exemplars. The sources need not even be drawings or paintings; photographs (even of statues) will also work.

stylized facial animation siggraph 2017 animations

Going forward, one of the main goals will be getting to the point where the program can run in real time, or at least close to it. Currently, it takes about 30 seconds just to compute the guiding channels and three minutes to perform full synthesis of a single, 1-megapixel frame using a quad-core processor running at 3 gigahertz, according to the research paper. However, the team has already demonstrated vast performance improvements possible by using a graphics processing unit (GPU), which reduces the time required for single-frame synthesis down to just 5 seconds. Given that most video runs at 24 or 30 frames per second, this is still a ways off from achieving real time synthesis, but the project is also still in very early development.

Whatever the eventual outcome of Stylized Facial Animation, it is clear that Adobe is very interested in continuing to develop the technology. The paper’s lead author, Jakub Fiser, started out as an intern at Adobe. This project was his Ph.D. thesis, and was enough for Adobe to bring him on full time. Fiser joins an experienced group of individuals, of whom Simons spoke highly, even while keeping modest with regard to his own role. “Everyone else is a Ph.D.,” he said. “They’re the real researchers.”

Emerging Tech

Awesome Tech You Can’t Buy Yet: Laptop screen extenders and self-healing tents

Check out our roundup of the best new crowdfunding projects and product announcements that hit the Web this week. You can't buy this stuff yet, but it sure is fun to gawk!
Movies & TV

The best shows on Netflix in July, from ‘Arrested Development’ to ‘Mad Men’

Looking for a new show to binge? Lucky for you, we've curated a list of the best shows on Netflix, whether you're a fan of outlandish anime, dramatic period pieces, or shows that leave you questioning what lies beyond.
Movies & TV

It was a shaky box-office weekend for ‘Ant-Man and the Wasp’ and ‘Skyscraper’

Marvel's 'Ant-Man and the Wasp' and Dwayne Johnson's 'Skyscraper' both had shaky weekends at the box office for different reasons, but animated sequel 'Hotel Transylvania 3' won the weekend easily.
Mobile

To celebrate World Emoji Day, Apple debuts over 70 new emojis coming to iOS 12

In honor of "World Emoji Day," Apple unveiled more than 70 new emojis that are coming iOS 12 later this year. From different hairstyles to exotic animals, the company shared a first look at what we can expect to see on our keyboards.
Photography

Is full Photoshop coming to iPad?

Will an upcoming version of Photoshop be as mobile-friendly as Lightroom CC? A report from anonymous company insiders says that Adobe is working on a full version of Photoshop for the iPad.
Home Theater

Looking to cut cable? Here’s everything you need to know about Pluto TV

Pluto TV offers plenty of entertainment in a fashion similar to live internet TV services, only at no cost — you don’t even need to register. Too good to be true? Here’s everything you need to know.
Movies & TV

15 epic sci-fi novels you should read before they become blockbuster films

You can get ahead of the next crop of science-fiction movies coming out of Hollywood by picking up the books that inspired them. We compiled a list of books you can add to your reading list now to get a glimpse of the future.
Photography

These point-and-shoot cameras make your smartphone pics look like cave paintings

If your smartphone camera just isn't giving you the results you're looking for, maybe it's time to step up your game. The latest and greatest point-and-shoot cameras offer large sensors, tough bodies, and long lenses - something no phone…
Mobile

We tried all the latest and greatest smartphones to find the best of 2018

Smartphones are perhaps the most important and personal piece of tech on the planet. That’s why it’s important to pick the best phone for your individual needs. Here are the best smartphones you can buy.
Home Theater

The best Dolby Atmos movies for your home theater sound as good as they look

If you've got your hands on some sweet Dolby Atmos gear, the next step is to find films that take advantage of it. These are our picks in every genre for the best Dolby Atmos movies currently available on Blu-ray and streaming services.
Movies & TV

The best movies on Netflix in July 2018, from ‘Coco’ to ‘Jurassic Park’

Save yourself from hours wasted scrolling through Netflix's massive library by checking out our picks for the streamer's best movies available right now, whether you're into explosive action, subdued humor, or anything in between.
Gaming

The best Xbox One games

More than four years into its lifespan, Microsoft's latest console is finally coming into its own. From 'Cuphead' to 'Middle-Earth: Shadow of War,' the best Xbox One games offer something for everyone.
Photography

Fujifilm announces ultra-wide 8-16mm f/2.8 and massive 200mm f/2 lenses

Two new premium lenses have joined Fujifilm's X Series, an ultra-wide zoom for landscapes and astrophotography and a fast telephoto prime for low-light sports and wildlife. At $2,000 and $6,000, both are decidedly made for professionals.
Mobile

These are the best portable chargers for quickly juicing your device

Battery life still tops the polls when it comes to smartphone concerns. If it’s bugging you, then maybe it’s time to snag yourself a portable charger. Here are our picks of the best portable chargers.
Photography

What does this button do? A quick guide to understanding your camera’s controls

Most DSLR cameras are user-friendly, but that doesn't mean you can make sense of every button on your own. Thankfully, our quick-hit guide will help you better acquaint yourself with your camera's operation.
Photography

With flip-out lenses, the Vuze XR transforms from 360- to 180-degree VR camera

The Vuze XR is a compact, dual-lens camera with an integrated handle and a neat party trick: The back-to-back lenses can flip forward to transition from 360-degree two-dimensional video, to 180-degree three-dimensional video.
Mobile

Here’s how to turn off camera shutter sound on your Android phone

That clicking shutter sound on your Android phone can get annoying if you like to take lots of pictures. Fortunately, you can disable it. We will walk you through how to turn off the camera shutter sound on your Android phone.
Photography

Ditch the smartphone, this is the camera you want in your pocket

Fujifilm's XF10 offers premium image quality in a portable form factor that may just be enough to get smartphone shooters to pick up a real camera. It's also Fujifilm's cheapest premium compact yet, at $500.
Photography

Brother’s new INKvestment Tank printer is made for people who hate buying ink cartridges

With a new ink cartridge design, Brother's new INKvestment Tank printer can run for up to a year before the cartridges need replacing -- ideal for users who want convenience and cost-savings.
Computing

Here are 5 free alternatives to Photoshop for all your photo-editing needs

Photoshop is a capable program, but it's also expensive. Lucky for you, there are plenty of great alternatives out there that allow for a range of versatility, without requiring you to break into your bank account.
Mobile

These might be the best photos shot with an iPhone in 2018

The iPhone Photography Awards (IPPAWARDS) announced its official winners of its 11th annual global competition. Each one was selected from thousands of entries spanning over 140 countries around the world.