Facebook touts groundbreaking app AI that powers artistic Live video filters

facebook prisma filters fb style transfer
Facebook is betting on video filters (the graphics pioneered by Snapchat that can be overlaid on faces in recorded clips) to get more users hooked on its live-streaming tool. Having already demoed its upcoming style transfer graphics (artistic filters that draw from the style of famous painters to transform videos into works of art), it is now offering more details about the artificial intelligence powering its new mobile tools.

Instead of Snapchat, the inspiration for these Live video filters comes from Prisma — an AI-powered photography app that boasts a range of artistic graphics that can be applied to both images and video. Facebook is currently testing style transfers as part of an update that gives added prominence to your smartphone camera within its flagship app in Ireland.

Unlike Snapchat (the app that pioneered filters or, as it calls them, “lenses”), Facebook wants to apply the software to live broadcasts in real time and not after you’ve captured a clip. It’s already started doing this with some of its traditional, Snapchat-style filters, but style transfers are — according to Facebook — its crowning achievement.

On Tuesday, the company published a new blog post explaining in-depth how the style transfer technique works in real-time. Instead of waiting for the image to be processed on huge computer servers in data centers, Facebook claims it has developed a world-first: a deep-learning platform on mobile that puts the cutting-edge technology in the palm of a user’s hand.

Facebook is essentially embedding its neural network (a machine learning system, loosely modelled on the human brain, that learns by consuming vast tons of data) into its Facebook app for iOS and Android. The company terms this deep learning AI “Caffe2Go,” describing it as a lightweight and modular framework built on top of the open-source “Caffe2” project — meaning it won’t drain your mobile device. Facebook is also optimizing style transfer tech, allowing it to be applied to video running at 20 frames per second on an iPhone 6S or above without a drop in picture quality.


“The style-transfer tool in the camera is the result of a marriage between two technologies: the Caffe2go runtime and style-transfer models,” write Facebook research scientists Yangqing Jia and Peter Vajda in their blog post. “It took both technologies to make it possible for you to feel like you have Van Gogh’s paintbrush in your hand when you pick up your phone to shoot a video.”

In a second blog post, Facebook CTO Mike Schroepfer details the company’s vision for the mobile AI: “We can create gesture-based controls, where the computer can see where you’re pointing and activate different styles or commands. We can recognize facial expressions and perform related actions, like putting a “yay” filter over your selfie when you smile.”

Facebook CEO Mark Zuckerberg demoed the style transfer feature in October, shortly after which Facebook launched its first Live video filters.

Facebook has not shied away from boasting about its plans for the live-streaming format, and video in general. Last month, the company’s chief product officer Chris CoxCox claimed it now predicts video will generate 70 percent of traffic across Facebook’s properties — such as Instagram, WhatsApp, and its flagship social network — in the next five years. Coincidentally, WhatsApp launched video calling for Android devices in late October — indicating that Facebook is wasting no time in making that future a reality.

Updated on 11-08-2016 by Saqib Shah: Added news of Facebook’s reveal of AI for mobile app

Emerging Tech

Here’s how Facebook taught its Portal A.I. to think like a Hollywood filmmaker

When Facebook introduced its Portal screen-enhanced smart speakers, it wanted to find a way to make video chat as intimate as sitting down for a conversation with a friend. Here's how it did it.

Photography news: Wacom’s slimmer pen, Leica’s cinema special edition

In this week's photography news, Wacom launches a new slimmer pen for pro users. Leica's upcoming M10-P is designed for cinema, inside and out, with built-in cinema modes in the updated software.
Product Review

Ring Video Doorbell 2 is the simplest entry into a smarter doorway

The Ring Video Doorbell 2 may lack the style and sophistication of premium door-dingers, but few can match its simplicity and versatility. The device, available in both wired and wireless configurations, is easy to set up and adds instant…
Social Media

Instagram test reveals direct messages may be coming to browsers

Instagram for the web has always been a minimalist affair compared to the feature-rich smartphone app, but in the last few years that's started to change. The latest news is that Instagram is considering adding direct messages.
Social Media

Periscope tool adds guests to feeds so streamers can become talk show hosts

Periscope users can now invite viewers to chime into the conversation with more than just the comment tool. By enabling the option to add guests, livestreamers can add guests to the conversation, in audio format only.

Crouching, climbing, and creeping, the perfect Instagram shot knows no bounds

Just how far will you go for the perfect Instagram? A recent survey shows just how willing Instagram users -- and Instagram husbands -- are to climb, lie down, embarrass themselves or let their food go cold for the perfect shot.
Social Media

Facebook’s long-promised ‘unsend’ feature arrives. Here’s how to use it

Send a message to the wrong person? Messenger now gives you 10 minutes to take it back. After an update beginning to roll out today, users can now retract messages if they act within the first 10 minutes after sending the message.
Social Media

YouTube boss admits even her own kids gave the ‘Rewind’ video a thumbs down

YouTube's 2018 Rewind video went down like a lead balloon at the end of last year, becoming the most disliked video in its history. And now YouTube's CEO has admitted that even her own kids thought it was pretty darn awful.
Social Media

Snapchat finally recovers from its redesign — so here comes an Android update

Snapchat's drop in users after launching a controversial redesign has finally stagnated. During the fourth quarter and 2018 earnings report, Snapchat shared that the company is rolling out an Android update designed to increase performance.
Social Media

Skype’s new ‘blur background’ feature could help keep you from blushing

Skype's latest feature for desktop lets you blur your background during video calls. The idea is that it keeps you as the focus instead of distracting others with whatever embarrassing things you might have on show behind you.
Social Media

Twitter users are declining but more people are seeing ads every day

Twitter's end-of-the-year report for 2018 is a mix of good and bad news. The good news is that more users are seeing adds daily, the metric the company will focus on moving forward. But the bad news is that monthly active users are…

Switch up your Reddit routine with these interesting, inspiring, and zany subs

So you've just joined the wonderful world of Reddit and want to explore it. With so many subreddits, however, navigating the "front page of the internet" can be daunting. Here are some of the best subreddits to get you started.

YouTube beats Apple, Netflix as the most trusted brand by millennials

The popular video sharing website YouTube climbed up in an annual Mblm study, moving up from third place in 2018 and coming ahead of both Apple and Netflix in final 2019 rankings. 
Social Media

LinkedIn finally gets around to launching its own live video tool

Live video is coming to LinkedIn for businesses and individuals on the site. The livestreaming feature is launching in beta in the U.S. before rolling out to the entire community.