Facebook touts groundbreaking app AI that powers artistic Live video filters

facebook prisma filters fb style transfer
Facebook is betting on video filters (the graphics pioneered by Snapchat that can be overlaid on faces in recorded clips) to get more users hooked on its live-streaming tool. Having already demoed its upcoming style transfer graphics (artistic filters that draw from the style of famous painters to transform videos into works of art), it is now offering more details about the artificial intelligence powering its new mobile tools.

Instead of Snapchat, the inspiration for these Live video filters comes from Prisma — an AI-powered photography app that boasts a range of artistic graphics that can be applied to both images and video. Facebook is currently testing style transfers as part of an update that gives added prominence to your smartphone camera within its flagship app in Ireland.

Unlike Snapchat (the app that pioneered filters or, as it calls them, “lenses”), Facebook wants to apply the software to live broadcasts in real time and not after you’ve captured a clip. It’s already started doing this with some of its traditional, Snapchat-style filters, but style transfers are — according to Facebook — its crowning achievement.

On Tuesday, the company published a new blog post explaining in-depth how the style transfer technique works in real-time. Instead of waiting for the image to be processed on huge computer servers in data centers, Facebook claims it has developed a world-first: a deep-learning platform on mobile that puts the cutting-edge technology in the palm of a user’s hand.

Facebook is essentially embedding its neural network (a machine learning system, loosely modelled on the human brain, that learns by consuming vast tons of data) into its Facebook app for iOS and Android. The company terms this deep learning AI “Caffe2Go,” describing it as a lightweight and modular framework built on top of the open-source “Caffe2” project — meaning it won’t drain your mobile device. Facebook is also optimizing style transfer tech, allowing it to be applied to video running at 20 frames per second on an iPhone 6S or above without a drop in picture quality.


“The style-transfer tool in the camera is the result of a marriage between two technologies: the Caffe2go runtime and style-transfer models,” write Facebook research scientists Yangqing Jia and Peter Vajda in their blog post. “It took both technologies to make it possible for you to feel like you have Van Gogh’s paintbrush in your hand when you pick up your phone to shoot a video.”

In a second blog post, Facebook CTO Mike Schroepfer details the company’s vision for the mobile AI: “We can create gesture-based controls, where the computer can see where you’re pointing and activate different styles or commands. We can recognize facial expressions and perform related actions, like putting a “yay” filter over your selfie when you smile.”

Facebook CEO Mark Zuckerberg demoed the style transfer feature in October, shortly after which Facebook launched its first Live video filters.

Facebook has not shied away from boasting about its plans for the live-streaming format, and video in general. Last month, the company’s chief product officer Chris CoxCox claimed it now predicts video will generate 70 percent of traffic across Facebook’s properties — such as Instagram, WhatsApp, and its flagship social network — in the next five years. Coincidentally, WhatsApp launched video calling for Android devices in late October — indicating that Facebook is wasting no time in making that future a reality.

Updated on 11-08-2016 by Saqib Shah: Added news of Facebook’s reveal of AI for mobile app


Latest Facebook bug exposed up to 6.8 million users’ private photos

An API bug recently left an impact on Facebook users. Though the issue has since been fixed, some of the apps on the platform had a wrongful access to consumers photos for 12 days between September 13 and September 25. 
Digital Trends Live

Digital Trends Live: Passenger spacecraft, Facebook data breach, and more

This episode of Digital Trends Live saw the fourth launch of passenger spacecraft SpaceShipTwo, and a Facebook data breach. We were also joined by the founder of Forter, Michael Reitblat, to discuss how to keep your data safe.
Smart Home

Facebook Portal and Portal+ video-calling devices gain new content and features

Facebook's Portal devices are video smart speakers with Alexa voice assistants built in that allow you to make calls. The 15-inch Portal+ model features a pivoting camera that follows you around the room as you speak.
Social Media

Walkie-talkie voice messaging finally comes to Instagram

In its latest grab from messaging apps, Instagram now lets you send walkie-talkie style voice messages. Apps such as Facebook Messenger, WhatsApp, Snapchat, and iMessage have offered the feature for some time.
Social Media

This event topped Facebook’s biggest moments of the year — again

As the year comes to a close, Facebook is looking back on what users discussed most over the last year. For two years in a row, International Women's Day topped the list. So what else is on the list?
Social Media

This band owns Twitter, according to list of top accounts and tweets for 2018

What was the biggest buzz on Twitter in 2018? Twitter's 2018 Year in Review highlights the biggest tweets, accounts, and hashtags. The most-tweeted celebrities, movies, TV shows, athletes, politicians and more in Twitter's 2018 trends.
Social Media

What do yodeling and Kylie Jenner have in common? YouTube’s top 2018 videos

In a true nod to the variety found on YouTube, the platform's top 10 list of videos from 2018 range from celebrities to sports, from perfectly tossing a picture frame on the wall to a kid yodeling in aisle 12 at Walmart.
Home Theater

It took Tom Cruise to raise awareness of this troublesome TV setting

Tom Cruise, in an unexpected PSA tweet, asks you to turn off motion interpolation on your TV, but stops short of how to do it. Here's more on the topic, along with links to a guide on how to rid your TV of the dreaded "soap opera effect."

Make a GIF of your favorite YouTube video with these great tools

Making a GIF from a YouTube video is easier today than ever, but choosing the right tool for the job isn't always so simple. In this guide, we'll teach you how to make a GIF from a YouTube video with our two favorite online tools.

Amazon scouted airport locations for its cashier-free Amazon Go stores

Representatives of Amazon Go checkout-free retail stores connected with officials at Los Angeles and San Jose airports in June to discuss the possibility of cashier-free grab-and-go locations in busy terminals.
Social Media

Snapchat facial recognition could soon power a new portrait mode, code suggests

Digging into Snapchat's code suggests a handful of upcoming camera features, including a portrait mode. The feature appears to use facial recognition A.I. to blur the background. The code also suggests an updated camera interface.

Google+ continues to sink with a second massive data breach. Abandon ship now

Google+ was scheduled to shut its doors in August 2019, but the second security breach in only a few months has caused the company to move its plan forward a few months. It might be a good idea to delete your account sooner than later.
Social Media

‘YouTube Rewind 2018’ is about to become its most disliked video ever

YouTube is about to achieve a record it really doesn't want — that of "most-disliked video." Yes, its annual recap of featuring popular YouTubers has gone down really badly this year.
Social Media

GIF almost anywhere with Giphy’s new keyboard and sticker maker

We all love GIFs, but not every app supports them. Fortunately, the new Giphy' keyboard brings GIFs to any iOS app that supports multimedia. The update also comes with a new tool for creating animated stickers.