Facebook touts groundbreaking app AI that powers artistic Live video filters

Facebook is betting on video filters (the graphics pioneered by Snapchat that can be overlaid on faces in recorded clips) to get more users hooked on its live-streaming tool. Having already demoed its upcoming style transfer graphics (artistic filters that draw from the style of famous painters to transform videos into works of art), it is now offering more details about the artificial intelligence powering its new mobile tools.

Instead of Snapchat, the inspiration for these Live video filters comes from Prisma — an AI-powered photography app that boasts a range of artistic graphics that can be applied to both images and video. Facebook is currently testing style transfers as part of an update that gives added prominence to your smartphone camera within its flagship app in Ireland.

Unlike Snapchat (the app that pioneered filters or, as it calls them, “lenses”), Facebook wants to apply the software to live broadcasts in real time and not after you’ve captured a clip. It’s already started doing this with some of its traditional, Snapchat-style filters, but style transfers are — according to Facebook — its crowning achievement.

On Tuesday, the company published a new blog post explaining in-depth how the style transfer technique works in real-time. Instead of waiting for the image to be processed on huge computer servers in data centers, Facebook claims it has developed a world-first: a deep-learning platform on mobile that puts the cutting-edge technology in the palm of a user’s hand.

Facebook is essentially embedding its neural network (a machine learning system, loosely modelled on the human brain, that learns by consuming vast tons of data) into its Facebook app for iOS and Android. The company terms this deep learning AI “Caffe2Go,” describing it as a lightweight and modular framework built on top of the open-source “Caffe2” project — meaning it won’t drain your mobile device. Facebook is also optimizing style transfer tech, allowing it to be applied to video running at 20 frames per second on an iPhone 6S or above without a drop in picture quality.

“The style-transfer tool in the camera is the result of a marriage between two technologies: the Caffe2go runtime and style-transfer models,” write Facebook research scientists Yangqing Jia and Peter Vajda in their blog post. “It took both technologies to make it possible for you to feel like you have Van Gogh’s paintbrush in your hand when you pick up your phone to shoot a video.”

In a second blog post, Facebook CTO Mike Schroepfer details the company’s vision for the mobile AI: “We can create gesture-based controls, where the computer can see where you’re pointing and activate different styles or commands. We can recognize facial expressions and perform related actions, like putting a “yay” filter over your selfie when you smile.”

Facebook CEO Mark Zuckerberg demoed the style transfer feature in October, shortly after which Facebook launched its first Live video filters.

Facebook has not shied away from boasting about its plans for the live-streaming format, and video in general. Last month, the company’s chief product officer Chris CoxCox claimed it now predicts video will generate 70 percent of traffic across Facebook’s properties — such as Instagram, WhatsApp, and its flagship social network — in the next five years. Coincidentally, WhatsApp launched video calling for Android devices in late October — indicating that Facebook is wasting no time in making that future a reality.

Updated on 11-08-2016 by Saqib Shah: Added news of Facebook’s reveal of AI for mobile app