Skip to main content

Facebook touts groundbreaking app AI that powers artistic Live video filters

Facebook is betting on video filters (the graphics pioneered by Snapchat that can be overlaid on faces in recorded clips) to get more users hooked on its live-streaming tool. Having already demoed its upcoming style transfer graphics (artistic filters that draw from the style of famous painters to transform videos into works of art), it is now offering more details about the artificial intelligence powering its new mobile tools.

Instead of Snapchat, the inspiration for these Live video filters comes from Prisma — an AI-powered photography app that boasts a range of artistic graphics that can be applied to both images and video. Facebook is currently testing style transfers as part of an update that gives added prominence to your smartphone camera within its flagship app in Ireland.

Recommended Videos

Unlike Snapchat (the app that pioneered filters or, as it calls them, “lenses”), Facebook wants to apply the software to live broadcasts in real time and not after you’ve captured a clip. It’s already started doing this with some of its traditional, Snapchat-style filters, but style transfers are — according to Facebook — its crowning achievement.

On Tuesday, the company published a new blog post explaining in-depth how the style transfer technique works in real-time. Instead of waiting for the image to be processed on huge computer servers in data centers, Facebook claims it has developed a world-first: a deep-learning platform on mobile that puts the cutting-edge technology in the palm of a user’s hand.

Facebook is essentially embedding its neural network (a machine learning system, loosely modelled on the human brain, that learns by consuming vast tons of data) into its Facebook app for iOS and Android. The company terms this deep learning AI “Caffe2Go,” describing it as a lightweight and modular framework built on top of the open-source “Caffe2” project — meaning it won’t drain your mobile device. Facebook is also optimizing style transfer tech, allowing it to be applied to video running at 20 frames per second on an iPhone 6S or above without a drop in picture quality.

fb_style_transfer
Image used with permission by copyright holder

“The style-transfer tool in the camera is the result of a marriage between two technologies: the Caffe2go runtime and style-transfer models,” write Facebook research scientists Yangqing Jia and Peter Vajda in their blog post. “It took both technologies to make it possible for you to feel like you have Van Gogh’s paintbrush in your hand when you pick up your phone to shoot a video.”

In a second blog post, Facebook CTO Mike Schroepfer details the company’s vision for the mobile AI: “We can create gesture-based controls, where the computer can see where you’re pointing and activate different styles or commands. We can recognize facial expressions and perform related actions, like putting a “yay” filter over your selfie when you smile.”

Facebook CEO Mark Zuckerberg demoed the style transfer feature in October, shortly after which Facebook launched its first Live video filters.

Facebook has not shied away from boasting about its plans for the live-streaming format, and video in general. Last month, the company’s chief product officer Chris CoxCox claimed it now predicts video will generate 70 percent of traffic across Facebook’s properties — such as Instagram, WhatsApp, and its flagship social network — in the next five years. Coincidentally, WhatsApp launched video calling for Android devices in late October — indicating that Facebook is wasting no time in making that future a reality.

Updated on 11-08-2016 by Saqib Shah: Added news of Facebook’s reveal of AI for mobile app

Saqib Shah
Saqib Shah is a Twitter addict and film fan with an obsessive interest in pop culture trends. In his spare time he can be…
Bluesky finally adds a feature many had been waiting for
A blue sky with clouds.

Bluesky has been making a lot of progress in recent months by simplifying the process to sign up while at the same time rolling out a steady stream of new features.

As part of those continuing efforts, the social media app has just announced that users can now send direct messages (DMs).

Read more
Reddit just achieved something for the first time in its 20-year history
The Reddit logo.

Reddit’s on a roll. The social media platform has just turned a profit for the first time in its 20-year history, and now boasts a record 97.2 million daily active users, marking a year-over-year increase of 47%. A few times during the quarter, the figure topped 100 million, which Reddit CEO and co-founder Steve Huffman said in a letter to shareholders had been a “long-standing milestone” for the site.

The company, which went public in March, announced the news in its third-quarter earnings results on Tuesday.

Read more
Worried about the TikTok ban? This is how it might look on your phone
TikTok splash screen on an Android phone.

The US Supreme Court has decided to uphold a law that would see TikTok banned in the country on January 19. Now, the platform has issued an official statement, confirming that it will indeed shut down unless it gets some emergency relief from the outgoing president.

“Unless the Biden Administration immediately provides a definitive statement to satisfy the most critical service providers assuring non-enforcement, unfortunately TikTok will be forced to go dark on January 19,” said the company soon after the court’s verdict.
So, what does going dark mean?
So, far, there is no official statement on what exactly TikTok means by “going dark.” There is a lot of speculation out there on how exactly the app or website will look once TikTok shutters in the US.

Read more