Here’s how Facebook taught its Portal A.I. to think like a Hollywood filmmaker

Facebook Portal+ review
Dan Baker/Digital Trends

When Mark Zuckerberg built the first version of Facebook in his college dorm room at Harvard, he imagined it as a window that would allow people to look in on the lives of other users. If Google was a search engine for information then Facebook, by contrast, was a search engine for people. Fifteen years later, Facebook has taken this ambition to the next level. By creating Portal and Portal+, its line of screen-enhanced smart speakers, launched in November 2018, the social media giant has established a far more literal window, letting Facebook users to make video calls to one another.

The Portal smart speakers literalize another Facebook dream, too. Where Facebook was, in essence, a search engine for people, Portal actually does search them out: with a roving 12-megapixel camera, boasting a 140-degree field of view, which follows you around the room to see what you’re doing. As Digital Trends put it in our review, “if you’re busy moving about the kitchen while asking Grandma how to make her famous meatballs, you can keep busy while listening to her talk.”

What exactly is the smart technology that drives Portal? And how does Facebook think it’s cracked the challenge of making regular video chat feel as personal as sitting down for a real conversation? The answer involves some impressive artificial intelligence — and an added human touch.

Facebook Portal+ review
Dan Baker/Digital Trends

Making cameras smarter

Right from the start, Facebook knew that the core to its Portal experience would be the so-called “Smart Camera” system. The idea of the Smart Camera was to move beyond the kind of static shot that services like Skype have been offering us for years, and to play a more creative role in the process. Just as a movie director or cinematographer knows when to employ a wide shot or when to zoom in for an intimate close-up, so Facebook challenged its engineers to imitate this same ability with Portal.

To give this camera the necessary human touch, Facebook worked with filmmakers to figure out the best way of distilling their wisdom into machine learnable insights. In one case, it asked them to demonstrate how they might shoot a scene in which it was impossible to capture all the relevant information from one fixed angle.

Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.

In another, Facebook engineers looked at the different photographic elements that camera operators prioritize in portrait and landscape shots. These observations formed the basis of software models which attempt to imbue Portal with some of the decision-making quirks we would normally attribute to human creativity.

“We wanted to create a hands-free video calling experience that removes feelings of physical distance and is more like hanging out together,” Eric Hwang, one of the engineers behind Portal, explained to Digital Trends.

The resulting system — which Facebook says took it “under two years” to create from scratch — allows Portal to make decisions designed to improve the flow of a conversation. In a newly published blog post, it details some of the illustrations of why this might be necessary. For example, if you’re in a crowded room, full of people interacting with one another, it must choose when to follow an individual out of frame or when to zoom out to accommodate new subjects.

Facebook software engineers Eric Hwang (sitting in chair initially) and Arthur Cavalcanti demonstrate the Portal's cinematic camera-like tracking and framing.

Similarly, it must learn to deal with changing light situations in real time. What do you do if your subject is lying down in a dark room, half covered by a blanket, but there are kids running around in the background causing motion blur? Portal weighs all of this information in less than the blink of an eye and tries to determine the best outcome. (If you want to manually control who it focuses on, that’s now possible too.)

Technical challenges

From a technical perspective, a a couple of things make Portal’s technology impressive. The first is that it can do all of this without the use of an actual moving camera. Early on in the development process, Portal’s engineers tried out prototypes which used a motorized camera, which swiveled to face subjects. However, this was decided against on the basis that it caused a lag and a point of potential mechanical failure. Instead, Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.

Second, the team working on Portal found a way to achieve its decision making processes without having to rely on cloud computing. According to Hwang, the computational firepower is all achieved in-device.

Evolution of the Facebook Portal
Early Portal prototypes relied on a motor to physically move the camera. Facebook Engineering

“Capturing everyone in a video frame isn’t a hard engineering problem, as many engineers can do that with today’s computer vision advancements,” he said. “The innovation is in capturing the relevant people or person in real-time, on-device, using just the small mobile chip inside Portal as processing power. Usually these types of A.I. tasks require dedicated, large servers. [We] overcame that obstacle by compressing complex computer vision models until they could fit on the chip we use for Portal and still run accurately and reliably.”

To do this, Portal draws on Facebook’s long-term investment in artificial intelligence. It uses a 2D pose-detection system which runs at 30 frames per second. The intentionality of these poses help Portal to make continuous decisions about what its subjects are doing — and when it might need to digitally pan or zoom as a result. It additionally utilizes research into depth cameras developed by Facebook Reality Labs as part of the social media giant’s virtual reality efforts.

A growing market

Facebook is convinced that it is onto a winner with Portal. It’s easy to see where its confidence comes from. Right now, the smart speaker market is booming. Although largely dominated by market leader Amazon, it is growing at more than 100 percent year-on-year. That’s good news for tech companies searching for the next big thing at a time of flattening smartphone sales.

Facebook Portal+ review
Dan Baker/Digital Trends

While Facebook was the last of the big four tech giants (Amazon, Alphabet, Facebook and Apple) to jump on the bandwagon, it is still one of the first wave of smart speakers centered around the screen as a communication device.

“Portal is the only product on the market of its kind,” Hwang said. “Today, smart speakers and displays are built around information and commerce. Portal is built to make it easier to connect with the people that matter most: our closest friends and family. And Portal is focused on connecting people — part of Facebook’s mission — which is not currently served well by the home device market.”

Privacy challenges ahead?

So what’s stopping stopping Facebook? Well, potentially privacy. Users have proven surprisingly willing to embrace “always listening” gadgets from companies like Google with a vested interest in user data. But a device that both watches and listens you is more invasive still. Furthermore, Facebook’s reputation is still suffering after last year’s Cambridge Analytica scandal.

Just days before this very article was published, the Washington Post reported that Facebook is negotiating a record breaking, multi-billion dollar settlement with the FTC for its privacy misdemeanors. With a growing backlash from many former users, it’s yet to be revealed if Facebook has an Amazon Echo-style hit on its hands — or an Amazon Fire Phone-style flop.

Facebook assured us that it does not listen to, view, or keep the contents of Portal video calls, which are additionally encrypted to avoid eavesdropping. The fact that Portal’s A.I. smarts run locally on the device, and not on Facebook servers, also means that this information does not leave your home. Voice commands are sent to the company only after you say “Hey Portal,” and users can delete their voice history in Facebook’s Activity Log at any time.

But there’s no getting around the fact that there is still a degree of data collection taking place. “While we don’t listen to, view, or keep the contents of your Portal video calls, or use this information to target ads, we do process some device usage information to understand how Portal is being used and to improve the product,” Facebook notes. (Portal’s privacy policy can be read here.)

Portal offers some very smart technology with massive implications for the future of video chat. There’s no doubt that the company has managed to pull off something very impressive from a technological point of view. But whether it can convince potential customers that this is a solution they need in their lives will, ultimately, prove to be the real achievement.

Product Review

Sony's Xperia 10 falls frustratingly short of budget brilliance

Better know for premium products, Sony does also have a budget smartphone range, but can the Sony Xperia 10 compete with great cheap phones from Nokia and Motorola? We put it through its paces to find out how it measures up.
Smart Home

Protect yourself: Here are some of the best home security systems

Looking for the best home security systems for your house? These systems offer the best mix of devices, smart features, monitoring services, and fees that you can afford (plus good customer service reports).

Whether by the pool or the sea, make a splash with the best waterproof phones

Whether you're looking for a phone you can use in the bath, or you just want that extra peace of mind, waterproof phones are here and they're amazing. Check out our selection of the best ones you can buy.
Product Review

Gate’s Smart Lock is locked and loaded but ultimately lacks important basics

In a world of video cameras and doorbells comes the Gate Smart Lock, a lock with a video camera embedded. It’s a great idea, but lacks some crucial functionality to make it a top-notch product.
Emerging Tech

Take a dip in the Lagoon Nebula in first image from SPECULOOS instrument

The European Southern Observatory has released the first image collected by their new SPECULOOS instrument, and it's a stunning portrait of the Lagoon Nebula, a swirling cloud of dust and gas where new stars are born.
Emerging Tech

Robot assistants from Toyota and Panasonic gear up for the Tokyo Olympics

Japan plans to use the 2020 Olympics to showcase a range of its advanced technologies. Toyota and Panasonic are already getting in on the act, recently unveiling several robotic designs that they intend to deploy at the event.
Emerging Tech

A.I.-generated text is supercharging fake news. This is how we fight back

A new A.I. tool is reportedly able to spot passages of text written by algorithm. Here's why similar systems might prove essential in a world of fake news created by smart machines.
Emerging Tech

Racing to catch a flight? Robot valet at French airport will park your car

Hate searching for parking at the airport when you need to catch a plane? Startup Stanley Robotics recently unveiled a new outdoor automated robotic valet system. Here's how it works.

Bags with brains: Smart luggage and gadgets are making travel smoother

The bag you use to tote your stuff can affect the experience of any trip. In response, suitcases are wising up, and there are now options for smart luggage with scales, tracking, and more. Here are our favorite pieces.

At $99, Nvidia’s Jetson Nano minicomputer seeks to bring robotics to the masses

Nvidia announced a new A.I. computer, the Jetson Nano. This computer comes with an 128-core GPU that Nvidia claims can handle pretty much any A.I. framework you could imagine. At $99, it's an affordable way for A.I. newbies to get involved.

Nvidia’s A.I. Playground lets you edit photos, experience deep learning research

Nvidia is making it easier to access information on deep learning research. It has launched an online space with three demos for image editing, styling, as well as photorealistic image synthesis. 

British Airways’ new Club Suite for business class comes with a door

British Airways is going after a bigger slice of the business class market with the imminent launch of the Club Suite. The plush seating option offers a more private space as well as an easier route to the bathroom.
Smart Home

Sony’s Aibo robot dog can now patrol your home for persons of interest

Sony released the all-new Aibo in the U.S. around nine months ago, and since then the robot dog has (hopefully) been melting owners' hearts with its cute looks and clever tricks. Now it has a new one up its sleeve.
Emerging Tech

Inflating smart pills could be a painless alternative to injections

Could an inflating pill containing hidden microneedles replace painful injections? The creators of the RaniPill robotic capsule think so — and they have the human trials to prove it.