Skip to main content

Here’s how Facebook taught its Portal A.I. to think like a Hollywood filmmaker

Facebook Portal+ review
Dan Baker/Digital Trends

When Mark Zuckerberg built the first version of Facebook in his college dorm room at Harvard, he imagined it as a window that would allow people to look in on the lives of other users. If Google was a search engine for information then Facebook, by contrast, was a search engine for people. Fifteen years later, Facebook has taken this ambition to the next level. By creating Portal and Portal+, its line of screen-enhanced smart speakers, launched in November 2018, the social media giant has established a far more literal window, letting Facebook users to make video calls to one another.

The Portal smart speakers literalize another Facebook dream, too. Where Facebook was, in essence, a search engine for people, Portal actually does search them out: with a roving 12-megapixel camera, boasting a 140-degree field of view, which follows you around the room to see what you’re doing. As Digital Trends put it in our review, “if you’re busy moving about the kitchen while asking Grandma how to make her famous meatballs, you can keep busy while listening to her talk.”

Related Videos

What exactly is the smart technology that drives Portal? And how does Facebook think it’s cracked the challenge of making regular video chat feel as personal as sitting down for a real conversation? The answer involves some impressive artificial intelligence — and an added human touch.

Facebook Portal+ review
Dan Baker/Digital Trends

Making cameras smarter

Right from the start, Facebook knew that the core to its Portal experience would be the so-called “Smart Camera” system. The idea of the Smart Camera was to move beyond the kind of static shot that services like Skype have been offering us for years, and to play a more creative role in the process. Just as a movie director or cinematographer knows when to employ a wide shot or when to zoom in for an intimate close-up, so Facebook challenged its engineers to imitate this same ability with Portal.

To give this camera the necessary human touch, Facebook worked with filmmakers to figure out the best way of distilling their wisdom into machine learnable insights. In one case, it asked them to demonstrate how they might shoot a scene in which it was impossible to capture all the relevant information from one fixed angle.

Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.

In another, Facebook engineers looked at the different photographic elements that camera operators prioritize in portrait and landscape shots. These observations formed the basis of software models which attempt to imbue Portal with some of the decision-making quirks we would normally attribute to human creativity.

“We wanted to create a hands-free video calling experience that removes feelings of physical distance and is more like hanging out together,” Eric Hwang, one of the engineers behind Portal, explained to Digital Trends.

The resulting system — which Facebook says took it “under two years” to create from scratch — allows Portal to make decisions designed to improve the flow of a conversation. In a newly published blog post, it details some of the illustrations of why this might be necessary. For example, if you’re in a crowded room, full of people interacting with one another, it must choose when to follow an individual out of frame or when to zoom out to accommodate new subjects.

Facebook software engineers Eric Hwang (sitting in chair initially) and Arthur Cavalcanti demonstrate the Portal's cinematic camera-like tracking and framing.

Similarly, it must learn to deal with changing light situations in real time. What do you do if your subject is lying down in a dark room, half covered by a blanket, but there are kids running around in the background causing motion blur? Portal weighs all of this information in less than the blink of an eye and tries to determine the best outcome. (If you want to manually control who it focuses on, that’s now possible too.)

Technical challenges

From a technical perspective, a a couple of things make Portal’s technology impressive. The first is that it can do all of this without the use of an actual moving camera. Early on in the development process, Portal’s engineers tried out prototypes which used a motorized camera, which swiveled to face subjects. However, this was decided against on the basis that it caused a lag and a point of potential mechanical failure. Instead, Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.

Second, the team working on Portal found a way to achieve its decision making processes without having to rely on cloud computing. According to Hwang, the computational firepower is all achieved in-device.

Evolution of the Facebook Portal
Early Portal prototypes relied on a motor to physically move the camera. Facebook Engineering

“Capturing everyone in a video frame isn’t a hard engineering problem, as many engineers can do that with today’s computer vision advancements,” he said. “The innovation is in capturing the relevant people or person in real-time, on-device, using just the small mobile chip inside Portal as processing power. Usually these types of A.I. tasks require dedicated, large servers. [We] overcame that obstacle by compressing complex computer vision models until they could fit on the chip we use for Portal and still run accurately and reliably.”

To do this, Portal draws on Facebook’s long-term investment in artificial intelligence. It uses a 2D pose-detection system which runs at 30 frames per second. The intentionality of these poses help Portal to make continuous decisions about what its subjects are doing — and when it might need to digitally pan or zoom as a result. It additionally utilizes research into depth cameras developed by Facebook Reality Labs as part of the social media giant’s virtual reality efforts.

A growing market

Facebook is convinced that it is onto a winner with Portal. It’s easy to see where its confidence comes from. Right now, the smart speaker market is booming. Although largely dominated by market leader Amazon, it is growing at more than 100 percent year-on-year. That’s good news for tech companies searching for the next big thing at a time of flattening smartphone sales.

Facebook Portal+ review
Dan Baker/Digital Trends

While Facebook was the last of the big four tech giants (Amazon, Alphabet, Facebook and Apple) to jump on the bandwagon, it is still one of the first wave of smart speakers centered around the screen as a communication device.

“Portal is the only product on the market of its kind,” Hwang said. “Today, smart speakers and displays are built around information and commerce. Portal is built to make it easier to connect with the people that matter most: our closest friends and family. And Portal is focused on connecting people — part of Facebook’s mission — which is not currently served well by the home device market.”

Privacy challenges ahead?

So what’s stopping stopping Facebook? Well, potentially privacy. Users have proven surprisingly willing to embrace “always listening” gadgets from companies like Google with a vested interest in user data. But a device that both watches and listens you is more invasive still. Furthermore, Facebook’s reputation is still suffering after last year’s Cambridge Analytica scandal.

Adding smarts to the Portal video chat camera (Facebook)

Just days before this very article was published, the Washington Post reported that Facebook is negotiating a record breaking, multi-billion dollar settlement with the FTC for its privacy misdemeanors. With a growing backlash from many former users, it’s yet to be revealed if Facebook has an Amazon Echo-style hit on its hands — or an Amazon Fire Phone-style flop.

Facebook assured us that it does not listen to, view, or keep the contents of Portal video calls, which are additionally encrypted to avoid eavesdropping. The fact that Portal’s A.I. smarts run locally on the device, and not on Facebook servers, also means that this information does not leave your home. Voice commands are sent to the company only after you say “Hey Portal,” and users can delete their voice history in Facebook’s Activity Log at any time.

But there’s no getting around the fact that there is still a degree of data collection taking place. “While we don’t listen to, view, or keep the contents of your Portal video calls, or use this information to target ads, we do process some device usage information to understand how Portal is being used and to improve the product,” Facebook notes. (Portal’s privacy policy can be read here.)

Portal offers some very smart technology with massive implications for the future of video chat. There’s no doubt that the company has managed to pull off something very impressive from a technological point of view. But whether it can convince potential customers that this is a solution they need in their lives will, ultimately, prove to be the real achievement.

Editors' Recommendations

CES 2023: Ring expands its watchful eye to vehicles with Ring Car Cam
The Ring Car Cam installed on a windshield.

Ring has long dominated the smart doorbell market, offering an intuitive product that makes it easy to see who (or what) has wandered up to your front door. The company is now turning its gaze towards vehicles, as it debuted the Ring Car Cam during CES 2023.

The Ring Car Cam is exactly what it sounds like -- a dash cam that attaches to your windshield and dashboard. It offers a dual-facing camera that captures both the interior of your vehicle and its surroundings, offering you peace of mind when parked or serving as evidence should you get in an accident. You can also communicate through the Ring Car Cam thanks to its built-in microphone and speaker.

Read more
Samsung reveals futuristic new smart home appliances for CES 2023
A person using the new Bespoke fridge touchscreen.

The first day of CES 2023 is right around the corner, but Samsung isn't waiting to introduce the world to its new lineup of smart home appliances. Specifically, the Bespoke lineup is now on full display, with new smart refrigerators, smart ovens, and smart washers making an appearance.

Samsung’s Bespoke lineup has long been a premium choice for smart home shoppers -- and that trend looks to continue throughout this year. One of the biggest upgrades is for the Bespoke 4-Door Flex Refrigerator with Family Hub+, which now offers a massive 32-inch touchscreen (up from a 21.5-inch display) that’s embedded directly into its glass panel door. The screen will support the new Family Hub software, allowing you to stream your favorite shows, share photos, or check the status of connected devices.

Read more
The truth about outdoor smart home gadgets and extreme cold
House buried in snow by blizzard.

Electronics and smart home gadgets bring convenience and automation to your home and often need minimal maintenance, save for the odd firmware update -- that is, unless you live in a place that gets an actual winter. While most shoppers are eager to set up and play with their new toys, they mainly worry about getting it with that luxurious same-day shipping and don’t think ahead to how that new device will operate when the weather turns harsh. The truth is, if you live somewhere it gets bitterly, extremely cold, your smart devices like wireless cameras, lights, and other components will likely stop working.
Pay attention to temperature range
When shopping for an outdoor device, we usually pay attention to the IP rating. Many people see this number and assume it means their gadget is impervious to any kind of weather. That might be true to some extent, but the IP rating doesn't extend to extreme heat or extreme cold. IP ratings only rate for water and/or dust ingression, not for how effectively cold or heat can penetrate. To know how a device might be able to withstand cold winters or hot summers, you need to check the temperature operating range.

Most outdoor devices will provide this operating range somewhere in the specs. If you don't see them, that's a bit of a red flag. It might be worth reaching out to customer service or checking user reviews to see how they’ve held up for others in real-world conditions.

Read more