Here’s how Facebook taught its Portal A.I. to think like a Hollywood filmmaker

Facebook Portal+ review
Dan Baker/Digital Trends

When Mark Zuckerberg built the first version of Facebook in his college dorm room at Harvard, he imagined it as a window that would allow people to look in on the lives of other users. If Google was a search engine for information then Facebook, by contrast, was a search engine for people. Fifteen years later, Facebook has taken this ambition to the next level. By creating Portal and Portal+, its line of screen-enhanced smart speakers, launched in November 2018, the social media giant has established a far more literal window, letting Facebook users to make video calls to one another.

The Portal smart speakers literalize another Facebook dream, too. Where Facebook was, in essence, a search engine for people, Portal actually does search them out: with a roving 12-megapixel camera, boasting a 140-degree field of view, which follows you around the room to see what you’re doing. As Digital Trends put it in our review, “if you’re busy moving about the kitchen while asking Grandma how to make her famous meatballs, you can keep busy while listening to her talk.”

What exactly is the smart technology that drives Portal? And how does Facebook think it’s cracked the challenge of making regular video chat feel as personal as sitting down for a real conversation? The answer involves some impressive artificial intelligence — and an added human touch.

Facebook Portal+ review
Dan Baker/Digital Trends

Making cameras smarter

Right from the start, Facebook knew that the core to its Portal experience would be the so-called “Smart Camera” system. The idea of the Smart Camera was to move beyond the kind of static shot that services like Skype have been offering us for years, and to play a more creative role in the process. Just as a movie director or cinematographer knows when to employ a wide shot or when to zoom in for an intimate close-up, so Facebook challenged its engineers to imitate this same ability with Portal.

To give this camera the necessary human touch, Facebook worked with filmmakers to figure out the best way of distilling their wisdom into machine learnable insights. In one case, it asked them to demonstrate how they might shoot a scene in which it was impossible to capture all the relevant information from one fixed angle.

Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.

In another, Facebook engineers looked at the different photographic elements that camera operators prioritize in portrait and landscape shots. These observations formed the basis of software models which attempt to imbue Portal with some of the decision-making quirks we would normally attribute to human creativity.

“We wanted to create a hands-free video calling experience that removes feelings of physical distance and is more like hanging out together,” Eric Hwang, one of the engineers behind Portal, explained to Digital Trends.

The resulting system — which Facebook says took it “under two years” to create from scratch — allows Portal to make decisions designed to improve the flow of a conversation. In a newly published blog post, it details some of the illustrations of why this might be necessary. For example, if you’re in a crowded room, full of people interacting with one another, it must choose when to follow an individual out of frame or when to zoom out to accommodate new subjects.

Facebook software engineers Eric Hwang (sitting in chair initially) and Arthur Cavalcanti demonstrate the Portal's cinematic camera-like tracking and framing.

Similarly, it must learn to deal with changing light situations in real time. What do you do if your subject is lying down in a dark room, half covered by a blanket, but there are kids running around in the background causing motion blur? Portal weighs all of this information in less than the blink of an eye and tries to determine the best outcome. (If you want to manually control who it focuses on, that’s now possible too.)

Technical challenges

From a technical perspective, a a couple of things make Portal’s technology impressive. The first is that it can do all of this without the use of an actual moving camera. Early on in the development process, Portal’s engineers tried out prototypes which used a motorized camera, which swiveled to face subjects. However, this was decided against on the basis that it caused a lag and a point of potential mechanical failure. Instead, Portal comprises an extremely wide-angle lens in which all movement and editing decisions are made entirely digitally.

Second, the team working on Portal found a way to achieve its decision making processes without having to rely on cloud computing. According to Hwang, the computational firepower is all achieved in-device.

Evolution of the Facebook Portal
Early Portal prototypes relied on a motor to physically move the camera. Facebook Engineering

“Capturing everyone in a video frame isn’t a hard engineering problem, as many engineers can do that with today’s computer vision advancements,” he said. “The innovation is in capturing the relevant people or person in real-time, on-device, using just the small mobile chip inside Portal as processing power. Usually these types of A.I. tasks require dedicated, large servers. [We] overcame that obstacle by compressing complex computer vision models until they could fit on the chip we use for Portal and still run accurately and reliably.”

To do this, Portal draws on Facebook’s long-term investment in artificial intelligence. It uses a 2D pose-detection system which runs at 30 frames per second. The intentionality of these poses help Portal to make continuous decisions about what its subjects are doing — and when it might need to digitally pan or zoom as a result. It additionally utilizes research into depth cameras developed by Facebook Reality Labs as part of the social media giant’s virtual reality efforts.

A growing market

Facebook is convinced that it is onto a winner with Portal. It’s easy to see where its confidence comes from. Right now, the smart speaker market is booming. Although largely dominated by market leader Amazon, it is growing at more than 100 percent year-on-year. That’s good news for tech companies searching for the next big thing at a time of flattening smartphone sales.

Facebook Portal+ review
Dan Baker/Digital Trends

While Facebook was the last of the big four tech giants (Amazon, Alphabet, Facebook and Apple) to jump on the bandwagon, it is still one of the first wave of smart speakers centered around the screen as a communication device.

“Portal is the only product on the market of its kind,” Hwang said. “Today, smart speakers and displays are built around information and commerce. Portal is built to make it easier to connect with the people that matter most: our closest friends and family. And Portal is focused on connecting people — part of Facebook’s mission — which is not currently served well by the home device market.”

Privacy challenges ahead?

So what’s stopping stopping Facebook? Well, potentially privacy. Users have proven surprisingly willing to embrace “always listening” gadgets from companies like Google with a vested interest in user data. But a device that both watches and listens you is more invasive still. Furthermore, Facebook’s reputation is still suffering after last year’s Cambridge Analytica scandal.

Just days before this very article was published, the Washington Post reported that Facebook is negotiating a record breaking, multi-billion dollar settlement with the FTC for its privacy misdemeanors. With a growing backlash from many former users, it’s yet to be revealed if Facebook has an Amazon Echo-style hit on its hands — or an Amazon Fire Phone-style flop.

Facebook assured us that it does not listen to, view, or keep the contents of Portal video calls, which are additionally encrypted to avoid eavesdropping. The fact that Portal’s A.I. smarts run locally on the device, and not on Facebook servers, also means that this information does not leave your home. Voice commands are sent to the company only after you say “Hey Portal,” and users can delete their voice history in Facebook’s Activity Log at any time.

But there’s no getting around the fact that there is still a degree of data collection taking place. “While we don’t listen to, view, or keep the contents of your Portal video calls, or use this information to target ads, we do process some device usage information to understand how Portal is being used and to improve the product,” Facebook notes. (Portal’s privacy policy can be read here.)

Portal offers some very smart technology with massive implications for the future of video chat. There’s no doubt that the company has managed to pull off something very impressive from a technological point of view. But whether it can convince potential customers that this is a solution they need in their lives will, ultimately, prove to be the real achievement.

Smart Home

Facebook says new Portal video-chat devices are coming in the fall

A Facebook executive has revealed the company is gearing up to refresh its video-calling smart displays later this year. The Portal and Portal+ devices launched to lukewarm reviews toward the end of last year.
Music

Apple Music vs. Spotify: Which service is the streaming king?

Apple Music is giving Spotify a run for its money, but which service is best for you? In our Apple Music vs. Spotify showdown, we compare and contrast all we know about the two streaming music services.
Home Theater

Smart speakers add convenience to your life, and these are the best right now

Stuck talking to yourself? Get an A.I. assistant to keep you company! Whether you put your stock in Alexa, Google Assistant, or Siri, we've picked out the best smart speaker for any smart home ecosystem.
Smart Home

Best deals on Amazon Echo and Google Home Nest devices for Father’s Day

You can save money on a wide range of devices for Amazon Echo and Google Home Nest smart homes for Father's Day. Here are the best discounts on smart speakers, smart displays, video doorbells, and smart outdoor security cameras.
Emerging Tech

Sloshed drone pilots in Japan can now be punished with jail time

If you're flying a drone in Japan, better not be sloshed when you send your bird skyward. A new law passed this week could see drunk drone pilots sent to jail for up to a year or hit with a hefty fine.
Emerging Tech

Mount Everest is now home to the world’s highest weather station

A team of scientists has created a new record with the installation of the world’s highest weather station atop Everest. Data from the expedition will help researchers better understand the effect of climate change on the region.
Emerging Tech

This drone with hands looks like a nightmare straight out of Black Mirror

This unlikely drone-with-hands creation is the work of Federico Ciccarese, the brains behind YouBionic, a bionic hand project that has evolved far beyond its original brief. Check it out.
Emerging Tech

This crazy-looking robot uses microspines on its legs to climb up walls

Researchers at Carnegie Mellon have built a bioinspired robot, which uses microspines on its feet to grip onto rough surfaces. This allows it to climb up very steep gradients. Check it out.
Digital Trends Live

Digital Trends Live: Florida’s autonomous vehicle law, E3 updates, and more

On this episode of DT Live, we take a look at the biggest trending stories in tech, including Florida allowing fully autonomous vehicles on the road, Atari’s new gaming system, E3 updates, high-speed rail, and more.
Emerging Tech

Got $400 million to burn? The world’s largest airplane is up for sale

Stratolaunch, the world's largest airplane, is up for sale. All it'll cost you is $400 million dollars. The brainchild of late Microsoft co-founder Paul Allen, the plane was supposed to make space travel more accessible and affordable.
Emerging Tech

Ex astris, scientia: Star Trek logo spotted on the surface of Mars

The Mars Reconnaissance Orbiter has been boldly going to Mars and capturing images since 2005, and now it has spotted something where no man has gone before: a structure on the planet's surface which will look familiar to Trekkies.
Emerging Tech

Adobe develops tool to identify Photoshopped images of faces

With deepfake videos making headlines, and campaigns against the Photoshopping of models, people are more aware than ever of the digital manipulation of images. Now Adobe wants to give tools to users to let them spot faked images.
Emerging Tech

NASA’s Mars 2020 rover will pave the way for manned missions to Mars

Survival on Mars is a massive challenge for humanity. To cope with the highly variable temperatures, lack of oxygen and water, and high levels of radiation, the Mars 2020 rover will carry instruments to pave the way for human exploration.
Emerging Tech

Facebook builds virtual homes to train A.I. agents in realistic environments

Researchers at Facebook have created Habitat, which is a platform that enables rapid training for A.I. agents. They will receive thousands of hours of training in just a few minutes in the virtual homes.