Skip to main content

A.I. doesn’t usually forget anything, but Facebook’s new system does. Here’s why

“Hello, HAL. Do you read me, HAL?” said astronaut Dave Bowman, desperately trying to keep his emotions in check.

There was a pause and then, in an emotionless monotone, the computer responded. “Affirmative, Dave. I read you.”

Related Videos

“Open the pod bay doors, HAL.”

Another long pause. Was HAL, the all-powerful A.I. that controlled the Discovery One spacecraft, really ignoring him? Impossible, surely, Bowman thought. Any moment now,  HAL would spring into action and obey —

“I’m sorry, Dave,” HAL continued. “I’m afraid I can’t do that.”

“What’s the problem?” Dave asked.

“Well, you see, Dave. I’ve forgotten how to open them.”

OK, that’s not exactly how things went in 2001: A Space Odyssey, but the jokes basically write themselves. After all, when it comes to artificial intelligence, a good memory seems to be one of the prerequisite qualities researchers are keen to imbue in their systems. Nobody wants an A.I. that patterns itself after the kind of everyday, unremarkable intelligence that forgets things. So why, then, is Facebook building a forgetful A.I.? And why, like the old joke about the waiter and the fly in a customer’s soup, are we soon all going to want one?

What we think about when we think about forgetting

The answer, as it turns out, is that we do want something that’s more like the everyday, unremarkable intelligence that forgets things here and there. Humans forget important things like anniversaries and wallets and shutting the garage before we go on vacation. That is suboptimal forgetting. But we also forget pieces of information because we don’t need to retain it, stopping it like a chunk of detritus caught in a sink drain’s food catcher before it can pass through short-term and into long-term memory.

Facebook AI

In a famous experiment, people were asked to correctly recognize a U.S. penny from a collection of pictures showing incorrect pennies. Although the participants likely saw, and used, pennies every day, they proved surprisingly poor at the task. As the researchers wrote: “On balance, the results were consistent with the idea that the visual details of an object, even a very familiar object, are typically available from memory only to the extent that they are useful in everyday life.”

While it’s not as simple as saying that the brain can fill up like hard drive storage, there are certainly short-term memories that appear to decay when they are no longer required, while others burrow their way into our brains and live there rent-free. An example? Think about where you packed away the Christmas decorations last holidays. Next, mentally walk from the front desk to your room in the last hotel you stayed at, one that you are unlikely to ever stay in again. Neither piece of information is vitally important to your wellbeing. Nonetheless, one piece of information is needed and the other isn’t. Somehow your brain knows which one to toss in the trash.

This is the idea behind Facebook’s new A.I. project, called Expire-Span. As artificial intelligence models are increasingly applied to long-form datasets like articles and books, the computational costs associated with these models ramp up as they try to memorize more information. The problem is becoming even more pressing as people collect more and more rich multimedia data about their lives.

“The brain is a very complex system that’s not fully understood, and there are many different types of memory that form human memory.”

“As the amount of content we have grows, the major question is storage,” Angela Fan, a research scientist at Facebook AI Research Paris, told Digital Trends. “A phone, for example, has a limited amount of memory. This is even greater a problem in wearables and other on-device-type applications, where privacy reasons may mean that people strictly want content stored on their device and not on a server or a cloud, accentuating storage challenges.”

Current A.I. models take something of a Frank Sinatra approach to memories — that is to say that they remember all or nothing at all. Either they store all the information created at every time step, or they forget them all after a predetermined time. Sainbayar Sukhbaatar, another Facebook A.I. research scientist, likens it to remembering perfectly everything that happened in the last week, but absolutely nothing beyond that.

Building forgetfulness into A.I.

Researchers have built “forgetting” into A.I. models before. Long Short-Term Memory (LSTM) models, for example, added a forgetting mechanism to recurrent neural networks (RNN), one of the core technologies that drives machine learning. “RNNs have an internal memory that consists of one vector, so forgetting it means overwriting it with new information,” Sukhbaatar told Digital Trends.

By contrast, Expire-Span adds a forgetting mechanism to external memory that can contain thousands of vectors. New information is observed, but Expire-Span — whose name recalls the kind of expiration labels found on bottles of milk — has the ability to determine how long that information should stay in memory. If information isn’t deemed crucial to the future, that piece of information can be made to gradually decay, before it’s ultimately eradicated from the A.I. model’s memory to make room for more useful information.

In order to do this, the model needs to be able to predict what is and isn’t relevant to a particular task, before assigning the correct expiration date, after which the information will vanish from the A.I. system. By dumping irrelevant information, Facebook’s A.I. is able to process information at larger scales.

Facebook AI

As with much of Facebook’s artificial intelligence research, the social networking giant hasn’t announced that Expire-Span will be baked into any of its core products in the immediate future. But, as Facebook deals with more and more user data (not to mention its rumored move into smart glasses), it’s not hard to see how this technology could apply. When future A.I. tools are are able to perform at high performance levels despite dealing with massive heaps of data, Expire-Span could be the reason.

Don’t make the mistake of assuming that making a forgetful A.I. marks another step toward making A.I. more human, however. The exact reasons humans forget things is still the subject of plenty of investigation from researchers. As with the brain-inspired neural networks that dominate modern A.I., Facebook’s forgetful algorithm is modeled on the theory of forgetfulness; not an attempt to replicate it with any biofidelity.

“The brain is a very complex system that’s not fully understood, and there are many different types of memory that form human memory,” Sukhbaatar said. “Expire-Span, like all other A.I. mechanisms, may be inspired by the human brain, but ultimately are not faithful reflections of how the brain actually works. Memory in particular is a very active research area in and of itself. The human memory analogy we describe is added for the purpose of clarity only, though it definitely inspires our work.”

And don’t you forget it!

Editors' Recommendations

Split Facebook News Feed doesn’t decrease fake news, report says
facebook, facebook local news

As Facebook gears up for another News Feed algorithm change, a new report suggests tests of the platforms earlier algorithm tests aren’t going as planned. In the fall, Facebook began testing a split News Feed, with posts from friends in one location and posts from news publications in another. Now, months into the test, a New York Times report suggests the split design not only brought less views for news, but inadvertently promoted fake news. The report offers some insight into potential pitfalls for the algorithm changes Facebook is planning to roll out in the next few months.

The split feed is only a test and at the time Facebook first shared the test, the company didn’t yet have plans to bring the setup worldwide. The tests, running in Sri Lanka, Bolivia, Slovakia, Serbia, Guatemala, and Cambodia, place all the posts from publishers such as news companies inside a new Explore section, requiring users to navigate outside the News Feed to see the posts unless they are shared by a friend.

Read more
NASA and Google A.I. team up and discover a new planet
kepler nasa emergency

In what might be the biggest Google search of the decade, CNET reports that NASA has used Google's machine learning technology to discover a new exoplanet.

Orbiting Kepler-90, a sun-like star 2,545 light years away from Earth, Kepler-90i is an extremely hot, rocky, and barren planet that orbits its star in 14.4 days. Aside from its extreme temperatures, the planet is notable for the fact that it is the eighth planet discovered in the Kepler-90 system, making the system itself tied with ours in the number of planets which orbit a star.

Read more
A.I. predicts how you vote by looking at where you live on Google Street View
street view ai politics 22 artificialin

Google Maps’ Street View feature is a great way to explore the world around you, but could it be revealing more about your neighborhood than you think? That's quite possible, suggests new research coming out of Stanford University. Computer science researchers there have been demonstrated how deep learning artificial intelligence can scour the images on Google Street View and draw conclusions about issues like the political leaning of a particular area — just by looking at the cars parked out on the street.

“We wanted to show that useful insight can be gained from images, the same way people do this for social networks or other textual-based data,” Timnit Gebru, one of the lead researchers on the paper, told Digital Trends. “Some of the car-politics or car-race associations were intuitive, but still surprising that we could capture from our data.”

Read more