Skip to main content

MIT, Adobe’s new A.I. could bring one-click background removal, social filters

AI-assisted Image Editing

Removing a background from a photo is a time-consuming process — but researchers from MIT, Adobe, and Microsoft may have just created technology that will allow for one-click background removal, even around tough edges like hair and fur. The process, called semantic soft segmentation, uses color and texture to automatically separate the image into layers. The technology is already good enough to be used for tasks like creating better Instagram and Snapchat filters, but the researchers anticipate future reiterations that will allow for one-click background removal for non-professional photo-editing apps and even video.

Recommended Videos

Selecting an object in a photograph is a common task for photo editors. With the image segmented into layers, it’s possible to create a transparent background, place the subject on an entirely different background, or apply changes just to the subject, or just to the background, like background blur.

Soft semantic segmentation uses machine learning to automate what’s typically a tedious task in photo editing. The neural network analyzes the image using texture and color as well as factors like object recognition. Using that information, the software separates the image into layers automatically. Once the image is segmented, then effects like background blur or background removal can be easily applied to the image.

The process currently works for still images, but the team says it hopes further research will allow the process to be used for videos as well. The technology also takes several minutes to work and further research could improve the system’s speed. Even in its early state, the team says that the technology could be used for creating photo filters for social networks that adjust only the background, or to mimic an image from a different kind of camera — which sounds a lot like a portrait mode without the dual camera.

“Instead of needing an expert editor to spend several minutes tweaking an image frame by frame and pixel by pixel, we’d like to make the process simpler and faster so that image editing can be more accessible to casual users,” Yagiz Aksoy, a visiting researcher at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “The vision is to get to a point where it just takes a single click for editors to combine images to create these full-blown, realistic fantasy worlds.”

The research isn’t the first to try to automatically separate the subject and the background — Photoshop already has a Select Subject tool that helps start a similar process and Instagram has a filter option for portrait-mode-like effects without the dual camera. The new process, however, uses what the researchers call “soft transitions” for all the layers. That creates a more realistic edge in the final image, particularly when working with edges that are traditionally difficult to select like hair and fur.

The research was presented during last week’s SIGGRAPH conference in Vancouver and published as a white paper.

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Astronaut’s stunning photo would look just fine in an art gallery
A star trail captured from the ISS.

In his final days aboard the International Space Station (ISS) in a mission that’s lasted six months, American astronaut Don Pettit has posted a sublime shot that wouldn’t look out of place in an art gallery.

The extraordinary image was captured through a window of a SpaceX Crew Dragon spacecraft docked at the ISS, and shows star trails and city lights on Earth some 250 miles below.

Read more
Cool space video shows star trails stretching over city lights
A screen grab from a video showing star trails stretching over city lights on Earth.

NASA astronaut Don Pettit dazzled us on Tuesday with a gorgeous image showing star trails and city lights as seen from the International Space Station (ISS).

On Wednesday, he turned the "wow" dial all the way up to 11 by sharing a video clip from which Tuesday’s image was taken.

Read more
Blue Ghost’s moon landing shown in dramatic new footage
Images of the lunar surface captured by the Blue Ghost lander.

When the Blue Ghost lander reached the lunar surface on March 2, Firefly Aerospace become the first commercial company to fully succeed in a soft moon landing at the first attempt.

A week on, and a team at NASA’s Langley Research Center in Virginia has just released first-of-its-kind footage of a lunar lander’s powerful engine plumes interacting with the moon’s surface, captured as the Blue Ghost touched down.

Read more