Removing a background from a photo is a time-consuming process — but researchers from MIT, Adobe, and Microsoft may have just created technology that will allow for one-click background removal, even around tough edges like hair and fur. The process, called semantic soft segmentation, uses color and texture to automatically separate the image into layers. The technology is already good enough to be used for tasks like creating better Instagram and Snapchat filters, but the researchers anticipate future reiterations that will allow for one-click background removal for non-professional photo-editing apps and even video.
Selecting an object in a photograph is a common task for photo editors. With the image segmented into layers, it’s possible to create a transparent background, place the subject on an entirely different background, or apply changes just to the subject, or just to the background, like background blur.
Soft semantic segmentation uses machine learning to automate what’s typically a tedious task in photo editing. The neural network analyzes the image using texture and color as well as factors like object recognition. Using that information, the software separates the image into layers automatically. Once the image is segmented, then effects like background blur or background removal can be easily applied to the image.
The process currently works for still images, but the team says it hopes further research will allow the process to be used for videos as well. The technology also takes several minutes to work and further research could improve the system’s speed. Even in its early state, the team says that the technology could be used for creating photo filters for social networks that adjust only the background, or to mimic an image from a different kind of camera — which sounds a lot like a portrait mode without the dual camera.
“Instead of needing an expert editor to spend several minutes tweaking an image frame by frame and pixel by pixel, we’d like to make the process simpler and faster so that image editing can be more accessible to casual users,” Yagiz Aksoy, a visiting researcher at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “The vision is to get to a point where it just takes a single click for editors to combine images to create these full-blown, realistic fantasy worlds.”
The research isn’t the first to try to automatically separate the subject and the background — Photoshop already has a Select Subject tool that helps start a similar process and Instagram has a filter option for portrait-mode-like effects without the dual camera. The new process, however, uses what the researchers call “soft transitions” for all the layers. That creates a more realistic edge in the final image, particularly when working with edges that are traditionally difficult to select like hair and fur.
The research was presented during last week’s SIGGRAPH conference in Vancouver and published as a white paper.
