Skip to main content

Google and MIT’s automatic photo software can edit your shots before you take them

Photo editing software often enhances a smartphone’s limited hardware, but what if those shots could be edited automatically — in real time? That’s the question researchers from the Massachusetts Institute of Technology (MIT) and Google asked when expanding an earlier machine-learning program that automatically improved shots after sending them to the cloud. Presented during this week’s Siggraph digital graphics conference, the new automatic photo software edits so quickly that the user actually sees the results on the screen in real time — before that shot is even taken.

The program is based on earlier MIT research that trained a computer to edit images automatically. Researchers taught that program how to automatically apply specific adjustments by feeding the system five variations of each image, all edited by a different photo editor. After repeating that process a few thousand times, the program learned how to identify and fix common image issues.

Recommended Videos

The new software is built on the same artificial intelligence platform but speeds up the program to the point where it completes edits in a tenth of the previous time. This allows the live view from the camera to show edits in real time instead of sending the shot to a cloud system for editing. So how did researchers achieve the speed boost? The output of the software actually isn’t an image, but an image formula. Calculating the changes (and only applying them if a photo is taken) speeds up the process.

The original image is also divided into grids, a form of downsizing the image for faster editing without losing pixels in the final image. The program works on each of the divided sections at once, editing hundreds of pixels simultaneously instead of factoring every single pixel.

With those two changes, the program only needs about 100 megabytes to perform each image edit, while without those two modifications the same software needed 12 gigabytes.

The research is a growing trend of looking to computational photography to solve the shortcomings of the hardware that can fit inside of a smartphone. Unlike earlier programs, the automatic photo software from MIT and Google looks to solve one of computational photography’s biggest obstacles, the limited processing power of a mobile device. In the researcher’s experiment, the program applied a high-dynamic range algorithm in real time, boosting the image’s colors and range of light beyond what the hardware itself could achieve.

“This technology has the potential to be very useful for real-time image enhancement on mobile platforms,” said Jon Barron, a Google researcher who worked on the project. “Using machine learning for computational photography is an exciting prospect but is limited by the severe computational and power constraints of mobile phones. This paper may provide us with a way to sidestep these issues and produce new, compelling, real-time photographic experiences without draining your battery or giving you a laggy viewfinder experience.”

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Astronaut’s stunning photo would look just fine in an art gallery
A star trail captured from the ISS.

In his final days aboard the International Space Station (ISS) in a mission that’s lasted six months, American astronaut Don Pettit has posted a sublime shot that wouldn’t look out of place in an art gallery.

The extraordinary image was captured through a window of a SpaceX Crew Dragon spacecraft docked at the ISS, and shows star trails and city lights on Earth some 250 miles below.

Read more
Cool space video shows star trails stretching over city lights
A screen grab from a video showing star trails stretching over city lights on Earth.

NASA astronaut Don Pettit dazzled us on Tuesday with a gorgeous image showing star trails and city lights as seen from the International Space Station (ISS).

On Wednesday, he turned the "wow" dial all the way up to 11 by sharing a video clip from which Tuesday’s image was taken.

Read more
Blue Ghost’s moon landing shown in dramatic new footage
Images of the lunar surface captured by the Blue Ghost lander.

When the Blue Ghost lander reached the lunar surface on March 2, Firefly Aerospace become the first commercial company to fully succeed in a soft moon landing at the first attempt.

A week on, and a team at NASA’s Langley Research Center in Virginia has just released first-of-its-kind footage of a lunar lander’s powerful engine plumes interacting with the moon’s surface, captured as the Blue Ghost touched down.

Read more