Artificial intelligence is now smart enough to paint in the missing regions of a face to fix a blink or turn a frown into a smile. The problem? Those A.I. programs use a stranger’s photograph, so while the programs may fix a bad blink, they also create a Frankenstein of you with someone else’s eyes. Facebook researchers, however, may have come up with a solution by training an A.I. that uses your actual eyes from a previous photograph to fix that blink. Facebook published the research on June 18.
Repairing a photograph with A.I. is nothing new — earlier this year, Nvidia researchers created an A.I. healing tool that could replace missing pixels in a portrait, including missing eyes. While those programs can fix something like a blink, the A.I. relied on a database of images of other people, painting in someone else’s eyes to fix that blink. Besides using someone else’s eyes, the technology also tended to be biased toward certain shapes and colors, Facebook suggests.
But Facebook probably already has a record of your face (or a few hundred selfies), so the company’s researchers decided to try to fix that awkward blink without making it look like you got a new pair of eyes. The researchers created two different approaches to the technique, called eye in-painting. The first uses a reference image to replace the eyes. The second stores an actual version of your eyes, similar to how Facebook stores data for facial recognition, to use in those fixes.
While many A.I. programs work with existing databases of photos to train the program, Facebook created its own data set. The set of celebrity photos taken on a runway by professional photographers didn’t match the badly lit selfies the program is more likely to encounter, while other datasets contained images with people wearing sunglasses. Instead, Facebook created a dataset of around 200,000 people, with at least three images of each person.
The program bases the results on the reference eye in order to help preserve identity, while the A.I. helps create a more realistic result. For example, the network used the data photos to learn how to create a realistic result when the pose or lighting in the reference eyes and the edited photos are different.
The resulting program, Facebook says, fixes eye blinks with results that look more accurately like that specific individual and without losing as much data as earlier programs. The results aren’t perfect — for individuals with hair obscuring part of one eye, for example, the program didn’t accurately recreate the shape of the eye, while in other scenarios, the eye color had slight variations.
Facebook isn’t done researching the possibility of automatically fixing those blinking selfies, but with some additional work, a future version of the social network could do much more with automatic photo fixes before sharing.
