Make Videos Say Whatever You Want With Face2Face

Amazing new video tweaking program lets users edit a person's lip movements in real time

Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral)

Thanks to rampant photo editing and fake images circulating the web these days, photographic evidence isn’t necessarily valid at face value anymore. And now, thanks to a system that will let anyone dub a video to change the very words a recorded person says, the same suspicion also holds true for videos. Face2Face, as it’s called, is a new software program that makes convincing edits to the moving lips of a person who is speaking. That way, existing videos will suddenly look like the person really is saying the audio dubbed (and incorrect) words. Real and fake will be nearly indistinguishable, on the surface.

Face2Face is still a work in progress, with a team of researchers from Stanford, the Max Planck Institute, and the University of Erlangen-Nuremberg constantly improving the program. The working prototype allows anyone to use a standard RGB webcam to capture video footage of their face, either saying words or making facial expressions. Pairing that footage with video footage of a celebrity, president, or public figure in the Face2Face program will make it look like the person is saying pretty much anything you want them to.

The system pairs two monocular video streams (captured with a single webcam or video source) to make the finished video dub. Face2Face tracks facial expressions of the source video and the target video in order to warp the mouth movement in the final video into something believable. The system then combines the two feeds using a “dense photometric consistency measure,” and re-renders the source expression onto the target face in the original video setting. It’s scary, but even from an unfinished prototype of the system, the Face2Face edited video is pretty convincing.

The challenge to making convincing video may have a lot to do with the target star’s popularity level – if someone we’re used to seeing on TV all the time suddenly makes facial expressions that are new to the repertoire, viewers’ brains will raise some red flags automatically. Nonetheless, some savvy video hackers could theoretically make very believable video and audio dubs of perfectly legitimate speeches or broadcast appearances. Some people are afraid that this kind of system poses a threat to technology by making it inherently less reliable or believable, but the truth is that hoaxes in every medium have been around for ages.

There’s no telling what this software might eventually be used for — but we’re definitely looking forward to all the hilarious speech remixes that will undoubtedly hit YouTube once this software goes mainstream.

Editors' Recommendations