Skip to main content

Amazing new video tweaking program lets users edit a person's lip movements in real time

Face2Face: Real-time Face Capture and Reenactment of RGB Videos (CVPR 2016 Oral)
Thanks to rampant photo editing and fake images circulating the web these days, photographic evidence isn’t necessarily valid at face value anymore. And now, thanks to a system that will let anyone dub a video to change the very words a recorded person says, the same suspicion also holds true for videos. Face2Face, as it’s called, is a new software program that makes convincing edits to the moving lips of a person who is speaking. That way, existing videos will suddenly look like the person really is saying the audio dubbed (and incorrect) words. Real and fake will be nearly indistinguishable, on the surface.

Face2Face is still a work in progress, with a team of researchers from Stanford, the Max Planck Institute, and the University of Erlangen-Nuremberg constantly improving the program. The working prototype allows anyone to use a standard RGB webcam to capture video footage of their face, either saying words or making facial expressions. Pairing that footage with video footage of a celebrity, president, or public figure in the Face2Face program will make it look like the person is saying pretty much anything you want them to.

Matthias Niessner

The system pairs two monocular video streams (captured with a single webcam or video source) to make the finished video dub. Face2Face tracks facial expressions of the source video and the target video in order to warp the mouth movement in the final video into something believable. The system then combines the two feeds using a “dense photometric consistency measure,” and re-renders the source expression onto the target face in the original video setting. It’s scary, but even from an unfinished prototype of the system, the Face2Face edited video is pretty convincing.

Recommended Videos

The challenge to making convincing video may have a lot to do with the target star’s popularity level – if someone we’re used to seeing on TV all the time suddenly makes facial expressions that are new to the repertoire, viewers’ brains will raise some red flags automatically. Nonetheless, some savvy video hackers could theoretically make very believable video and audio dubs of perfectly legitimate speeches or broadcast appearances. Some people are afraid that this kind of system poses a threat to technology by making it inherently less reliable or believable, but the truth is that hoaxes in every medium have been around for ages.

Please enable Javascript to view this content

There’s no telling what this software might eventually be used for — but we’re definitely looking forward to all the hilarious speech remixes that will undoubtedly hit YouTube once this software goes mainstream.

Chloe Olewitz
Former Digital Trends Contributor
Chloe is a writer from New York with a passion for technology, travel, and playing devil's advocate. You can find out more…
Hyundai Ioniq 5 sets world record for greatest altitude change
hyundai ioniq 5 world record altitude change mk02 detail kv

When the Guinness World Records (GWR) book was launched in 1955, the idea was to compile facts and figures that could finally settle often endless arguments in the U.K.’s many pubs.

It quickly evolved into a yearly compilation of world records, big and small, including last year's largest grilled cheese sandwich in the world.

Read more
Global EV sales expected to rise 30% in 2025, S&P Global says
ev sales up 30 percent 2025 byd sealion 7 1stbanner l

While trade wars, tariffs, and wavering subsidies are very much in the cards for the auto industry in 2025, global sales of electric vehicles (EVs) are still expected to rise substantially next year, according to S&P Global Mobility.

"2025 is shaping up to be ultra-challenging for the auto industry, as key regional demand factors limit demand potential and the new U.S. administration adds fresh uncertainty from day one," says Colin Couchman, executive director of global light vehicle forecasting for S&P Global Mobility.

Read more
Faraday Future could unveil lowest-priced EV yet at CES 2025
Faraday Future FF 91

Given existing tariffs and what’s in store from the Trump administration, you’d be forgiven for thinking the global race toward lower electric vehicle (EV) prices will not reach U.S. shores in 2025.

After all, Chinese manufacturers, who sell the least expensive EVs globally, have shelved plans to enter the U.S. market after 100% tariffs were imposed on China-made EVs in September.

Read more