Skip to main content

Adobe and Stanford just taught AI to edit videos — with impressive results

Computational Video Editing for Dialogue-Driven Scenes (Full Video)
Just one minute of video typically takes several hours of editing — but Stanford and Adobe researchers have developed an artificial intelligence (AI) program that partially automates the editing process, while still giving the user creative control over the final result.

The program starts by organizing all of the footage, which is often from multiple takes and camera angles. Those clips are matched to the script, so it’s easy to find several video options for each line of dialogue.

The program then works to recognize exactly what is inside those clips. Using facial recognition alongside emotion recognition and other computational imaging effects, the program determines what is in each frame. For example, the program flags whether the shot is a wide-angle or a close-up and which characters the shot includes.

With everything organized, the video editor then instructs the program in just how the videos should be edited using different styles and techniques the researchers call idioms. For example, a common style is to show the face of the character during their lines. If the editor wants that to happen, he or she just drags that idiom over. The idioms can also be negative. For example, the idiom “avoid jump cuts,” can be added to actually avoid them, or negatively to intentionally add jump cuts whenever possible.

The editor can drag over multiple idioms to instruct the program on an editing style. In a video demonstrating the technology, the researchers created a cinematic edit by using idioms that tell the software to keep the speaker visible while talking, to start with a wide-angle shot, to mix with close-ups and to avoid jump cuts. To edit the video in a completely different, fast-paced style, the researchers instead dragged over idioms for including jump cuts, using fast performance, and keeping the zoom consistent.

Editing styles can be saved to recall later, and with the idioms in place, a stylized video edit is generated with a click. Alternative clips are arranged next to the computer’s edit so editors can quickly adjust if something’s not quite right.

The program speeds up video editing using artificial intelligence, but also allows actual humans to set the creative parameters in order to achieve a certain style. The researchers did acknowledge a few shortcomings of the program. The system is designed for dialogue-based videos, and further work would need to be done for the program to work with other types of shots, such as action shots. The program also couldn’t prevent continuity errors, where the actor’s hands or a prop is in a different location in the next clip.

The study, conducted by Stanford University and Adobe Research, is included in the July issue of the ACM Transactions on Graphics Journal.

Editors' Recommendations

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Adobe Premiere Rush now allows Android users to edit video without the laptop
adobe premiere rush android launch lizzy pierce

Adobe’s new video app Premiere Rush is designed for streamlining video editing -- and now that workflow is available for Android users. Adobe expanded Premiere Rush to the Android operating system, opening up downloads of the video editor through the Google Play Store and Samsung Galaxy App Store.

Adobe Rush brings several of the tools from Adobe’s pro-level video editor, Premiere, to a streamlined platform. While missing a handful of the more high-end features, Rush is designed for editing social media projects, as well as for newbie editors. 

Read more
‘Photoshopped’ royal photo causes a stir
The Princess of Wales with her children.

[UPDATE: In a message posted on social media on Monday morning, Princess Kate said that she herself edited the image, and apologized for the fuss that the picture had caused. “Like many amateur photographers, I do occasionally experiment with editing," she wrote, adding, "I wanted to express my apologies for any confusion the family photograph we shared yesterday caused."]

Major press agencies have pulled a photo of the U.K.’s Princess of Wales and her children amid concerns that it has been digitally manipulated.

Read more
Help NASA in its quest to learn more about our sun
Scientists have used the ESA/NASA Solar Orbiter’s Extreme Ultraviolet Imager (EUI) in a new mode of operation to record part of the Sun’s atmosphere that has been almost impossible to image until now. By covering the Sun’s bright disc with an ‘occulter’ inside the instrument, EUI can detect the million-times fainter ultraviolet light coming from the surrounding corona.

NASA is calling on citizen astronomers in the U.S. to help it learn more about our sun.

It’s using April’s solar eclipse as an opportunity to conduct a special experiment in which millions of people can participate.

Read more