Skip to main content

Adobe and Stanford just taught AI to edit videos — with impressive results

Computational Video Editing for Dialogue-Driven Scenes (Full Video)
Just one minute of video typically takes several hours of editing — but Stanford and Adobe researchers have developed an artificial intelligence (AI) program that partially automates the editing process, while still giving the user creative control over the final result.

The program starts by organizing all of the footage, which is often from multiple takes and camera angles. Those clips are matched to the script, so it’s easy to find several video options for each line of dialogue.

Recommended Videos

The program then works to recognize exactly what is inside those clips. Using facial recognition alongside emotion recognition and other computational imaging effects, the program determines what is in each frame. For example, the program flags whether the shot is a wide-angle or a close-up and which characters the shot includes.

Please enable Javascript to view this content

With everything organized, the video editor then instructs the program in just how the videos should be edited using different styles and techniques the researchers call idioms. For example, a common style is to show the face of the character during their lines. If the editor wants that to happen, he or she just drags that idiom over. The idioms can also be negative. For example, the idiom “avoid jump cuts,” can be added to actually avoid them, or negatively to intentionally add jump cuts whenever possible.

The editor can drag over multiple idioms to instruct the program on an editing style. In a video demonstrating the technology, the researchers created a cinematic edit by using idioms that tell the software to keep the speaker visible while talking, to start with a wide-angle shot, to mix with close-ups and to avoid jump cuts. To edit the video in a completely different, fast-paced style, the researchers instead dragged over idioms for including jump cuts, using fast performance, and keeping the zoom consistent.

Editing styles can be saved to recall later, and with the idioms in place, a stylized video edit is generated with a click. Alternative clips are arranged next to the computer’s edit so editors can quickly adjust if something’s not quite right.

The program speeds up video editing using artificial intelligence, but also allows actual humans to set the creative parameters in order to achieve a certain style. The researchers did acknowledge a few shortcomings of the program. The system is designed for dialogue-based videos, and further work would need to be done for the program to work with other types of shots, such as action shots. The program also couldn’t prevent continuity errors, where the actor’s hands or a prop is in a different location in the next clip.

The study, conducted by Stanford University and Adobe Research, is included in the July issue of the ACM Transactions on Graphics Journal.

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Forget Dall-E, you can sign up to create AI-generated videos now
A frame from an AI-generated video in claymation style.

Dall-E, ChatGPT, and other AI-generation technologies continue to amaze us. Still, AI image-generation tools like Midjourney might seem boring once you see the new, AI-powered video-generation abilities that will soon be available to us all.

Runway provides an advanced online video editor that offers many of the same features as a desktop app. The company has distinguished its service from others, however, by pioneering the use of AI tools that help with various time-consuming video chores, such as masking out the background.

Read more
You can now export videos 10x faster in Adobe Premiere Pro
Man using Adobe Premiere Pro on iMac

Adobe has just announced that Premiere Pro users will now be able to export videos up to 10 times faster than previously once they update to Premiere Pro 22.4.0. While the improvements won’t benefit all users, they offer significant improvements if you’re using a supported system.

In a blog post detailing the updates coming to Premiere Pro in May, Adobe explained that the changes affect Apple users whose Macs are powered by the M1 Pro, M1 Max, and M1 Ultra chips, as well as Windows PCs equipped with AMD graphics cards. The changes specifically impact 10-bit 4:2:0 HEVC footage.

Read more
Adobe brings real-time editing to Premiere Pro, After Effects
Someone using the iMac Pro.

Following its acquisition of Frame.io late last year, Adobe is finally making it easier for editors to collaborate on video projects by integrating tools from its newly acquired company into Adobe Premiere Pro.

As part of its announcement at the National Association of Broadcasters Show in Las Vegas, Adobe showed off a streamlined workflow that's designed to speed up the video-editing and approval processes thanks to Frame.io's integrated ability to quickly make edits to video -- even while the camera is still capturing footage -- quickly upload the edits for approval, and push the final product live to various video and social media platforms.

Read more