Skip to main content

YouTubers can soon replace ugly video backgrounds — no green screen required

Google Research
Google Research

Replacing the background on a video typically requires advanced desktop software and plenty of free time, or a full-fledged studio with a green screen — but Google is developing an artificial intelligence alternative that works in real time, from a smartphone camera. On Thursday, March 1, Google shared the tech behind a new feature that’s now being beta tested on the YouTube Stories app.

Google’s video segmentation technique started out like most A.I.-based imaging programs — with people manually marking the background by hand on more than 10,000 images. With that data set, the group trained the program to separate the background from the foreground. (Adobe has a similar background removal tool now inside Photoshop for still images.)

That data set, however, was trained on single images — so how did Google go from a single image to replacing the background on every frame of video? Once the software masks out the background on the first image, the program uses that same mask to predict the background in the next frame. When that next frame has only minor adjustments from the first, such as slight movement from the person talking on camera, the program will make small adjustments to the mask. When the next frame is much different from the last — like if another person joins the video — then the software will discard that mask prediction entirely and create a new mask.

While the ability to separate the background is impressive by itself, Google wanted to go one step further and make that program run on the more limited hardware on a smartphone rather than a desktop computer. The programmers behind the video segmentation then made several adjustments to the program in order to improve the speed, including segmentation, downsampling, and reducing the number of channels. The team then worked to enhance quality by adding layers to create smoother edges between the foreground and background.

Google says those changes allow the app to replace the background in real time — applying changes at over 100 fps on an iPhone 7 and over 40 fps on the Google Pixel 2. The training set, Google says, had a 94.8 percent accuracy rating. All of the examples Google shared are videos of a person — the company didn’t say if the feature works on objects.

Inside the beta test of the feature, YouTubers can change the background in the video by selecting from the different effects, from a night scene to a blank background in black and white. Some of the effects in the sample even added lighting effects, like a lens flare at one corner.

The video segmentation tool is already rolling out — but only as a beta test, so the feature isn’t yet widely available. After taking in results of that test though, Google says it plans to expand the segmentation effects along with adding the feature to other programs, including augmented reality options.

Editors' Recommendations

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
YouTube brings pinch to zoom and video navigation changes to everyone
The red and white YouTube logo on a phone screen. The phone is on a white background.

YouTube is updating its user interface with a slew of changes, and chief among them are the pinch-to-zoom feature and "precise" video navigation.

On Monday, YouTube announced quite a few updates to its viewing experience on mobile and web. Notably, the video-sharing platform said that it was finally "launching pinch to zoom and precise seeking to all users starting today."

Read more
Searches for health topics on YouTube now highlights personal stories
The red and white YouTube logo on a phone screen. The phone is on a white background.

Google and TikTok aren't the only places people look for information on health issues. YouTube is another resource people look to for educating themselves on health-related topics. Now, YouTube has launched a new feature in an attempt to further support those queries in a different way.

On Wednesday, the video-sharing website announced its latest feature via a blog post. Known as a Personal Stories shelf, the new search-related feature will yield a "shelf" of personal story videos about the health topics users search for. Essentially, if you search for a health topic, a Personal Stories shelf may appear in your search results and it will be populated with YouTube videos that feature personal stories about people who have experienced the health issue you searched for.

Read more
Save 35% on this SanDisk 128GB SD card for a limited time
The 128GB version of the SanDisk Extreme Pro SD card, on a white background.

When you buy from camera deals, you should also purchase an SD card or two to make sure that you have ample storage for your photos and videos. Unfortunately, the costs will start racking up if you're also going to buy accessories, so you should be on the lookout for offers like this one from StackSocial -- the SanDisk Extreme Pro 128GB for only $20, following a 35% discount on its original price of $31. That's $11 in savings on a dependable SD card, but you'll need to be quick in completing the transaction because there's no telling when the bargain ends.

Why you should buy the SanDisk Extreme Pro 128GB SD card
DSLR cameras and mirrorless cameras usually use SD cards as their storage devices, according to our guide on how to pick the right memory card for your digital camera. If you need one, the SanDisk Extreme Pro is an excellent choice because it offers shot speeds of up to 90 MB/s, which is perfect for recording 4K Ultra HD videos, photos in burst mode, and other types of content that will require a high-performance SD card to keep up with them.

Read more