YouTubers can soon replace ugly video backgrounds — no green screen required

google background swap youtube stories youtubebackground
Google Research
Google Research

Replacing the background on a video typically requires advanced desktop software and plenty of free time, or a full-fledged studio with a green screen — but Google is developing an artificial intelligence alternative that works in real time, from a smartphone camera. On Thursday, March 1, Google shared the tech behind a new feature that’s now being beta tested on the YouTube Stories app.

Google’s video segmentation technique started out like most A.I.-based imaging programs — with people manually marking the background by hand on more than 10,000 images. With that data set, the group trained the program to separate the background from the foreground. (Adobe has a similar background removal tool now inside Photoshop for still images.)

That data set, however, was trained on single images — so how did Google go from a single image to replacing the background on every frame of video? Once the software masks out the background on the first image, the program uses that same mask to predict the background in the next frame. When that next frame has only minor adjustments from the first, such as slight movement from the person talking on camera, the program will make small adjustments to the mask. When the next frame is much different from the last — like if another person joins the video — then the software will discard that mask prediction entirely and create a new mask.

While the ability to separate the background is impressive by itself, Google wanted to go one step further and make that program run on the more limited hardware on a smartphone rather than a desktop computer. The programmers behind the video segmentation then made several adjustments to the program in order to improve the speed, including segmentation, downsampling, and reducing the number of channels. The team then worked to enhance quality by adding layers to create smoother edges between the foreground and background.

Google says those changes allow the app to replace the background in real time — applying changes at over 100 fps on an iPhone 7 and over 40 fps on the Google Pixel 2. The training set, Google says, had a 94.8 percent accuracy rating. All of the examples Google shared are videos of a person — the company didn’t say if the feature works on objects.

Inside the beta test of the feature, YouTubers can change the background in the video by selecting from the different effects, from a night scene to a blank background in black and white. Some of the effects in the sample even added lighting effects, like a lens flare at one corner.

The video segmentation tool is already rolling out — but only as a beta test, so the feature isn’t yet widely available. After taking in results of that test though, Google says it plans to expand the segmentation effects along with adding the feature to other programs, including augmented reality options.

Computing

Adobe’s craziest new tools animate photos, convert recordings to music in a click

Adobe shared a glimpse behind the scenes at what's next and the Creative Cloud future is filled with crazy A.I.-powered tools, moving stills, and animation reacting to real-time tweets.
Product Review

Google’s Pixel 3 is a hair away from pocket-sized perfection

Google’s Pixel 3 smartphone is the best Android phone you can buy. It doesn’t have the best looks or the best hardware, but you’ll be hard pressed to find better software and unique A.I. functionalities.
Home Theater

GoPro changed how we watch. Now Jabra wants to change how we listen

Jabra and Red Bull spent two years developing the perfect action microphone, aiming to capture the sounds of athletes without wind or background noise. Here’s how the game-changing tech works, and where you’ll hear it in the future.
Photography

Adobe’s Premiere Rush is a video-editing app designed for social media projects

At Adobe MAX 2018, Adobe unveiled updates across the board for all of its Creative Cloud apps, from the release of Premiere Rush CC, a social-focused video editor, to Project Gemini, a digital drawing and painting tool.
Product Review

The design still says retro, but Fujifilm's X-T3 is all about the future

If the X-T2 brought Fujifilm into the modern era, the X-T3 is focused on the future. With a new sensor and processor, completely revamped autofocus, and vastly upgraded video, it's the new APS-C camera to beat.
Photography

When you're ready to shoot seriously, these are the best DSLRs you can buy

For many photographers the DSLR is the go-to camera. With large selection of lenses, great low-light performance, and battery endurance, these DSLRs deliver terrific image quality for stills and videos.
Mobile

Huawei and Leica’s monochrome lens is dead, so we celebrate its life

The Huawei Mate 20 and Mate 20 Pro do not have a dedicated monochrome camera lens, unlike the P20 Pro, and various Huawei and Leica phones before it. It's the end of an era, and also the start of a new one, as Leica has worked on its…
Photography

Remove photo bombs, other unwanted objects with Photoshop’s new Content-Aware Fill

Photoshop's newest A.I-powered tool helps remove objects or fill in gaps for a distraction-free photo in the new Adobe Photoshop CC 2019. Here's how to remove an object in Photoshop using the new Content-Aware Fill.
Photography

Adobe Premiere Rush CC is the cloud-based video editing app you've been waiting for

On stage at Adobe MAX 2018, Adobe announced its cloud-centric, social video-editing application, Adobe Premiere Rush CC. We took some time to put it through its paces to see what it offers, how it works, and what's missing.
Photography

Adobe MAX 2018: What it is, why it matters, and what to expect

Each year, Adobe uses its Adobe MAX conference to show off its latest apps, technologies, and tools to help simplify and improve the workflow of creatives the world over. Here's what you should expect from this year's conference.
Social Media

Over selfies and an onslaught of ads? Here's how delete your Instagram account

Despite its outstanding popularity and photo-sharing dominance, Instagram isn't for everyone. Thankfully, deleting your account is as easy as logging into the site and clicking a few buttons. Here's what you need to do.
Emerging Tech

Keep your holiday gift list high tech and under budget with these gadgets

Modern technology doesn't always come cheap, but there plenty of premium devices that don't carry a premium price. Whether you're looking for a streaming device or a means of capturing photos from above, our list of the best tech under $50…
Emerging Tech

The best drone photos from around the world

Most of today's drones come equipped with high-end cameras, which are quickly revolutionizing the world of aerial photography as we know it. Here are some of the best drone photos from around the world.
Photography

Camera shootout! Testing the latest Pixel, iPhone, and Galaxy Note in real life

Which takes the best photos, the Pixel 3 XL, iPhone XS Max, Galaxy Note 9, or Pixel 2 XL? We put the cameras on all these top-notch phones through their paces to see which performs best in the real world, from low light to portrait mode…