Skip to main content

Give YouTube a round of [APPLAUSE] — sound effects are now auto captioned

YouTube deserves a round of applause — because the video platform can now include [APPLAUSE] and other sound effects in a video’s closed captions automatically. The caption expansion, announced on Thursday, March 23, is made possible by deep neural networks, a form of artificial intelligence.

For now, YouTube can only automatically label applause, music, and laughter, but those three sound effects were the descriptions content creators added manually over any other closed captioning noise. The latest feature builds on the automatic captioning feature launched in 2009 for text, but adds the first sound effects to the system.

Recommended Videos

YouTube says the program works similarly to detecting objects in images, but faced a few more difficulties over object recognition. To get the program to recognize just those three sounds, YouTube engineers had to teach the program to detect those sounds, separate them temporarily and then insert that recognized sound into the captions.

The system also tended to struggle with sound effects that occurred at the same time as other sounds, like laughter and talking. Another challenge was to find a large enough data set to train the system that wasn’t already adequately labeled by manually inputting the data.

The deep learning network analyzes short segments in sequence, and is able to predict the likelihood of those sounds effects at a rate of about 100 frames per second. YouTube engineers, however, built the system in a way that will allow additional sound effects to be added to the system later.

So why applause, music, and laughter? Besides just being the most frequently manually adjusted labels in the close captioning system, each of those sounds also only has one meaning. A “ring,” YouTube explained, in offering an example, could be a ring from a doorbell, a phone, or an alarm, presenting a whole new challenge for the software.

According to YouTube, over 15 million videos with automatic captions are viewed every day. In a test of the latest update to the auto captions, two thirds said the sound effect labels enhanced the overall experience.

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
YouTube rolling out some three-dozen new features this fall
The Digital Trends YouTube channel as seen on an iPhone and on a TV in the background.

The tweaks keep coming to the world's largest video provide. Today YouTube is taking the wraps off some three-dozen (more or less) new features. It's a cross-platform announcement, with the goods hitting phones and tablets, as well as televisions and wherever else you do your YouTube viewing.

Here's what you have to look forward to:

Read more
YouTube makes it easier for new creators to earn money
A young woman with a smartphone walks past signage of the Youtube logo.

The top YouTubers make eye-watering sums of money, with a Forbes list released last year revealing that the top 10 highest earners on the video streaming platform raked in a colossal $300 million between them in the space of a year.

But while the top YouTubers grab all the headlines, there are millions of other creators on the platform working diligently to build up loyal followings with carefully produced content. And while they may have fewer subscribers than YouTube’s big hitters, and get fewer views for each video, they can still earn a tidy sum that allows them to give up the day job, or at the least supplement their main income.

Read more
YouTube Stories are going away starting June 26
The Digital Trends YouTube channel on an iPhone.

YouTube today announced that it's going to kill off its Story feature — like the similarly named Instagram Stories, basically its answer to Snapchat — starting June 26. That's the last day you'll be able to post a new YouTube Story. And seven days after that, any story that already was live will die an unceremonious death.

That doesn't mean there won't be an alternative to a full-blown YouTube video or a smaller YouTube Short. (Which is, in and of itself, YouTube's answer to Tiktok.) YouTube is pointing creators to "YouTube Community posts" instead, which it says "are a great choice if you want to share lightweight updates, start conversations, or promote your YouTube content to your audience." Community posts essentially are ephemeral updates that also allow for text, polls, quizzes, filters, and stickers.  It added that "amongst creators who use both posts and Stories, posts on average drive many times more comments and likes compared to Stories."

Read more