Skip to main content

Give YouTube a round of [APPLAUSE] — sound effects are now auto captioned

youtube tv live streaming service launch 28691186 ml
Image used with permission by copyright holder
YouTube deserves a round of applause — because the video platform can now include [APPLAUSE] and other sound effects in a video’s closed captions automatically. The caption expansion, announced on Thursday, March 23, is made possible by deep neural networks, a form of artificial intelligence.

For now, YouTube can only automatically label applause, music, and laughter, but those three sound effects were the descriptions content creators added manually over any other closed captioning noise. The latest feature builds on the automatic captioning feature launched in 2009 for text, but adds the first sound effects to the system.

YouTube says the program works similarly to detecting objects in images, but faced a few more difficulties over object recognition. To get the program to recognize just those three sounds, YouTube engineers had to teach the program to detect those sounds, separate them temporarily and then insert that recognized sound into the captions.

The system also tended to struggle with sound effects that occurred at the same time as other sounds, like laughter and talking. Another challenge was to find a large enough data set to train the system that wasn’t already adequately labeled by manually inputting the data.

The deep learning network analyzes short segments in sequence, and is able to predict the likelihood of those sounds effects at a rate of about 100 frames per second. YouTube engineers, however, built the system in a way that will allow additional sound effects to be added to the system later.

So why applause, music, and laughter? Besides just being the most frequently manually adjusted labels in the close captioning system, each of those sounds also only has one meaning. A “ring,” YouTube explained, in offering an example, could be a ring from a doorbell, a phone, or an alarm, presenting a whole new challenge for the software.

According to YouTube, over 15 million videos with automatic captions are viewed every day. In a test of the latest update to the auto captions, two thirds said the sound effect labels enhanced the overall experience.

Editors' Recommendations

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Don’t watch this YouTube video if you have a Pixel 7
Someone holding the Google Pixel 7 Pro.

Reports of another "cursed" piece of content have been making the internet rounds as a video on YouTube has been causing Pixel devices to crash. The video, a clip from the 1979 movie Alien, seems to cause Pixel 7, Pixel 7 Pro, and some Pixel 6 and Pixel 6a smartphones to instantly reboot without warning.

As first reported on Reddit and spotted by Mishaal Rahman, the video will begin to play for only a second or two and then instantly reboot the Pixel 7 it's being played on. Digital Trends can confirm the bug to be active and working, too, with the video instantly rebooting a Pixel 7 Pro we tested it on.

Read more
What is Ambient Mode on YouTube?
The red and white YouTube logo on a phone screen. The phone is on a white background.

YouTube recently announced a number of changes to its video-watching interface. One of these changes is a new feature known as Ambient Mode.

In this guide, we'll go over exactly what Ambient Mode on YouTube is and show you how to enable or disable it.
What is Ambient Mode on YouTube?
Ambient Mode is basically a lighting effect that surrounds a given YouTube video with a soft, glowing light that usually reflects the colors featured in the video itself. The effect kind of resembles the glow of a television screen in a room where the lights are off (which YouTube has said was the inspiration for the effect). YouTube also said that Ambient Mode is supposed to make the video-watching experience of its dark mode "more immersive." Ambient Mode was officially announced in late-October 2022 and is available on the web, Android, and iOS.

Read more
YouTube is rolling out handles. Here’s what you need to know
The red and white YouTube logo on a phone screen. The phone is on a white background.

YouTube recently announced that it would be rolling out handles for YouTube channels. Whether you've just heard about them or you already got an email from YouTube saying it's time to choose one, you're probably wondering what they are and how they're different from YouTube channel names.

In this guide, we'll go over what a YouTube handle is and answer a few questions about them so you can better understand what they are and be better prepared to choose one when it's rolled out to you.

Read more