It would be an understatement to say that the video content industry is currently at an inflection point. On one hand, we have AI supercharging the creative potential of content creators, but on the other side of the ocean, the problem of AI slop and misinformation lingers. The sheer potential of AI, however, can’t be ignored.
The folks over at YouTube are putting it to good use with a focus on accessibility and realism. So, what’s next? Making the lips move naturally to the tune of any language, even if the speaker in the video doesn’t speak it. Building on the auto-dubbing feature that was launched last year, the team has now come up with the new AI-powered lip sync feature.
Machine-translated audio has improved dramatically over the past few quarters, and it now almost sounds natural. Audio overviews in Google’s NotebookLM are a great example. But when it comes to videos, they fall flat because the lip movement simply doesn’t match what the speaker is saying with a translated version of the script.
It’s pretty jarring and off-putting. The AI-powered lip sync feature wants to overcome that audio-visual dissonance. And from the samples that I’ve seen so far, they feel uncannily natural. I sat down with YouTube Product Lead, Autodubbing, Buddhika Kottahachchi, to understand how lip sync was developed, its impact, and the road ahead.
Digging into the technical side
In less than a year since its launch, YouTube’s auto-dubbing feature has been used to dub over 60 million videos across 20 languages. But preserving a natural tone with all the nuances of a conversational speech, and then matching it with realistic lip movements, is a whole new challenge.
On the surface level, Kottahachchi tells me that the lip sync system “modifies the pixels on the screen to match the translated speech.” It’s a custom tech stack, the Google executive tells me, adding that they needed to develop a 3D understanding of the world, lip shapes, teeth, posture, and face.
For now, the tech is suited for full-HD (1080), but not tuned to 4K videos, as of now. “But generally, it should work with the video resolutions that you upload,” he points out. As far as the language support goes, YouTube’s AI-powered lip sync feature supports English, Spanish, German, Portuguese, and French.
That’s a pretty restricted pool, but Kottahachchi tells me that the team is scaling up and lip sync will eventually support the same set of languages as the auto-dubbing feature can handle (which currently stands at over 20 languages). For comparison, Meta’s AI-fueled lip sync feature for Facebook and Instagram supports only English, Spanish, Hindi, and Portuguese.
Now, AI-powered lip syncing is not entirely an alien concept. Adobe already offers an auto lip sync functionality. Then there are third-party options such as HeyGen, which claim to do it for free. But when it comes to YouTube, we are talking about a built-in system at a massive scale on a platform where 20 million videos are uploaded on a daily basis.
The AI Babel fish for your face
So, what’s next in terms of availability? “We are not ready to make any broad statements about how broadly we will make it available, but we do want to make it available to more creators and understand the compute constraints and the quality,” Kottahachchi tells me. And that brings us to the crucial cost question.
When I enquired about it, the YouTube executive told me that they can’t make predictions about the fee involved, if at all. That also explains why the feature is still part of a pilot project among a small pool of trusted testers to understand the market and compute costs. To recall, this is a complex vision-based implementation of AI.
So, just like AI-generated videos, where you can create a few clips for free but need to pay up for higher resolution or attempts, YouTube will have to factor in the compute costs and decide on the rollout. But from a creator’s perspective, if I am chasing a wider reach, I’d likely pay the subscription fee.
The AI dilemma
Ever since AI visuals started flooding the internet, the debate around authenticity and fair disclosure has heated up. “What is even real?” Social media users have been asking that question with more fervor soon after the uncannily realistic videos generated by OpenAI’s Sora app started popping up.
These videos have a visible watermark, but there are already free and paid tools out there that will remove the Sora label from the AI-generated clips. Or any other AI content generator, for that matter. Google, one of the biggest developers and adopters of AI, knows that all too well.
The company was one of the early leaders in the AI fingerprinting race with its SynthID system, and also launched a SynthID Detector tool earlier this year to help users check the origins of multimedia content.
The YouTube videos that rely on Google’s AI-powered lip sync feature will take an even more cautious approach. “We will have a proper disclosure saying that both the audio and video in this video have been synthetically created or altered,” Kottahachchi tells me. “The video content, itself, gets fingerprinted, as well.”
The text disclosures will appear in the description box underneath the title of YouTube videos, just as they appear for videos that have used the auto dub system. But how are other platforms going to treat the AI-dubbed, lip-synced YouTube videos if a creator posts them on Instagram or TikTok?
Will the algorithms warm up?
TikTok recently announced that it would label videos that were “made or edited” using AI tools, and would also fingerprint them so that users can check their origins using C2PA’s Verify tool. Meta has a similar system in place. So, what’s the fate of AI-edited videos that are cross-posted to other social video platforms?
Will they be algorithmically downranked, or blocked from appearing in certain feeds? The situation is a bit tricky and unpredictable. “It’s something we are monitoring carefully, but it’s a little early because platforms have made statements, but we haven’t seen how they are effectively implemented,” he tells me. “Generally, we are translating translations, but not new content.”
I also brought up the issue with bad actors using videos of creators without their due consent, translating the audio, and pushing them from a different channel, or platform. Auto-dubbing and AI lip syncing technically make that unscrupulous act easier to execute, but it likely won’t devolve into total chaos.
“If your likeness is being used somewhere else on the platform, you can tell us about it and ask us to take it down,” Kottahachchi told me. It would be interesting to see how auto-dubbing, expressive audio, and lip-synced videos will make the YouTube experience more diverse. On the surface, it seems like a win.
I can’t wait to see myself speak in Spanish, though I abandoned my Duolingo streak years ago.
