Skip to main content

A Netflix data scientist taught an A.I. to recognize smooching scenes in movies

A senior data scientist at Netflix has taught an artificial intelligence (A.I.) algorithm to be able to recognize smooches. Amir Ziai developed the tool, which is able to watch movies and recognize scenes in which the characters lock lips, as part of his work to obtain an A.I. graduate certificate from Stanford University.

He selected kissing scenes because, he told Digital Trends, they can be tough to detect using traditional video-processing techniques. That’s because machines get easily confused and produce false positives when seeing other scenarios in which two people have their heads close to one another — such as talking scenes or ones in which characters walk in close proximity.

Recommended Videos

“Training was done using a database of Hollywood movies spanning multiple decades and genres,” Ziai said. “I annotated kissing and non-kissing segments in 100 of these movies, and used those segments to train a multimodal neural network that uses both audio and visual features from 1-second segments. The major challenge with training these models is twofold. First, I had to make sure that I’ve annotated a representative set of training examples that’ll help with generalizing to a diverse set of movies. Second, training deep-learning models on video can be very resource intensive.”

Please enable Javascript to view this content

Nonetheless, he pulled off the feat, and the resulting tool turns out to be impressively accurate. The system employs a two-phase process. First, it uses a binary classifier to predict whether kissing is or isn’t taking place using features extracted from still frames and audio waves. The second component then aggregates the binary labels for “contiguous non-overlapping segments” into a set of kissing scenes. The final result achieves a validation F1 score of 0.95 on a diverse database of movies.

Ziai, it should be noted, isn’t the only person interested in getting machine intelligence to recognize kisses. Recently, Google unveiled a new feature for its Photobooth tool that prompts Pixel smartphones to automatically take photos when they recognize that the subjects in a frame are kissing.

“System like the kissing detector can be used to automatically add metadata to movies,” Ziai said. “This metadata can be used to search and retrieve relevant snippets. For example, a video editor can use such metadata to quickly find relevant segments and to speed up the process of editing a movie.”

A paper describing the work, titled Detecting Kissing Scenes in a Database of Hollywood Films, is available to read on the preprint server arXiv.

Luke Dormehl
Former Digital Trends Contributor
I'm a UK-based tech writer covering Cool Tech at Digital Trends. I've also written for Fast Company, Wired, the Guardian…
The best Netflix original movies right now
Cameron Diaz and Jamie Foxx in Back in Action.

One thing is certain about this month's selection of original movies on Netflix: They aren't Oscar bait. Back in Action is an action comedy that's purely a crowd-pleaser, but Netflix thrives on films like that. It's the equivalent of having a modest box office hit that suddenly shows up in your Netflix feed.
Our other pick for the month, Don't Die: The Man Who Wants to Live Forever, is a documentary about Bryan Johnson, a man who has taken some very aggressive steps to lengthen his life. What Johnson —and by extension, the viewers and the filmmakers — probably didn't expect is that the movie underlines the personal cost of his quest beyond the money that he invested in himself.

You can find these films and the rest of the best Netflix original movies below.

Read more
Netflix 2025 movie slate features Knives Out 3, Frankenstein, Happy Gilmore 2, and more
Daniel Craig and Josh O'Connor stand in a church in Knives Out 3.

Netflix has revealed its 2025 movie slate, including anticipated films such as Knives Out 3, Frankenstein, Happy Gilmore 2, and more. The 2025 film and TV schedules were announced at the streamer's star-studded "Next on Netflix" event.

Daniel Craig's Benoit Blanc returns for another mystery in Wake Up Dead Man: A Knives Out Mystery. The third Knives Out installment from Rian Johnson promises to be Blanc's "most dangerous case yet." Besides Craig, Knives Out 3 stars Josh O’Connor, Glenn Close, Josh Brolin, Mila Kunis, Jeremy Renner, Kerry Washington, Andrew Scott, Cailee Spaeny, Daryl McCormack, and Thomas Haden Church. Wake Up Dead will premiere this fall.

Read more
Stop! And watch these 3 great movies leaving Netflix by February 1
Three teens stand in front of a fire in Project X.

Netflix started January off strong with the new movie Back in Action, an action comedy starring Jamie Foxx and Cameron Diaz. Back in Action has been a resounding hit, registering the biggest Netflix premiere weekend since The Adam Project. While Back in Action will remain in the service, several notable movies will leave by February 1.
The list of movies leaving Netflix this month includes We're the Millers, Zero Dark Thirty, Dune, Despicable Me, and White House Down. The list continues below with three more movies: an action comedy, a high school party feature, and a sci-fi thriller.
We also have guides to the best movies on Netflix, the best movies on Hulu, the best movies on Amazon Prime Video, the best movies on Max, and the best movies on Disney+.
22 Jump Street (2014)

First, they conquered prom. Now, it's time to dominate spring break. When we last saw Schmidt (Jonah Hill) and Jenko (Channing Tatum) in 21 Jump Street, they were infiltrating a high school drug ring. Two years later, the pair are asked to go undercover again. Instead of high school, the boys are heading to college in 22 Jump Street.
The unlikely pairing is looking for the supplier of a synthetic drug that killed a student. Jenko embraces his inner jock and joins the football team, while Schmidt enters the performing arts. A lot of the same humor used in 21 Jump Street is applied to 22 Jump Street. However, Hill and Tatum remain as charming as ever in this successful sequel.

Read more