A Netflix data scientist taught an A.I. to recognize smooching scenes in movies

A senior data scientist at Netflix has taught an artificial intelligence (A.I.) algorithm to be able to recognize smooches. Amir Ziai developed the tool, which is able to watch movies and recognize scenes in which the characters lock lips, as part of his work to obtain an A.I. graduate certificate from Stanford University.

He selected kissing scenes because, he told Digital Trends, they can be tough to detect using traditional video-processing techniques. That’s because machines get easily confused and produce false positives when seeing other scenarios in which two people have their heads close to one another — such as talking scenes or ones in which characters walk in close proximity.

“Training was done using a database of Hollywood movies spanning multiple decades and genres,” Ziai said. “I annotated kissing and non-kissing segments in 100 of these movies, and used those segments to train a multimodal neural network that uses both audio and visual features from 1-second segments. The major challenge with training these models is twofold. First, I had to make sure that I’ve annotated a representative set of training examples that’ll help with generalizing to a diverse set of movies. Second, training deep-learning models on video can be very resource intensive.”

Nonetheless, he pulled off the feat, and the resulting tool turns out to be impressively accurate. The system employs a two-phase process. First, it uses a binary classifier to predict whether kissing is or isn’t taking place using features extracted from still frames and audio waves. The second component then aggregates the binary labels for “contiguous non-overlapping segments” into a set of kissing scenes. The final result achieves a validation F1 score of 0.95 on a diverse database of movies.

Ziai, it should be noted, isn’t the only person interested in getting machine intelligence to recognize kisses. Recently, Google unveiled a new feature for its Photobooth tool that prompts Pixel smartphones to automatically take photos when they recognize that the subjects in a frame are kissing.

“System like the kissing detector can be used to automatically add metadata to movies,” Ziai said. “This metadata can be used to search and retrieve relevant snippets. For example, a video editor can use such metadata to quickly find relevant segments and to speed up the process of editing a movie.”

A paper describing the work, titled Detecting Kissing Scenes in a Database of Hollywood Films, is available to read on the preprint server arXiv.

Emerging Tech

IBM’s Wimbledon-watching A.I. is poised to revolutionize sports broadcasts

IBM has developed a smart A.I. with an appreciation for what makes a great tennis match like the recent epic at Wimbledon. Here's how IBM developed it -- and why tools like it are the future of sport broadcasting.
Mobile

A.I. photo filters use neural networks to make photos look like Picassos

Artificial Intelligence apps put a novel spin on social network-style photo filters. Instead of just overlaying an effect, these photo filters use machine learning and neural networks to transform an image into a predetermined style.
Emerging Tech

Elon Musk’s Neuralink wants to start operating on human brains next year

Elon Musk shed more light on his Neuralink company on Tuesday, revealing new technology for brain surgery that would allow people with paralysis to use thoughts to control smartphones and computers. But that wasn't all ...
Emerging Tech

Stallone in Terminator 2? How one deepfake prankster is changing cinema history

Ever wanted to see The Shining with Jim Carrey instead of Jack Nicholson? How about Stallone in Terminator 2: Judgement Day instead of everyone's favorite governator? Thanks to deepfakes, it's now possible -- just ask YouTuber Ctrl Shift…
Emerging Tech

Genetically modified plants could help get to the root of climate change

Researchers have been investigating ways to engineer plants so that they grow with more robust and deeper roots, capable of storing increasing amounts of carbon underground for longer.
Photography

With object tracking, the lightweight DJI Ronin-SC is still heavy on features

Designed for mirrorless cameras, the DJI Ronin-SC packs several features from the Ronin-S -- and then some -- into a lighter, one-handed gimbal. Despite the smaller size, the DJI Ronin-S adds new object tracking and expanded remote control.
News

SpaceX’s Starhopper rocket bursts into flames during tests

SpaceX ran into trouble Tuesday evening when a small fire erupted from the engine of a prototype rocket it was testing at the company’s facility in Boca Chica, Texas. It's not clear if the fire caused any damage to the rocket itself
Emerging Tech

Buying on a budget? Here's all the best tech you can snag for $25 or less

We live in a world where you can get a cheeseburger for $1, a functioning computer for $5, and thousands of HD movies for $10 -- so it stands to reason that you should be able to pick up some pretty sweet gear for $25.
Digital Trends Live

Digital Trends Live: Emoji Day, Apollo 11 broadcast, drone flamethrowers

On this episode of DT Live, we discuss the top stories in tech, including Emoji Day festivities, the extended battery life of the new Nintendo Switch, an Apollo 11 real-time broadcast, and a functional flamethrower attachment for drones.
Emerging Tech

6 questions we have about Elon Musk’s Neuralink brain interface technology

Elon Musk's Neuralink sounds like an exciting leap forward for human-computer relations, but brain implants raise the specter of Black Mirror-esque privacy invasions. We have a few questions about how this would work.
News

Canadian medical project demonstrates the health care potential of smart homes

A medical project involving smart homes demonstrates the technology's potential in treating mental illness and providing patients with a level of independence previously thought impossible.
Emerging Tech

Implant restores sight in blind patients by beaming images directly to the brain

Engineers have developed a neural implant which could help restore vision for completely blind people by bypassing non-functioning optical nerves and inputting images directly into their brains.
Deals

Ride in style with the Xiaomi Mi electric scooter for $97 less post-Prime Day

Scooters started out as a plaything for kids. But now they are larger and have become a common and efficient means to commute, like the Xiaomi Mi electric scooter. It's available on Amazon as a post-Prime Day deal for $97 less.
Emerging Tech

Trippy VR demo reads your brain waves to create sleep-inducing visuals

Struggle to get to sleep? Researchers in Australia have been experimenting with ways to combine a person’s brain activity with virtual reality to create a kind of VR lullaby machine.