Skip to main content

Searching for objects and locations inside video footage is getting much easier

google cloud video intelligence api
Image used with permission by copyright holder
During the Google Cloud Next Conference in San Francisco, Google revealed a new machine learning application program interface (API) called Cloud Video Intelligence. With this API, developers can create applications capable of detecting objects within video and making them searchable and discoverable. Both nouns and verbs can be applied to those objects, such as “dog” and “run.”

An API is essentially a bridge between a service and an application. In this case, the API connects to the Google Cloud Machine Learning platform for the compute aspect and stores annotated videos on Google Cloud Storage. Thus, due to this “bridge,” an application based on Google’s new API will have access to this specific functionality to provide end-users with a better way of searching through videos.

“You can now search every moment of every video file in your catalog and find every occurrence as well as its significance,” Google states. “It helps you identify key nouns entities of your video, and when they occur within the video. Separate signal from noise, by retrieving relevant information at the video, shot or per frame.”

In a demo, users can search for animals in an MP4 video file lasting just over a minute and a half. The labels generated by Cloud Video Intelligence consist of Animal (99 percent), Wildlife (94 percent), Zoo (91 percent), Terrestrial Animal (54 percent), Nature (51 percent), Tourism (47 percent), and Tourist Destination (43 percent). The sample video focuses on the Los Angeles Zoo presented by Disney’s Zootopia CGI-animated movie.

However, what’s really neat about the new API is how it can detect a scene in a video. In the same clip, Cloud Video Intelligence can detect 48 scene changes and in real time detect and label objects as the scenes change. For instance, in one scene that displays just Nick the fox, the API will generate seven labels. In another scene focusing on the zoo’s sign, the system only generates two labels … again, all in real time.

What Google has done is create a tool that enables users to search through a video catalog just like they would with text documents. According to the company, this will be highly useful for businesses to separate signals that are buried under noise. It can also “detect features of a signal providing only relevant entities at video, shot or frame level.”

“Google has a long history working with the largest media companies in the world, and we help them find value from unstructured data like video,” said Fei-Fei Li, Chief Scientist of Google Cloud AI and Machine Learning. “This API is for large media organizations and consumer technology companies, who want to build their media catalogs or find easy ways to manage crowd-sourced content.”

The new API is now in a private beta and will also be offered to Google’s partners such as Cantemo, which will use the API to connect its video management software to the Google Cloud Machine Learning platform.

Editors' Recommendations

Kevin Parrish
Former Digital Trends Contributor
Kevin started taking PCs apart in the 90s when Quake was on the way and his PC lacked the required components. Since then…
This $7 app doubled the performance of my gaming PC — seriously
Cyberpunk 2077 running on a gaming monitor.

I know it sounds too good to be true, but there's a $7 app that will double your performance in any game, with any graphics card. There are some caveats, as you probably suspect, but the app works shockingly well considering what it does -- and I haven't stopped using it since downloading it on Steam.

The app in question is called Lossless Scaling. It does two things. When the app released in 2018, it allowed you to upscale games (or any other content) using a more sophisticated algorithm than what your monitor uses. More recently, Lossless Scaling has added frame generation, and just a couple of weeks ago, that frame generation was overhauled.

Read more
Is LastPass safe? Here’s what we know about its security history
LastPass website on a laptop.

LastPass has been in the news quite a bit over the past decade. Following some data breaches and security incidents, you may be wondering if it’s now safe to use the well-known password manager -- whether you’re a previous, current, or potential LastPass user.

Let’s take a look at LastPass’ current features and security measures along with the previous incidents.
What is LastPass?

Read more
Gaming monitors just smashed through an important milestone
Forza Horizon 5 running on the HP Omen 27k.

The key difference between a normal monitor and a gaming monitor is the refresh rate, and TCL just revealed the fastest gaming monitor we've ever seen. At Display Week, the company demoed a gaming monitor that's capable of a 4K resolution at 1,000Hz, which vastly exceeds the options now available on the market.

Currently, Asus holds the crown for the fastest monitor on the market, with a 1080p resolution and a 540Hz refresh rate. Alienware isn't far behind with its 500Hz gaming monitor, which is also locked to 1080p. The fact that TCL has a display capable of 1,000Hz is impressive enough, but the 4K resolution really stands out.

Read more