Shutterstock’s visual search engine could make browsing photos less of a chore


Perhaps somewhat ironically, we are used to searching for photos through text. Looking for a photo of a cat? Just type in “cat” in the Google image search bar, and it will return relevant photos (provided they were tagged as such, of course). Keywords will do the job at the most basic level, but what if you are looking for a specific type of photo? You could type “yellow cat” or some sort of generic description, but things become difficult as the description becomes more complex.

To address this, photo agency Shutterstock has just launched a new tool, called Reverse Image Search, that allows customers to upload a photo (up to 5MB) and find images that are similar. Using computer vision, Shutterstock says the tool breaks through the limiting ceiling of metadata.

Besides keywords, “the technology now relies instead on pixel data within images,” wrote Kevin Lester, who is Shutterstock’s vice president of engineering, in a blog post. “It has studied our 70 million images and 4 million video clips, broken them down into their principal features, and now recognizes what’s inside each and every image, including shapes, colors, and the smallest of details; this visual and conceptual data is represented numerically.”

Although this kind of computer vision-based search technology has existed for years (Google’s image search lets you do the same thing), when compared with similar tools offered by other stock photo services, Shutterstock says its technology is the most refined.

“It isn’t the first, but it’s the best on the market,” says Lawrence Lazare, Shutterstock’s product director for search and discovery.

The big benefits of using computer vision are accuracy and speed, and for Shutterstock’s customer base, it solves a major problem with search. It cuts down on the amount of time spent on searching for an image. If you are looking for inspiration or something generic, metadata (keywords) is easier, Lazare says. But if you are creating something specific — an ad campaign, for example — and you have specific requirements for what needs to be in that photo, then words aren’t as successful.

“Words are fallible — some pictures are hard to describe,” Lazare adds. “Some photos would require a short story to describe, and people don’t search like that.”

For example, typing in “sunset” into the search bar will result in 14,394 pages encompassing 1,439,383 photos, illustrations, and vector art that depict a sunset. And the photos are dependent on whether the photographer added keywords properly (sometimes a photographer will use a bunch of keywords to tag a batch of photos, say a wedding, but then may include photos that aren’t related).

The image on the left shows varied results when searching by keywords. The image on the right shows visually similar results when using visual search.
The image on the left shows varied results when searching by keywords. The image on the right shows visually similar results when using visual search.

You could narrow down the search results by adding additional keywords, like “city and architecture,” but, as it turns out, you’ll still have 140,330 options to browse through. It’s even more difficult when the photo in your head has nuances like the angle of a building or the color of a sunset.

Which is why visual similarity is more useful than keyword similarity, Lazare says, but this type of search requires a significant amount of machine learning, and it is not an easy task. When an image is uploaded, the computer breaks it down numerically — in a manner that it can understand — so that it can compare and contrast the important aspects of the image. The computer has to compare it against the millions of photos in Shutterstock’s archive, and do so incredibly quickly; it takes less than 20 milliseconds for the algorithms to compare and contrast 70 million images in real time. For the computer, some photos are easier to decipher, but when you have things like abstract art or colors, it’s a bit harder, and the computer is more likely to return “false positives.”

To achieve its success rates, the neural network utilized by Shutterstock’s computers required a lot of training. At the beginning, the first attempts weren’t good, but over time, the responses — reflecting the learning they were doing on their own — improved. Lester, who oversees search as well as the computer vision team, told us that in about a year’s time, the company managed to go from having nothing to having something that works well.

From our own experiments (the feature is live, and anyone can try it out by uploading an image), we can say the visual search tool is pretty good. Although it has trouble with complicated photos, it’s more successful with simpler ones. But Shutterstock, of course, isn’t the only company to develop a visual search engine: We noticed equally good results via Google’s image search, and many of Shutterstock’s competitors offer visual search as well (although Shutterstock showed us similar technology from competitors, and claims they aren’t as successful, hence one reason why they decided to build it from scratch).

We uploaded a fairly complicated photo, and threw the computer off. However, it does recognize that it's some type of architecture.
We uploaded a fairly complicated photo, and threw the computer off. However, it does recognize that it’s some type of architecture.

This all shows just how far along computer vision and machine learning have come in a relatively short time. And it’s only going to get better: Shutterstock is adding new tools to its network that would allow users to give its computers feedback about the quality of the search results, and will soon unveil visual search for its four million video footage assets, which is an even greater challenge than static photos.

Emerging Tech

Awesome Tech You Can’t Buy Yet: heat-powered watches, phone cases with reflexes

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!
Computing

Want to save a webpage as a PDF? Just follow these steps

Need to quickly save and share a webpage? The best way is to learn how to save a webpage as a PDF file, as they're fully featured and can handle images and text with ease. Here's how.
Photography

Starting your very own vlog? Here are the best cameras to buy

Any camera that shoots video can be used to vlog, but a few models stand out from the crowd thanks to superior image quality, ergonomics, and usability. When it comes to putting your life on YouTube, here are the best cameras for the job.
Photography

What to look for and what to avoid when buying a camera

Looking to buy a new camera? Our comprehensive camera guide for 2016 has answers to any camera or photography questions you might ask, whether in regards to pricing, image quality, or weatherproofing.
Home Theater

Optoma’s all-in-one laser projector gives you 120 inches of 4K for $3,000

All too often, a really big image size for your home theater has meant tons of money for a large TV, or putting up with the compromises of a decent projector. Optoma's new P1 4K laser projector puts an end to that dilemma.
Photography

From 11K to just OK: The biggest photo gear announcements at CES 2019

From 11K cameras to 1 TB media cards, CES 2019 brought a peek at new gear for photographers and videographers. But what photography gear grabbed our attention the most? Here are the biggest photo gear announcements from CES 2019.
Photography

Authentic, holistic, retro photography is in: Here are 2019’s predicted trends

What types of imagery are we most drawn to? According to recent stock photography data from Adobe, StoryBlocks, and Shutterstock, authentic, holistic, and humanitarian content will be in high demand in 2019.
Social Media

No yolk! A photo of an egg has become the most-liked post on Instagram

Until this weekend, the most-liked post on Instagram was of Kylie Jenner's baby daughter, which has around 18 million likes. It's now been knocked off the top spot not by a stunning sunset or even a cute cat, but by an egg.
Photography

Going somewhere? Capture more than your phone can with the best travel cams

Hitting the road or doing some globetrotting this year? Bring along the right camera to capture those once-in-a-lifetime vacation memories. Here's a list of some of our current favorites.
Photography

From 4K powerhouses to tiny action cams, here are the best video cameras

Although not as popular as they once were, dedicated video cameras still have their benefits. From travel vlogging to home movies to recording your kid's little league game, here are the best video cameras you can buy right now.
Photography

The best mirrorless cameras pack all the power of a DSLR, minus the bulk

Mirrorless cameras offer a lot of photography firepower, inside a compact body. Explore the best mirrorless cameras, from the pro-level to the beginner-friendly shooters, in this guide.
Photography

This A.I.-powered camera follows the action to produce epic selfie videos

Want to capture more epic action selfies? The Obsbot Tail is a camera-gimbal combo that uses artificial intelligence to follow the action. Using a handful of different modes, the camera works to keep the action in the frame.
Photography

Sony crams its best camera tech into the new $900 A6400

Love Sony's autofocus, but can't stomach the full-frame price? The Sony A6400 mirrorless camera uses some of the same autofocus technology and the processor of the A9 in a compact, more affordable crop-sensor camera.