Skip to main content

Hot dog! Yelp photo classification makes food shots easier to browse

yelp photo classification sandwiches somacrop
A picture may be worth a thousand words — but it can also be worth lunch. Yelp visitors often rely on photographs to determine their next dining spot, and now, updated artificial intelligence is helping to both prioritize the best images and categorize each shot. So how can a computer determine a salad from a slice of pizza? Lindsay Goyne, Yelp consumer product manager Lindsay Goyne and software engineer Alex Miller recently sat down with Digital Trends to share just how new software is pushing more relevant photos to foodies through new Yelp photo classification.


Unlike the new app inspired from HBO’s Silicon Valley, which doesn’t actually have much real world use outside of a few laughs, Yelp’s new hot dog-identifying app works to create a better experience for Yelp users. The neural network identifies over 20 different types of food, categorizing each dish so users can find out what that new diner’s burger looks like without digging through a few hundred photos. Hungry for pizza? Tap the photos, the food option, then click the pizza tab and see if that restaurant you’ve always wanted to try puts big chunks of tomato on top or whether they have deep dish.

“Photos have been trending up through multiple photo apps and the web,” said Goyne, “it’s a new favorite way to consume information. For some of these lower-risk options, like finding an option for lunch where you don’t need to do a bunch of research and you aren’t spending much money, photos are a great way to do that research.”

The software, launching earlier this year, is a result of user feedback. Yelp said that, when it comes to restaurant reviews, some of the most common questions were whether they had a certain dish. By organizing photos uploaded from other users, Yelp is able to present that information in a more precise way.


The convolutional neural network was based on a model built by Google researchers, Miller said. Artificial intelligence platforms are frequently shared among developers so new programs don’t need to start from scratch — like Google’s Wi-Fi-free object recognition algorithm. Yelp gave the system expanded object recognition capabilities by feeding the system thousands of images for each food category. With that data, the computer was able to learn what makes a hot dog different from a sandwich.

Along with the food categories, images are automatically sorted by shots of the interior and exterior and photos of the menu.

Yelp also uses both user response and artificial intelligence to determine which photographs to show first. While the company is continuing to develop software for ordering the photos, the software mixes user feedback — including clicks on “helpful” and “like” — and mixes that with an AI program that puts blurry, dark photos toward the end of the list and images with a nice depth of field and more contrast to the top of the list. The program also tries to prevent similar photos from showing up right next to each other, like three burgers in a row, and determines what image to use as the main photo.


Yelp’s software engineers are also working to teach the computer to recognize what the most important object in the photograph is, so that when the images is cropped to fit on different areas of the web page, the subject is still included.

The software is helping users sift through review photos that, on the most popular restaurants, can number over 1,000. Users won’t see the feature universally, since locations with only a few photographs do not integrate the category organization.

Yelp plans to bring the image classification system into more areas and is currently researching how a similar feature could help users looking for home services and automotive.

Editors' Recommendations

Hillary K. Grigonis
Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Hey photographers, AI is helping editors determine which photos make the cut
dreamstime ai photo editor 27072305  rear view of editors working on computer in a bright office

Artificial intelligence is now helping photo editors determine which shots make the cut. Dreamstime, a stock photography company shared on March 15 a new platform that’s learning from human editors which images fit the company’s parameters.

Dreamstime already uses AI to tag photos with relevant metadata to improve the search results. The newest machine learning software isn’t going to replace actual photo editors, Dreamstime says, but will help make the process faster by labeling submissions and continue learning from the editor’s choices.

Read more
Redesign makes Google Photos easier to navigate on Android
google photos android 28 54843347 ml

Finding an image on Google Photos just got a tad easier. In an Android-only update Thursday, Feb. 2, Google reworked the albums feature, adding enhanced organization to the ways albums are displayed.

The albums tab is now divided into three parts. Along with sorting your actual albums, the app now separates images based on what app you shot them with. A third category auto-organizes photos by location and what’s inside them. While the auto albums isn’t a new feature, the new organization makes them easier to find.

Read more
Google Photos finds your ‘best memories’ using artificial intelligence
google photos adds recent memories googlephotos

Google is using artificial intelligence to scope out your favorite memories. As part of an update rolled out yesterday, Google Photos will now use facial recognition and machine learning to both find your best shots and bring up old, but similar photos. The update also allows users to create GIFs from videos, and automatically rotates upside-down photos in both the iOS and Android apps, as well as the web-based system.

Google Photo’s new memories feature uses facial recognition software to detect the people in your latest photos -- then finds a few old photos of the same people, organizing them onto a card for easy browsing. The feature is rather similar to those Facebook videos that pop up on a “Friendiversary,” only the memories are triggered by the photos you take rather than by a date. The feature brings easy ways to reminisce with old photos -- including the shots that you upload to Google Photos but don’t share publicly.

Read more