MIT's Pic2Recipe A.I. Can Predict Food Ingredients By Analyzing a Photo

Pic2Recipe: Predicting recipes from photos
Scrolling through food photography can bring on the desire to recreate a dish at home, but what if the ingredients aren’t listed? Could there be a way to find out just by analyzing the image? That’s what researchers at the Massachusetts Institute of Technology asked when they set out to create a deep learning algorithm that could predict a recipe based just on a photo. The research, published on July 20, resulted in a program called Pic2Recipe that could accurately predict a dish’s recipe based on a photo, with a 65 percent success rate.

Earlier attempts to turn photos into recipes were limited by smaller datasets — although “small” is relative to all the possible recipes available. One study used 65,000 recipes, but it only included traditional Chinese cuisine; another only had about a 50 percent accuracy in initial testing. Because deep learning algorithms “learn” from being fed large quantities of data, these resulting programs were missing large gaps in potential ingredients, affecting the program’s accuracy.

Recommended Videos

To create a larger database, the researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) knew the software would have to be based on a wide-ranging set of data. So to solve that narrow dataset, the team turned to large sets of photos and recipes that already exists — food websites. Compiling data from places like Food.com and All Recipes, the team created Recipe1M, a dataset of over one million recipes.

Using those recipes and the associated images, the team was able to train the software to use object recognition to pick up on what each dish’s ingredients might be. With a list of ingredients, the system then selected  the recipe that best matched the list. Pic2Recipe was able to recognize ingredients like flour, eggs, and butter.

The program doesn’t actually identify a recipe from the photo — it creates a list of ingredients. With that list, the program can then go through that one-million-recipe database and choose the one with ingredients that match the list from the photo.

“In computer vision, food is mostly neglected because we don’t have the large-scale datasets needed to make predictions,” said Yusuf Aytar, a postdoctoral associate who co-wrote the paper with MIT professor Antonio Torralba. “But seemingly useless photos on social media can actually provide valuable insight into healthy habits and dietary preferences.”

Since the computer already has that large dataset, it is also able to able to pick up on a number of different patterns, like that the average recipe has nine ingredients and the most popular are salt, butter, sugar, olive oil, water, eggs, garlic cloves, milk, flour, and onion.

The software could have a number of different real-world uses. A person could snap a photo at a restaurant to learn how to make the dish at home, or to track her personal nutrition.

The program, while it contains a wider dataset than earlier attempts, still has a few gaps. The researchers said the program has trouble with dishes that are a bit more ambiguous, like smoothies and sushi rolls. Similar recipes with a number of different variations, like lasagna for example, also tended to confuse the program.

The group plans to continue developing the program and even hopes to give the system the ability to tell how something is cooked, like picking up the difference between stewed and diced. Future work could also expand the program’s ability to recognize specific ingredients, like determining the type of onion instead of just listing onion.

You don’t have to wait until Pic2Recipe becomes a full fledged app to try it out. An online version allows users to upload images and try it out.

Editors' Recommendations

Hillary never planned on becoming a photographer—and then she was handed a camera at her first writing job and she's been…
Create apocalyptic A.I. worlds with this camera app that removes people from pics

What would that travel photo look like without tourists? How about that selfie without any “self,” or any photograph without any people, ever? A new app aims to make a statement about what the world would look like -- literally  -- if artificial intelligence replaced all the people. Bye Bye Camera is an iOS app that uses A.I. to nix the people from a photo.

While the app could have some real-world uses for when too many tourists enjoying the view ruin the actual view, Bye Bye Camera is an art project that aims to evoke questions about the growth of A.I. Designed by mononymous artist Damjanski and the art collective Do Something Good, the app looks at what a world without real intelligence may look like. Damjanski designed the app with two additional collaborators.

Read more
‘Photoshopped’ royal photo causes a stir

[UPDATE: In a message posted on social media on Monday morning, Princess Kate said that she herself edited the image, and apologized for the fuss that the picture had caused. “Like many amateur photographers, I do occasionally experiment with editing," she wrote, adding, "I wanted to express my apologies for any confusion the family photograph we shared yesterday caused."]

Major press agencies have pulled a photo of the U.K.’s Princess of Wales and her children amid concerns that it has been digitally manipulated.

Read more
Best camera deals: Save on DSLR, action cameras, point-and-shoot

While utilizing one of the best camera phones can be nice for capturing everyday happenings on the fly, a more traditional camera offers superior optics, sensor size, and overall image quality. A camera by a brand such as Canon, Sony, or Nikon is a great way to more deeply explore the photography process, or to simply ensure you’re getting higher quality photos of the moments you don’t want to miss. Right now there are a a lot of savings out there if a traditional camera interests you, and there’s big savings to pounce on whether you’re looking for a DSLR camera, an action camera, or a point-and-shoot camera. We’ve rounded up all of the best camera deals out there right now, so read onward for more details.
Best DSLR camera deals

DSLR cameras are bulkier than mirrorless cameras, but they produce high-quality images that will let you capture the most beautiful moments. You'll be able to see directly through the lens with the mirrored optical viewfinder while composing your shot, and you've got your pick among years' worth of lenses, depending on what kind of pictures you want to take. DSLR camera deals are always in demand because they're expensive though, so hurry if you want to pocket the savings from the bargains we've gathered below.

Read more