MIT's Pic2Recipe A.I. Can Predict Food Ingredients By Analyzing a Photo

Scrolling through food photography can bring on the desire to recreate a dish at home, but what if the ingredients aren’t listed? Could there be a way to find out just by analyzing the image? That’s what researchers at the Massachusetts Institute of Technology asked when they set out to create a deep learning algorithm that could predict a recipe based just on a photo. The research, published on July 20, resulted in a program called Pic2Recipe that could accurately predict a dish’s recipe based on a photo, with a 65 percent success rate.

Earlier attempts to turn photos into recipes were limited by smaller datasets — although “small” is relative to all the possible recipes available. One study used 65,000 recipes, but it only included traditional Chinese cuisine; another only had about a 50 percent accuracy in initial testing. Because deep learning algorithms “learn” from being fed large quantities of data, these resulting programs were missing large gaps in potential ingredients, affecting the program’s accuracy.

To create a larger database, the researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) knew the software would have to be based on a wide-ranging set of data. So to solve that narrow dataset, the team turned to large sets of photos and recipes that already exists — food websites. Compiling data from places like Food.com and All Recipes, the team created Recipe1M, a dataset of over one million recipes.

Using those recipes and the associated images, the team was able to train the software to use object recognition to pick up on what each dish’s ingredients might be. With a list of ingredients, the system then selected  the recipe that best matched the list. Pic2Recipe was able to recognize ingredients like flour, eggs, and butter.

The program doesn’t actually identify a recipe from the photo — it creates a list of ingredients. With that list, the program can then go through that one-million-recipe database and choose the one with ingredients that match the list from the photo.

“In computer vision, food is mostly neglected because we don’t have the large-scale datasets needed to make predictions,” said Yusuf Aytar, a postdoctoral associate who co-wrote the paper with MIT professor Antonio Torralba. “But seemingly useless photos on social media can actually provide valuable insight into healthy habits and dietary preferences.”

Since the computer already has that large dataset, it is also able to able to pick up on a number of different patterns, like that the average recipe has nine ingredients and the most popular are salt, butter, sugar, olive oil, water, eggs, garlic cloves, milk, flour, and onion.

The software could have a number of different real-world uses. A person could snap a photo at a restaurant to learn how to make the dish at home, or to track her personal nutrition.

The program, while it contains a wider dataset than earlier attempts, still has a few gaps. The researchers said the program has trouble with dishes that are a bit more ambiguous, like smoothies and sushi rolls. Similar recipes with a number of different variations, like lasagna for example, also tended to confuse the program.

The group plans to continue developing the program and even hopes to give the system the ability to tell how something is cooked, like picking up the difference between stewed and diced. Future work could also expand the program’s ability to recognize specific ingredients, like determining the type of onion instead of just listing onion.

You don’t have to wait until Pic2Recipe becomes a full fledged app to try it out. An online version allows users to upload images and try it out.

Smart Home

4 small appliances that have changed the way we cook food (for the better)

These are exciting times for home cooks seeking to advance or simplify their technique in the kitchen, thanks in part to four small appliances that have made a splash in the past few years. Here’s why you should consider buying them.
Smart Home

OK Google, what else can you do? The best tips and tricks for Google Home

The Home functions in a similar fashion to its main competitor, the Amazon Echo, but has the added benefit of select Google services. Here are few tips to help you make the most of the newfangled device.
Deals

Instant Pot prices have dropped on these 5 models by up to $30

Whether you're an Instant Pot beginner or Instant Pot veteran these popular multi-cookers are worth the hype. There are so many Instant Pot models on sale right now, it might be hard to choose. We've rounded up all the deals here so you can…
Emerging Tech

Statistician raises red flag about reliability of machine learning techniques

Machine learning is everywhere in science and technology. But how reliable are these techniques really? A statistician argues that questions of accuracy and reproducibility of machine learning have not been fully addressed.
Photography

Fujifilm XP140 squeezes more durability, low-light ability into a waterproof cam

Fujifilm's waterproof compact can now head even further underwater. The Fujifilm XP140 features several upgrades, including a more durable body, a wider ISO range for low light, and expanded auto modes.
Photography

From f/1.2 primes to the mysterious DS, here are Canon’s upcoming RF lenses

Canon's EOS R mirrorless series will gain six new lenses this year. Canon just shared a list of six lenses under development, including four zooms and two prime lenses. One has a mysterious new feature called Defocus Smoothing.
Mobile

OnePlus 6T vs. Honor View 20: We compare the cameras in these ‘flagship killers’

For less than $600, you can buy either the OnePlus 6T or the Honor View 20, two extremely capable smartphones with plenty of exciting features. But which one has the best camera? We found out on a recent trip to France.
Photography

From DSLRs to mirrorless, these are the best cameras you can buy right now

From entry-level models to full-frame flagships, many cameras take great photos and video. The best digital cameras, however, push the industry forward with innovative sensors and improved usability, among other things. Here are our…
Photography

Photography news: Wacom’s slimmer pen, Leica’s cinema special edition

In this week's photography news, Wacom launches a new slimmer pen for pro users. Leica's upcoming M10-P is designed for cinema, inside and out, with built-in cinema modes in the updated software.
Emerging Tech

Awesome Tech You Can’t Buy Yet: Grow veggies indoors and shower more efficiently

Check out our roundup of the best new crowdfunding projects and product announcements that hit the web this week. You may not be able to buy this stuff yet, but it sure is fun to gawk!
Mobile

Be careful who you bokeh, jokes Apple’s latest iPhone ad

With iPhone sales under pressure, you'd think there wouldn't be much to laugh about at Apple HQ. But the company has seen fit to inject some humor into its latest handset ad, which highlights the camera's Depth Control feature.
Photography

What’s the difference between Lightroom CC and Lightroom Classic?

Lightroom CC has evolved into a capable photo editor, but is it enough to supplant Lightroom Classic? We took each program for a test drive to compare the two versions and see which is faster, more powerful, and better organized.
Photography

Luminar’s libraries gain speed, drop need for you to manually import images

Luminar 3 just got a performance boost. Skylum Luminar 3.0.2 has improved speed over December's update, which added the long-promised libraries feature giving editors a Lightroom alternative.
Photography

When you're ready to shoot seriously, these are the best DSLRs you can buy

For many photographers the DSLR is the go-to camera. With large selection of lenses, great low-light performance, and battery endurance, these DSLRs deliver terrific image quality for stills and videos.