MIT's Pic2Recipe A.I. Can Predict Food Ingredients By Analyzing a Photo

Scrolling through food photography can bring on the desire to recreate a dish at home, but what if the ingredients aren’t listed? Could there be a way to find out just by analyzing the image? That’s what researchers at the Massachusetts Institute of Technology asked when they set out to create a deep learning algorithm that could predict a recipe based just on a photo. The research, published on July 20, resulted in a program called Pic2Recipe that could accurately predict a dish’s recipe based on a photo, with a 65 percent success rate.

Earlier attempts to turn photos into recipes were limited by smaller datasets — although “small” is relative to all the possible recipes available. One study used 65,000 recipes, but it only included traditional Chinese cuisine; another only had about a 50 percent accuracy in initial testing. Because deep learning algorithms “learn” from being fed large quantities of data, these resulting programs were missing large gaps in potential ingredients, affecting the program’s accuracy.

To create a larger database, the researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) knew the software would have to be based on a wide-ranging set of data. So to solve that narrow dataset, the team turned to large sets of photos and recipes that already exists — food websites. Compiling data from places like Food.com and All Recipes, the team created Recipe1M, a dataset of over one million recipes.

Using those recipes and the associated images, the team was able to train the software to use object recognition to pick up on what each dish’s ingredients might be. With a list of ingredients, the system then selected  the recipe that best matched the list. Pic2Recipe was able to recognize ingredients like flour, eggs, and butter.

The program doesn’t actually identify a recipe from the photo — it creates a list of ingredients. With that list, the program can then go through that one-million-recipe database and choose the one with ingredients that match the list from the photo.

“In computer vision, food is mostly neglected because we don’t have the large-scale datasets needed to make predictions,” said Yusuf Aytar, a postdoctoral associate who co-wrote the paper with MIT professor Antonio Torralba. “But seemingly useless photos on social media can actually provide valuable insight into healthy habits and dietary preferences.”

Since the computer already has that large dataset, it is also able to able to pick up on a number of different patterns, like that the average recipe has nine ingredients and the most popular are salt, butter, sugar, olive oil, water, eggs, garlic cloves, milk, flour, and onion.

The software could have a number of different real-world uses. A person could snap a photo at a restaurant to learn how to make the dish at home, or to track her personal nutrition.

The program, while it contains a wider dataset than earlier attempts, still has a few gaps. The researchers said the program has trouble with dishes that are a bit more ambiguous, like smoothies and sushi rolls. Similar recipes with a number of different variations, like lasagna for example, also tended to confuse the program.

The group plans to continue developing the program and even hopes to give the system the ability to tell how something is cooked, like picking up the difference between stewed and diced. Future work could also expand the program’s ability to recognize specific ingredients, like determining the type of onion instead of just listing onion.

You don’t have to wait until Pic2Recipe becomes a full fledged app to try it out. An online version allows users to upload images and try it out.

Product Review

The Brava Oven takes all the thinking out of cooking

Using bulbs to cook food at different light frequencies, the Brava Oven lets even the clueless cook a tasty meal. But your own smart chef doesn’t come cheap.
Smart Home

These accessories take your already awesome Instant Pot to a whole new level

If you're familiar with the Instant Pot, you know that this bad boy acts as a pressure cooker, rice cooker, cake maker, egg boiler, and a lot more. Here are some must-have accessories to make Instant Pot cooking even better.
Product Review

Meet Z6, the breakout star in Nikon's new mirrorless lineup

The Nikon Z6 is the sibling to the new mirrorless Z7 -- but for some photographers, the cheaper Z6 may be the better option. Read where the $2,000 camera beats the $3,400 one (and where it doesn’t) in our Nikon Z6 review.
Smart Home

Put away that sponge and let us help you pick the best dishwasher for your buck

Tired of doing dishes by hand? Take a look at our picks of the four best dishwashers currently available and let a machine do the dirty work for you. They’ll do a much better job, anyway.
Social Media

#ThrowbackThursday is only the start: Instagram hashtags for every day of the week

Not getting your hashtag fill with #ThrowbackThursday or #ManCrushMonday? Here's a list of some of the more popular Instagram hashtags, so you can outfit your next post with the proper tag, regardless of what day it is.
Photography

From DSLRs to mirrorless, these are the best cameras you can buy right now

From entry-level models to full-frame flagships, many cameras take great photos and video. The best digital cameras, however, push the industry forward with innovative sensors and improved usability, among other things. Here are our…
Photography

Photographers can now customize the layout of Lightroom Classic controls

Tired of scrolling past Lightroom tools that you don't use? Adobe Lightroom Classic now allows users to reorganize the Develop panel. The update comes along with new sharing options in Lightroom CC, and updates to the mobile Lightroom app.
Social Media

Instagram could be making a special type of account for influencers

Instagram influencers fall somewhere between a business profile and a typical Instagram, so the company is working on developing a type of account just for creators. The new account type would give creators more access to analytical data.
Cars

Best Products of 2018

Our reception desk has so many brown boxes stacked up, it looks like a loading dock. We’re on a first-name basis with the UPS guy. We get new dishwashers more frequently than most people get new shoes. What we’re trying to say is: We…
Photography

These are the best action cameras money can buy

Action cameras are great tools for capturing videos of your everyday activities, whether it's a birthday party or the steepest slope you've ever descended on your snowboard. These are the best money can buy.
Photography

Canon holiday sale features the Rebel T6 2-lens kit for just $449

If you have a budding photographer in your life in need of a real camera, the Canon EOS Rebel T6 could make the perfect gift. Canon is currently offering the camera in a two-lens bundle for just $449 through December 29.
Emerging Tech

Light, speed: Lighting kit for DJI Mavic 2 lets you fly and film in the dark

Lume Cube, maker of small battery-powered LED lights for mobile photography, has announced a new lighting kit built specifically for the DJI Mavic 2 -- the first of its kind. Already our favorite drone, this makes the Mavic 2 even better.
Mobile

Leave the laptop at home, the iPad Pro is the travel buddy to take on vacay

The iPad Pro is a powerful tablet that's perfect for creatives and professionals. How does it fare when traveling with it as a laptop replacement? We took it on a two week trek in Japan to find out.
Social Media

Instagram’s 2018 year in review shines a light on where our hearts are

What did Instagram users share the most in 2018? A lot of heart emojis, heart face filters, and heart GIFs. The platform recently shared the year's top trends, including hashtags like #fortnite and #metoo along with a few surprises.