MIT's Pic2Recipe A.I. Can Predict Food Ingredients By Analyzing a Photo

Scrolling through food photography can bring on the desire to recreate a dish at home, but what if the ingredients aren’t listed? Could there be a way to find out just by analyzing the image? That’s what researchers at the Massachusetts Institute of Technology asked when they set out to create a deep learning algorithm that could predict a recipe based just on a photo. The research, published on July 20, resulted in a program called Pic2Recipe that could accurately predict a dish’s recipe based on a photo, with a 65 percent success rate.

Earlier attempts to turn photos into recipes were limited by smaller datasets — although “small” is relative to all the possible recipes available. One study used 65,000 recipes, but it only included traditional Chinese cuisine; another only had about a 50 percent accuracy in initial testing. Because deep learning algorithms “learn” from being fed large quantities of data, these resulting programs were missing large gaps in potential ingredients, affecting the program’s accuracy.

To create a larger database, the researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) knew the software would have to be based on a wide-ranging set of data. So to solve that narrow dataset, the team turned to large sets of photos and recipes that already exists — food websites. Compiling data from places like Food.com and All Recipes, the team created Recipe1M, a dataset of over one million recipes.

Using those recipes and the associated images, the team was able to train the software to use object recognition to pick up on what each dish’s ingredients might be. With a list of ingredients, the system then selected  the recipe that best matched the list. Pic2Recipe was able to recognize ingredients like flour, eggs, and butter.

The program doesn’t actually identify a recipe from the photo — it creates a list of ingredients. With that list, the program can then go through that one-million-recipe database and choose the one with ingredients that match the list from the photo.

“In computer vision, food is mostly neglected because we don’t have the large-scale datasets needed to make predictions,” said Yusuf Aytar, a postdoctoral associate who co-wrote the paper with MIT professor Antonio Torralba. “But seemingly useless photos on social media can actually provide valuable insight into healthy habits and dietary preferences.”

Since the computer already has that large dataset, it is also able to able to pick up on a number of different patterns, like that the average recipe has nine ingredients and the most popular are salt, butter, sugar, olive oil, water, eggs, garlic cloves, milk, flour, and onion.

The software could have a number of different real-world uses. A person could snap a photo at a restaurant to learn how to make the dish at home, or to track her personal nutrition.

The program, while it contains a wider dataset than earlier attempts, still has a few gaps. The researchers said the program has trouble with dishes that are a bit more ambiguous, like smoothies and sushi rolls. Similar recipes with a number of different variations, like lasagna for example, also tended to confuse the program.

The group plans to continue developing the program and even hopes to give the system the ability to tell how something is cooked, like picking up the difference between stewed and diced. Future work could also expand the program’s ability to recognize specific ingredients, like determining the type of onion instead of just listing onion.

You don’t have to wait until Pic2Recipe becomes a full fledged app to try it out. An online version allows users to upload images and try it out.

Emerging Tech

How emotion-tracking A.I. will change computing as we know it

Affectiva is just one of the startups working to create emotion-tracking A.I. that can work out how you're feeling. Here's why this could change the face of computing as we know it.
Smart Home

Get baked while you bake with these clever cannabis cooking gadgets

Ready to step up your edibles game? Here are the best cannabis cooking gadgets for your kitchen, from an "Instant Pot" for your cannabutter to the best grinders and decarbing devices on the market. Check out the full list for more.
Smart Home

Whip it, blend it, and mix it up with the best hand mixers of 2019

A stand mixer takes up a lot of room on the counter, and it can also be quite expensive. Need an option that's not only affordable but also takes up very little real estate in your kitchen cabinets? Check out these hand mixers.
Emerging Tech

How MIT hacked horticulture to cultivate a hyper-flavorful basil plant

At MIT, Caleb Harper used his personal food computers to alter the climate in which he grew basil. Exposing it light for 24 hours a day changed the flavor profile of the plant, making it spicier and stronger.
Deals

The best budget-friendly GoPro alternatives that won’t leave you broke

Cold weather is here, and a good action camera is the perfect way to record all your adventures. You don't need to shell out the big bucks for a GoPro: Check out these great GoPro alternatives, including some 4K cameras, that won’t leave…
Photography

Etch-A-Snap camera puts a modern spin on one of your favorite childhood toys

Can't draw on an Etch A Sketch? Snap a photo with the Etch-A-Snap and the camera will draw out the scene for you. The weirdly cool camera designed by Martin Fitzpatrick replaces the usual LCD screen with an old-school Etch A Sketch.
Photography

The Black Eye Pro Cinema Wide G4 is a knockout lens for any smartphone

Where cheaper wide-angle accessory lenses add distortion, and costlier models don't always justify their higher prices, the Black Eye Pro Cinema Wide G4 offers a valuable balance of modest price and high quality optics.
Photography

Family feud: Huawei P30 Pro vs. P20 Pro vs. Mate 20 Pro camera shootout

The Huawei P30 Pro's camera has an amazing zoom mode and low light capabilities. But take these away, and how does it compare when facing its sibling phones, the P20 Pro and Mate 20 Pro, taking regular photos?
Photography

Nikon Z 7 vs. Sony A7R III: High-res mirrorless cameras compared

The Nikon Z 7 and Sony A7R III both have over 40 megapixels, but which one comes out on top? With similar image quality, the answer comes down to speed, autofocus, battery life, and design.
Deals

The Canon EOS Rebel T6 DSLR camera gets a steep price cut at Walmart

Modern smartphones can snap pretty impressive pics, but if you want pro-quality photos, you need a dedicated digital camera. The Canon EOS Rebel T6 is one of the best entry-level DSLR cameras on the market, and it’s on sale right now for…
Photography

Panasonic Lumix S1R vs. Nikon Z 7: When megapixels matter, which do you choose?

The 47-megapixels Lumix S1R and 46-megapixel Nikon Z 7 are the two highest-resolution, full-frame mirrorless cameras on the market. The S1R features a high-resolution mode that can take 187MP images, but the Nikon is lighter and cheaper.
Photography

Sweet 16: Wacom’s Cintiq 16 pen display makes retouching photos a breeze

Wacom’s Cintiq pen displays are usually reserved for the pros (or wealthy enthusiasts), but the new Cintiq 16 brings screen and stylus editing to an approachable price. Does it cut too much to get there?
Mobile

China bans selfies at gigantic Aperture Spherical Telescope

You can't take a selfie with the world's largest single-dish radio telescope anymore, as the Chinese government has banned everything from smartphones to digital cameras in the surrounding 5-kilometer area.
Photography

Light on price but rich on features, these are the best cameras for students

Need pro-level features on a budget? The best cameras for students mix advanced features with a more palatable price point. From $2K entry level full frame cameras to $600 budget picks, here are five of the best cameras for students.