MIT's Pic2Recipe A.I. Can Predict Food Ingredients By Analyzing a Photo

Scrolling through food photography can bring on the desire to recreate a dish at home, but what if the ingredients aren’t listed? Could there be a way to find out just by analyzing the image? That’s what researchers at the Massachusetts Institute of Technology asked when they set out to create a deep learning algorithm that could predict a recipe based just on a photo. The research, published on July 20, resulted in a program called Pic2Recipe that could accurately predict a dish’s recipe based on a photo, with a 65 percent success rate.

Earlier attempts to turn photos into recipes were limited by smaller datasets — although “small” is relative to all the possible recipes available. One study used 65,000 recipes, but it only included traditional Chinese cuisine; another only had about a 50 percent accuracy in initial testing. Because deep learning algorithms “learn” from being fed large quantities of data, these resulting programs were missing large gaps in potential ingredients, affecting the program’s accuracy.

To create a larger database, the researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) knew the software would have to be based on a wide-ranging set of data. So to solve that narrow dataset, the team turned to large sets of photos and recipes that already exists — food websites. Compiling data from places like Food.com and All Recipes, the team created Recipe1M, a dataset of over one million recipes.

Using those recipes and the associated images, the team was able to train the software to use object recognition to pick up on what each dish’s ingredients might be. With a list of ingredients, the system then selected  the recipe that best matched the list. Pic2Recipe was able to recognize ingredients like flour, eggs, and butter.

The program doesn’t actually identify a recipe from the photo — it creates a list of ingredients. With that list, the program can then go through that one-million-recipe database and choose the one with ingredients that match the list from the photo.

“In computer vision, food is mostly neglected because we don’t have the large-scale datasets needed to make predictions,” said Yusuf Aytar, a postdoctoral associate who co-wrote the paper with MIT professor Antonio Torralba. “But seemingly useless photos on social media can actually provide valuable insight into healthy habits and dietary preferences.”

Since the computer already has that large dataset, it is also able to able to pick up on a number of different patterns, like that the average recipe has nine ingredients and the most popular are salt, butter, sugar, olive oil, water, eggs, garlic cloves, milk, flour, and onion.

The software could have a number of different real-world uses. A person could snap a photo at a restaurant to learn how to make the dish at home, or to track her personal nutrition.

The program, while it contains a wider dataset than earlier attempts, still has a few gaps. The researchers said the program has trouble with dishes that are a bit more ambiguous, like smoothies and sushi rolls. Similar recipes with a number of different variations, like lasagna for example, also tended to confuse the program.

The group plans to continue developing the program and even hopes to give the system the ability to tell how something is cooked, like picking up the difference between stewed and diced. Future work could also expand the program’s ability to recognize specific ingredients, like determining the type of onion instead of just listing onion.

You don’t have to wait until Pic2Recipe becomes a full fledged app to try it out. An online version allows users to upload images and try it out.

Emerging Tech

MIT whiz kids got a robot to take on the viral #BottleCapChallenge

It turns out the ultra-smart folks at MIT think about the same silly viral meme challenges as the rest of us. Check out their attempt to get a robot to take on the #BottleCapChallenge.
Mobile

It’s World Emoji Day! Here are the emojis coming to Android and iPhone this year

It's World Emoji Day! Google and Apple are celebrating by releasing sneak previews of their new emojis. With new inclusions for those with disabilities and new gender-neutral options, here are some of the new emojis heading our way.
Deals

Save up to 49% on professional-grade Vitamix blenders during Amazon Prime Day

Simplify life in the kitchen with a full-functioning blender that can whip up quite a list of concoctions that are not limited to dessert. Two Vitamix professional grade blenders are on sale just for Prime Day 2019.
Movies & TV

Here are the best shows on Netflix right now (July 2019)

Looking for a new show to binge? Lucky for you, we've curated a list of the best shows on Netflix, whether you're a fan of outlandish anime, dramatic period pieces, or shows that leave you questioning what lies beyond.
Photography

The best microfiber towels for cleaning your lens for 2019

The best microfiber towel should keep your lenses clean and be easy to transport, and that's why we've picked the Nikon 8072. But there are plenty of other quality towels out there. Here are the best.
Deals

The Canon EOS Rebel T6 two-lens kit gets $300 off ahead of Prime Day

If you’re looking for an entry-level camera, the Canon EOS Rebel T6 is a good choice. Amazon is offering the camera in a two-lens bundle at the price of $428. The kit normally retails for $726. You get to save $300.
Photography

Oppo Reno 10x Zoom vs. Huawei P30 Pro camera shootout: Zooming in on the action

When Huawei introduced its zoom lens on the P30 Pro, it opened up a whole new world of photographic opportunity on a smartphone. Now, Oppo's challenging it with the Reno 10x Zoom, so we've found out which is best.
Photography

The Canon Rebel SL2 slides in at under $400 for the best Prime Day camera deal

The Canon EOS Rebel SL2 is an easy entry into the world of DSLR photography -- and that entry just got even easier with a 45% discount on Tuesday, thanks to Amazon Prime Day 2019. The deal puts the camera with a kit lens at less than $400.
Photography

With 61 megapixels, the Sony A7R IV is the highest-resolution full-frame camera

With a 61-megapixel, backside-illuminated sensor, the Sony A7R IV is officially the highest-resolution full-frame camera you can buy. It also boasts a stronger body, better viewfinder, and new autofocus system.
Photography

These are the best camera straps you can get your hands on

Choosing the right camera strap for your needs can be a tough decision. To help sort through the junk, we've rounded up the best camera straps on the market, from stylish neck straps to functional shoulder slings.
Photography

With object tracking, the lightweight DJI Ronin-SC is still heavy on features

Designed for mirrorless cameras, the DJI Ronin-SC packs several features from the Ronin-S -- and then some -- into a lighter, one-handed gimbal. Despite the smaller size, the DJI Ronin-S adds new object tracking and expanded remote control.
Emerging Tech

Buying on a budget? Here's all the best tech you can snag for $25 or less

We live in a world where you can get a cheeseburger for $1, a functioning computer for $5, and thousands of HD movies for $10 -- so it stands to reason that you should be able to pick up some pretty sweet gear for $25.
Photography

Digitize old photos and film with the best photo scanners for 2019

Got a box of old photos to scan, or a batch of fresh film to digitize? The best photo scanners offer high-resolution, colorful scans along with features to make the scanning process less tedious.
Photography

The magic hour creates magical photos. Here’s how to capture dreamy stunners

The golden hour, aka the magic hour, is a special time for photographers that happens twice a day. Here are some simple tips for making the most of this time to capture stunning portraits, landscapes, and the like.