Skip to main content

Google is bringing AI superpowers to your smartphone camera, and it looks ridiculous

Google Astra on a phone.

It seems AI assistants are antique, or that’s what Google wants you to believe, for we are in the era of AI agents — and Google I/O 2024 has quickly proven that. Say hello to Project Astra, a generative AI agent with vision, text, and speech capabilities, with a sprinkling of memory and spatial awareness capabilities in tow.

Think of it as eyes for your phone that can make sense of the world around you. Point it at a mathematical equation, and it will solve it. Pointing the camera at a cat? Astra will suggest an apt name for the feline meow-ster. Ask it where you left your earbuds, and if the camera sensor has seen them, it will say something like, “You left them on the sofa.”

Astra can make sense of code appearing on a screen, identify objects and explain what they do, identify buildings, and more. Think of it as Google Lens but for the entire world, and can make sense of most anything in front of the camera’s lens.

Unlike Google Assistant, you don’t need to prompt it. Just point the camera at anything, utter the audio query, and Astra will explain in a natural language response. Google says Project Astra will be rolled out via the Gemini app later this year.

An all-seeing AI agent?

Project Astra demonstration on a phone.

If Project Astra sounds familiar, that’s because OpenAI demoed a similar feature for ChatGPT — powered by the new GPT-4o model — just a day ago. OpenAI’s tool is currently under the red-teaming phase for safety testing and will be released in a phased manner, starting with ChatGPT Plus subscribers.

Google won’t say whether Astra will have a price tag. But given the near spontaneity of responses, the kind of visual data crunching involved, and the generative chops required to offer a meaningful response, it’s unlikely Astra will be served as a free perk owing to the compute requirements.

The best example would be the Google One AI Premium subscription, which already hides some of the new Gemini-powered experiences behind a paywall. But so far, Astra looks like the most amazing AI innovation Google has showcased ever since it wowed the world with Duplex’s capabilities a few years ago.

Editors' Recommendations

Nadeem Sarwar
Nadeem is a tech journalist who started reading about cool smartphone tech out of curiosity and soon started writing…
The 4 biggest things Google didn’t announce at Google I/O 2024
A photo of Sundar at the Google I/O 2024 keynote.

Google’s big keynote at its I/O 2024 developer conference was mostly focused on Gemini, its AI tool. There are big changes coming to Google Search, Google Photos, Google Workspaces, Android 15, and more, as expected.

If you were expecting more from the Google I/O keynote that didn’t involve Gemini AI, then you may be disappointed. There were no hardware announcements, though there was a possible tease of something in the future.

Read more
Watch Google’s 10-minute recap of its AI-filled I/O keynote
The stage for Google I/O 2024.

Google unveiled a slew of generative-AI goodies at its annual I/O event on Tuesday during a packed keynote that lasted almost two hours.

If you couldn’t watch it at the time, or really don’t want to sit through all 110 minutes of it on Google’s YouTube channel, the web giant has kindly shared a video that compresses the best bits of the event into a mere 10 minutes. You can watch it below:

Read more
Android is getting an AI overhaul. Here’s what it looks like
Android 15 on stage at Google I/O 2024.

At Google I/O 2024, Google has shown off a lot of new improvements coming to Android, thanks to Gemini. With Gemini, Android will be aware of the context on the screen in a variety of scenarios, which will make your life a lot easier. At least, according to Google.

Circle to Search, which was first shown off during Samsung’s Galaxy Unpacked event earlier this year, will be getting some new enhancements. Now, Circle to Search will be a great new study buddy for students. Why? Because it will be able to help with homework, like physics and math. You can simply circle a prompt on the screen, and it will give you step-by-step instructions on how to solve it.

Read more