Google is bringing AI superpowers to your smartphone camera, and it looks ridiculous

By Nadeem Sarwar Published May 14, 2024

Google Astra on a phone. — Google

It seems AI assistants are antique, or that’s what Google wants you to believe, for we are in the era of AI agents — and Google I/O 2024 has quickly proven that. Say hello to Project Astra, a generative AI agent with vision, text, and speech capabilities, with a sprinkling of memory and spatial awareness capabilities in tow.

Think of it as eyes for your phone that can make sense of the world around you. Point it at a mathematical equation, and it will solve it. Pointing the camera at a cat? Astra will suggest an apt name for the feline meow-ster. Ask it where you left your earbuds, and if the camera sensor has seen them, it will say something like, “You left them on the sofa.”

Recommended Videos

Astra can make sense of code appearing on a screen, identify objects and explain what they do, identify buildings, and more. Think of it as Google Lens but for the entire world, and can make sense of most anything in front of the camera’s lens.

Unlike Google Assistant, you don’t need to prompt it. Just point the camera at anything, utter the audio query, and Astra will explain in a natural language response. Google says Project Astra will be rolled out via the Gemini app later this year.

An all-seeing AI agent?

Project Astra demonstration on a phone. — Google

If Project Astra sounds familiar, that’s because OpenAI demoed a similar feature for ChatGPT — powered by the new GPT-4o model — just a day ago. OpenAI’s tool is currently under the red-teaming phase for safety testing and will be released in a phased manner, starting with ChatGPT Plus subscribers.

Google won’t say whether Astra will have a price tag. But given the near spontaneity of responses, the kind of visual data crunching involved, and the generative chops required to offer a meaningful response, it’s unlikely Astra will be served as a free perk owing to the compute requirements.

The best example would be the Google One AI Premium subscription, which already hides some of the new Gemini-powered experiences behind a paywall. But so far, Astra looks like the most amazing AI innovation Google has showcased ever since it wowed the world with Duplex’s capabilities a few years ago.

Contributor

Nadeem is a tech and science journalist who started reading about cool smartphone tech out of curiosity and soon started…

Topics

Mobile

You don’t need a Pixel to experience Google Gemini’s live camera mode

Gemini Live App on the Galaxy S25 Ultra broadcast to a TV showing the Gemini app with the camera feature open

Earlier this year, Google announced a long-awaited Gemini AI feature, previously known as Project Astra. Recently, it has been confirmed that Gemini Live’s camera and screen-sharing features will not be exclusive to Pixel devices.

As uncovered by 9to5Google, a Google support article states that camera and screen sharing in Gemini Live will be available on “any Android device with Gemini Advanced.” This means that the tools will be compatible with any device running Android 10 or later. Initially, this feature was believed to be only available on Pixel devices and possibly the Samsung Galaxy S25 series.

Mobile

Google says it’s using AI to respond to more of your ailment searches

Google search on Android app.

Over the last year, Google has been expanding its AI overviews in search results, offering responses compiled with generative AI rather than a list of links. The goal, as you would expect, is to make information directly accessible while helping you save time from clicking each link individually and deciding its merit for yourself. Now, Google seems confident these overviews can help you better with your medical queries too.

At its annual health-centric event, The Check Up, Google announced a several ways in which its products will improve how you find information about health-related topics. More specifically, the company said it is now expanding AI overviews in search results "to cover thousands more health topics," beyond the basic ones such as flu. These results are also being expanded to other languages besides English, including Spanish, Portuguese and Japanese, for queries made on mobile.

Mobile

Here’s your first look at the leaked Google Pixel 10 Pro XL

Google Pixel 9 Pro XL.

With Apple, Samsung, and OnePlus releasing their initial rounds of new phones for 2025, attention is shifting to Google, which is expected to announce the Pixel 10 series in the coming months. Today, we have the first leaked images of one of these upcoming devices.

Images of the Pixel 10 Pro XL have been revealed courtesy of Android Headlines. This phone features a 6.8-inch display and is expected to resemble its predecessor, the Pixel 9 Pro XL. The leaked dimensions indicate that the new model will measure 162.7 x 76.6 x 8.5 mm, reflecting only a 0.1 mm difference in height compared to the existing model.