Google Lens adds unprecedented intelligence to your smartphone camera

how to use google lens keynote 4
Google Lens computer vision technology lets you find context with what your camera sees.

Want to know the name of that flower or bird you encounter during your stroll through a park? Soon, Google Assistant will be able to tell you, using the camera and artificial intelligence.

Google jump-started its 2017 I/O conference around AI and machine learning, and one computer vision technology it highlighted is Google Lens, which lets the camera do more than just capture an image — it gives greater context around what it is that you’re seeing.

Coming to Google Assistant and Google Photos, the Google Lens technology can “understand what you’re looking at and help you take action,” Google CEO Sundar Pichai said during the keynote. For example, if you point the camera at a concert venue marquee, Google Assistant can tell you more about the performer, as well as play music, help you buy tickets to the show, and add it to your calendar, all within a single app.

When the camera’s pointed at an unfamiliar object, Google Assistant, through image recognition, can tell you what it is. Point it at a shop sign, and using location info, can give you meaningful information about the business. All this can be done through the “conversational” voice interaction the user has with Assistant.

“You can point your phone at it and we can automatically do the hard work for you,” Pichai said.

If you use Google’s Translate app, you have already seen how the technology works: Place a camera over some text and the app will translate it to a language you understand. In Google Assistant, Google Lens will take this further. In a demonstration, Google showed that Google Assistant not only will translate foreign text, but also display images of what the text is describing, to give more information.

google lens io 2017 keynote 1
In a demo, Scott Huffman, Google VP of engineering for Assistant, demoed how Google Lens within Google Assistant can translate the Japanese text of an image, but also give further context about what the word is.

Image recognition technology isn’t new, but Google Lens shows how advanced machine learning is becoming. Pichai said that as with its work on speech, Google is seeing great improvements in vision. The computer vision technology not only helps recognize what something is, but can even help repair or enhance an image. Took a blurry photo of the Eiffel Tower? Because the computer recognizes the object and knows what it’s suppose to look like, it can automatically enhance that image based on what is already knows.

“We can understand the attributes behind a photo,” Pichai said. “Our computer vision systems now are even better than humans at image recognition.”

google lens io 2017 keynote 2
No longer will you need to write down what’s in your vacation photos. Google VP Anil Sabharwal for Google Photos showed how Google Lens can recognize objects in a photo and bring up relevant information about it.

To make Lens effective at its job, Google is employing sophisticated computational architecture of Cloud Tensor Processing Unit (TPU) chipsets, to handle training and inference for its machine learning. Its second-generation TPU technology can handle 180 trillion floating point operations per second; 64 TPU boards in one super computer can handle 11.5 petaflops. With this computing power, new TPU can handle both training and inference simultaneously, which wasn’t possible in the past (the previous TPU could only handle inference work, but not the more complex training). Machine learning takes time, but this hardware will help accelerate the effort.

Google Lens will also power the next update of Google Photos. Image recognition is already used in Photos to recognize faces, places, and things to help with organization and search. With Google Lens, Google Photos can give you greater information about the things in your photos, like the name and description of a building; tapping on a phone number in a photo will place a call, pulling up more info on an artwork you saw in a museum, or even enter the Wi-Fi password automatically from a photo you took of the back of a Wi-Fi router.

google lens io 2017 keynote 3
Hate entering Wi-Fi network passwords? Snap a photo of the wireless settings, and Google Lens technology through Google Photos can automatically enter it for you.

Assistant and Photos will be the first apps to use Google Lens, but it will be rolled out into other apps. And with the announcement of support for Assistant in iOS, iPhone users will be able to utilize the Google Lens technology as well.

Cars

Volvo plans to face the future without sacrificing its identity

Volvo is embarking on an ambitious push to fill its lineup with electric and hybrid cars, as well as an infotainment partnership with Google. Volvo Americas CEO Anders Gustafsson explains how the Swedish automaker plans to pull that off.
Product Review

Mediocre battery and a big notch slight Google's otherwise perfect Pixel phone

Google’s Pixel 3 XL has two big flaws: The gigantic notch on the front, and mediocre battery life. That being said, this is the best Android experience you can find in a smartphone today.
Mobile

We toured the Google Hardware Store pop-up in New York: Here’s what it’s like

To show off its slew of new products, Google is opening up pop-up shops in New York and Chicago starting October 18. You'll be able to buy products -- such as the Pixel 3 or the Google Home Hub -- at the Google Hardware Store.
Mobile

Google Pixel 3 vs. Apple iPhone XS: Does Google’s A.I. take down Apple?

The Google Pixel 3 is here, boasting top-tier specs like a Qualcomm Snapdragon 845 and 4GB of RAM, and some of the world's best artificial intelligence features on a phone. But can it take out the Apple iPhone XS?
Product Review

Samsung’s Galaxy Book 2 is a Surface Pro alternative with one big advantage

The 2-in-1 form factor is clearly a big deciding factor for anyone looking to buy a new device, which is why Samsung is again getting in the action this year with the new Galaxy Book 2.
Mobile

Google Maps brings its real-time journey-tracking feature to iPhone

It's been available on Android for a while now, and now Google Maps has brought its real-time journey-tracking feature to iOS. It lets you choose who to share a journey with, and tracking ends automatically when you arrive.
Mobile

The five-camera LG V40 ThinQ is now available from the major carriers

LG has finally taken the wraps off the new LG V40 ThinQ, the company's latest and greatest flagship phone that packs a whopping five cameras. Here's how to buy the new LG V40 ThinQ.
Wearables

A strap for everyone: The best Apple Watch bands you can buy right now

If you have an Apple Watch, you know how easy it is to take off the strap it came with, so why not buy yourself another one? Here we've gathered the best Apple Watch bands we've seen so far and there's something for everyone.
Mobile

Need a quick battery boost? Try one of our favorite portable chargers

Battery life still tops the polls when it comes to smartphone concerns. If it’s bugging you, then maybe it’s time to snag yourself a portable charger. Here are our picks of the best portable chargers.
Mobile

Which new iPhone is the best? iPhone XS vs. iPhone XS Max vs. iPhone XR

Apple has three new iPhone models to choose from this year, making the choice a little harder than usual. What's the difference between the iPhone XS, the iPhone XS Max, and the iPhone XR, and which is best?
Mobile

Google may charge up to $40 per Android device for app suite following EU ruling

Google announced that it will be charging Android device manufacturers in Europe a licensing fee to use its apps and services. The announcement is part of an effort to comply with new European Commission regulations.
Mobile

Need a do-over? Here's how to factory reset an iPhone, from XS on down

Resetting an iPhone can alleviate all sorts of software woes, and wipe away personal data should you sell your device or give it to someone else. Here's how to factory reset an iPhone from within iOS or iTunes.
Product Review

Huawei’s monster Mate 20 X makes the Galaxy Note 9 look small

The Huawei Mate 20 X has a 7.2-inch screen, but is surprisingly manageable to hold, yet still a little too big to carry around. Huawei’s pushing the phone’s ability as a mobile gaming handheld, challenging the Nintendo Switch.
Mobile

How to sell your old Google Pixel or Pixel 2 for the most money

So, it's time for a expensive new smartphone, and you'd like to partially fund the purchase by selling your old Google Pixel. Find all the information you need to get as much money as possible for your Pixel or Pixel 2 here in our guide.