Google Lens adds unprecedented intelligence to your smartphone camera

how to use google lens keynote 4
Google Lens computer vision technology lets you find context with what your camera sees.

Want to know the name of that flower or bird you encounter during your stroll through a park? Soon, Google Assistant will be able to tell you, using the camera and artificial intelligence.

Google jump-started its 2017 I/O conference around AI and machine learning, and one computer vision technology it highlighted is Google Lens, which lets the camera do more than just capture an image — it gives greater context around what it is that you’re seeing.

Coming to Google Assistant and Google Photos, the Google Lens technology can “understand what you’re looking at and help you take action,” Google CEO Sundar Pichai said during the keynote. For example, if you point the camera at a concert venue marquee, Google Assistant can tell you more about the performer, as well as play music, help you buy tickets to the show, and add it to your calendar, all within a single app.

When the camera’s pointed at an unfamiliar object, Google Assistant, through image recognition, can tell you what it is. Point it at a shop sign, and using location info, can give you meaningful information about the business. All this can be done through the “conversational” voice interaction the user has with Assistant.

“You can point your phone at it and we can automatically do the hard work for you,” Pichai said.

If you use Google’s Translate app, you have already seen how the technology works: Place a camera over some text and the app will translate it to a language you understand. In Google Assistant, Google Lens will take this further. In a demonstration, Google showed that Google Assistant not only will translate foreign text, but also display images of what the text is describing, to give more information.

google lens io 2017 keynote 1
In a demo, Scott Huffman, Google VP of engineering for Assistant, demoed how Google Lens within Google Assistant can translate the Japanese text of an image, but also give further context about what the word is.

Image recognition technology isn’t new, but Google Lens shows how advanced machine learning is becoming. Pichai said that as with its work on speech, Google is seeing great improvements in vision. The computer vision technology not only helps recognize what something is, but can even help repair or enhance an image. Took a blurry photo of the Eiffel Tower? Because the computer recognizes the object and knows what it’s suppose to look like, it can automatically enhance that image based on what is already knows.

“We can understand the attributes behind a photo,” Pichai said. “Our computer vision systems now are even better than humans at image recognition.”

google lens io 2017 keynote 2
No longer will you need to write down what’s in your vacation photos. Google VP Anil Sabharwal for Google Photos showed how Google Lens can recognize objects in a photo and bring up relevant information about it.

To make Lens effective at its job, Google is employing sophisticated computational architecture of Cloud Tensor Processing Unit (TPU) chipsets, to handle training and inference for its machine learning. Its second-generation TPU technology can handle 180 trillion floating point operations per second; 64 TPU boards in one super computer can handle 11.5 petaflops. With this computing power, new TPU can handle both training and inference simultaneously, which wasn’t possible in the past (the previous TPU could only handle inference work, but not the more complex training). Machine learning takes time, but this hardware will help accelerate the effort.

Google Lens will also power the next update of Google Photos. Image recognition is already used in Photos to recognize faces, places, and things to help with organization and search. With Google Lens, Google Photos can give you greater information about the things in your photos, like the name and description of a building; tapping on a phone number in a photo will place a call, pulling up more info on an artwork you saw in a museum, or even enter the Wi-Fi password automatically from a photo you took of the back of a Wi-Fi router.

google lens io 2017 keynote 3
Hate entering Wi-Fi network passwords? Snap a photo of the wireless settings, and Google Lens technology through Google Photos can automatically enter it for you.

Assistant and Photos will be the first apps to use Google Lens, but it will be rolled out into other apps. And with the announcement of support for Assistant in iOS, iPhone users will be able to utilize the Google Lens technology as well.

Emerging Tech

Google’s radar-sensing tech could make any object smart

Computer scientists have shown how Google’s Soli sensor can be used to make dumb objects smart. Here's why radar-powered computing could finally make the dream of smart homes a reality.
Computing

Always have way too many tabs open? Google Chrome might finally help

Google is one step closer to bringing tab groups to its Chrome browser. The feature is now available in Google's Chrome Canady build with an early implementation that can be enabled through its flag system.
Smart Home

Project Alias is a ‘smart parasite’ that stops smart speakers from listening

Two designers chose to do something about nosy smart speakers. The result is Project Alias, a "smart parasite" that whispers nonsense to Google Home and Alexa until it hears a specific wake word.
Wearables

Check out the four cool Swatch watches you can use for mobile payments

Swatch has announced its Swatchpay technology is now available in Switzerland, enabling mobile payments from your Swatch watch. It works in a similar way to Apple Pay and Google Pay. Here's everything about it.
Mobile

Moschino glams up the 48-megapixel Honor View 20 at Paris launch

After its success with the View 10 in 2018, Honor has announced its sequel, the Honor View 20 with an entirely new type of display which has a hole-punch for the camera rather than a notch.
Wearables

10 top features you should be using on your Apple Watch

The Apple Watch can do more than just tell you the time, but you may not be aware of all the different functions it has. Our list of the 10 most often used functions and features will help you understand what it can really do.
Mobile

Photograph your meal with the Foodvisor app for caloric, nutritional estimates

Many of us understand the importance of a healthy diet, but counting calories and diving into nutritional data is a lot of work. The Foodvisor app for Android or iOS gives you a calorific and nutritional breakdown of your food from a photo.
Mobile

Get your photos on billboards with Apple’s 2019 ‘Shot on iPhone’ program

Like previous years, Apple has once again launched its Shot on iPhone program, which encourages users to submit the best photos they've taken with their iPhone. Ultimately, the winners will get their photos featured on billboards.
Mobile

Embrace your inner Dr. Frankenstein with the Huawei Mate 20 Pro and this cool app

Ever wanted to make your favorite toy come alive? If so, and you have a Huawei Mate 20 Pro, then you need to grab the 3D Live Maker app and try out some augmented reality magic. Here's how to use it.
Mobile

Samsung Galaxy S10 Plus photo leaks with dual front-facing camera

It won't be long now; With 2019 underway, the Samsung Galaxy S10 is almost here. Before it arrives, here's absolutely everything you need to know about all three of Samsung's next flagships.
Computing

Breeze through security with these checkpoint-friendly laptop bags

Getting through airport security is a drag, but your laptop bag shouldn’t be. Thankfully, these checkpoint-friendly laptop bags will get you and your gear to your destination with ease.
Social Media

Google will begin shutting down the classic Hangouts app in October

Google confirmed that it will begin retiring the classic Google Hangouts app in October. The company will start by pushing users to move to the new Google Hangouts Chat and Hangouts Meet.
Smart Home

Starbucks coffee delivery lands in 6 more cities via Uber Eats

Starbucks is in the process of expanding its coffee delivery service far beyond Miami. Available via the Uber Eats app, the service has just launched in San Francisco, with five additional cities joining in the coming weeks.
Mobile

How to make a contact group on an iPhone

There's no obvious way to crate contact groups on your iPhone, but you can do it through iCloud or you can install a third-party app to help. We explain both methods as we explore how to make a contact group on an iPhone.