Microsoft is using artificial intelligence to tackle a huge problem — helping visually impaired users interact with the world around them. To that end, it has developed Seeing AI, an iPhone app that recognizes people, places, documents, currency, and more to narrate your surroundings.
Open the app and point your phone’s camera at a friend, and Seeing AI will tell you who you’re looking at and the expression on their face. Point it at a can of soup in the grocery store, and it will read off the brand as well as the directions. The app can also translate printed text to speech, whether it’s on a paper or a label on a door. It can even tell dollar bills apart from one and another.
Seeing AI isn’t only intended for your phone’s camera, either. You can import images to the app through iOS’ share menu, and it will describe the picture and read text if any is contained.
The app sports a few experimental features as well, though they require you to have a data connection to function. There’s a “scene” mode, which will literally describe your environment, rather than simply reading text. Another mode can decipher and read out handwriting.
One of the more impressive aspects of Seeing AI is that everything the app can do — outside of those last two features — is handled locally on the device. Most of the time, you’ll never need the cloud’s help to use Seeing AI, which is a massive benefit. Everyone expects to be able to connect to the internet wherever they go, but the reality is much spottier — so it’s reassuring to know Microsoft has planned for the inevitable.
Microsoft’s latest project comes at an interesting time for the company, just a day after it ended support for its Windows Phone 8.1 operating system. While Microsoft may have struggled in building its own mobile devices, it’s currently producing some of its most interesting, unique, and helpful apps for other platforms.
Seeing AI is debuting on iOS, and can be downloaded for free. Microsoft has not disclosed when or if Android users can expect the app.
