New beta software automatically tags and captions historic images

gado images rebrands p120513lj 0116 1
Lawrence Jackson/White House
The photograph above is pretty straightforward — it doesn’t take too much expertise to see that it’s of President Barack Obama. And even if you’ve been living under a rock for the past decade or so, the White House and presidential seals are pretty big clues — for a human, that is. But for a computer? Not so much. But one newly re-branded company is working to change that with software that can identify historical figures from centuries before the word “artificial intelligence” was even thought of.

Project Gado first began as a way to develop new technology for quickly digitizing visual history through scanning hardware, from images to sheet music. Six years later, they’ve accomplished that goal and set out on another challenge — making sense of all that visual data, quickly and efficiently.

Now re-branded as Gado Images, the company is teaching computers how to interpret each image, automatically adding tags, labeling historical figures and even writing a caption. That photo of Obama? Not only did the beta software correctly identify the president, where he was, and what he was doing, but it also knew enough to indicate that he’s the 44th president and that the image was taken on December 5, 2013. And the system works to identify much older historical figures as well, along with their significance — which is pretty good considering most American’s don’t know that Franklin Roosevelt was a president. Currently, the system can recognize over 60,000 well-known personalities, both current and historic.

While a historian could properly identify many images, the Cognitive Metadata Platform makes it possible to label large volumes of visual history quickly and efficiently, simplifying a challenge that many museums and art galleries face.

The web-based Cognitive Metadata Platform, currently in beta testing, is expected to launch later this year.  The software works by combining a few existing concepts (like IBM’s Watson and Google Vision) with original programming. The system combines facial recognition and object recognition as well as optical character recognition to identify and read text to generate a few keywords.

Using neural networks, the system is able to make sense of all that those keywords and even add related terms. When the facial recognition identifies Martin Luther King Jr. in an image, it will also add related keywords like “civil rights,” for example.

“By combining all the different inputs together [the system] can become quite confident in what it’s seeing in the image,” Gado Images’ CEO, Thomas Smith, told Digital Trends. “It’s a learning system, learning relationships between entities and specific collections.” If the program knows that two people are often seen together, it will often tag both people or generate related keywords, he added.

The system can even estimate a time frame for when the image was taken, even on scans of photographs that come with no digital metadata. Using historical information like the person’s birth date and age-approximation software, the program can venture a pretty good estimate on when the photo was shot.

For historical purposes, the optical character recognition is also playing a big role. Many archival photographs will have a date or notes from the photographer on the back. With the OCR and a scan of both the front and back, the system can add the original notes, as well as adding more keywords relevant to those notes.

The program works on more than just photographs, and can help archive a number of different types of visual history, including drawings and even sheet music. Here’s how the system interpreted a few more visuals, including a postcard and a drawing:

The program was even able to correctly identify Digital Trends from a text-free logo, adding “consumer electronics” and “news” among the list of keywords.

As a neural network system, the platform will continue to “learn,” tracking any human input to use with future scans.

The platform will even put the keywords together into a brief caption. “In some cases, we have human annotators write a caption for the image, and use these entities as keywords,” Smith explained. “In other cases, the CMP actually uses natural language processing to turn the entities into an automatically generated, human-readable sentence caption for the image.”

In the image of Obama, the software generated this short caption: “44th President of the United States Barack Obama at the White House, Washington, DC, December 5, 2013.”

Through a partnership with Getty Images, Gado Images is working to allow museums, art galleries, and private collectors to monetize their collections. That gives non-profit historical organizations a recurring revenue source, and also saves time (and money) on the scanning and labeling process. In order for buyers to find an image they would like to use, they need to be tagged correctly, which is where the keywords and captions from the Cognitive Matadata Platform come in.

“Its a great way for a cultural heritage organization to take their archives and turn it into a source of recurring revenue,” Smith said. Along with serving as a platform to monetize historical collections, the automatic keyword and caption generation saves organizations from the time — and cost — required to add the information manually.

While the company has moved from physical hardware to software development, the new program still jives with the company’s mission to digitize and share the world’s history. One of the company’s earliest projects was to help the Baltimore Afro American — the longest running African American Newspaper — digitize a 1.5-million image archive. “It’s really expensive and challenging to digitize and annotate archives at large. They had only scanned about 5,000 images at the time, so we launched Gado to help organizations to scan images and then make sense of those photos in a way that makes sense for a small organization,” Smith said.

While the software’s current focus is historical visual data, Smith says the software has the potential to be used in a variety of industries in the future. For example, a media company could use the system to create an archive and easily find images of a specific location, object, or person, or photographers could use the system to archive their work and quickly bring up an image of a specific client.


Alpha Female: Sony awards five women grants to support artisan diversity

Women can face several challenges in launching a photography career -- Sony's latest initiative aims to help propel women in the industry forward. Sony recently announced the winners of the Alpha Female program.
Home Theater

The best movies on Netflix in November, from 'Buster Scruggs’ to ‘Dracula’

Save yourself from hours wasted scrolling through Netflix's massive library by checking out our picks for the streamer's best movies available right now, whether you're into explosive action, witty humor, or anything else.

Here's why 64-bit (not 32-bit) dominates modern computing

Today's computing world isn't the same as it once was. With 64-bit processors and operating systems replacing the older 32-bit designs, we look at what 32-bit vs. 64-bit really means for you.

These are the 5 best free antivirus apps to protect your MacBook

Malware protection is more important than ever, even if you eschew Windows in favor of Apple's desktop platform. Thankfully, protecting your machine is as easy as picking from the best free antivirus apps for Mac suites.

The 100 best Android apps turn your phone into a jack-of-all-trades

Choosing which apps to download is tricky, especially given how enormous and cluttered the Google Play Store has become. We rounded up 100 of the best Android apps and divided them neatly, with each suited for a different occasion.
Emerging Tech

The best drone photos from around the world

Most of today's drones come equipped with high-end cameras, which are quickly revolutionizing the world of aerial photography as we know it. Here are some of the best drone photos from around the world.
Product Review

Fewer pixels, better camera? The Nikon Z6 shows the beauty of restraint

The Nikon Z6 is the sibling to the new mirrorless Z7 -- but for some photographers, the cheaper Z6 may be the better option. Read where the $2,000 camera beats the $3,400 one (and where it doesn’t) in our Nikon Z6 review.
Social Media

Build a wish list and shop videos with Instagram’s latest shopping update

Eyeing a product on Instagram? Now there are more ways to shop from the social network. Instagram just rolled out options to save products in a collection as users can also now shop from videos.

See the National Forests like never before in these awe-inspiring drone videos

What's the difference between a National Park and a National Forest? Drones. With no ban on drones in National Forests -- at least, not yet -- filmmakers have a way to capture the immensity of these locations with stunning results.
Product Review

With outstanding image quality, the ‘basic’ Sony A7 III excels in every way

Replacing the four-year-old A7 II as the new entry-level model in Sony's full-frame line, the A7 III is an impressively capable camera that gives more expensive models a run for their money.
Social Media

Addicted to Instagram? Its new ‘activity dashboard’ is here to help

Ever get that nagging feeling you're spending too much time on Instagram? Well, a new "activity dashboard" has a bunch of features designed to help you better control how you use the addictive photo-sharing app.

Photography news: Best spot for fall photos, new firmware from Fuji and Nikon

Where's the best spot to take fall photos? Michigan, according to social media and a Nikon contest. The results and more in this week's photography news, including significant firmware updates for the Fujifilm X-T3, X-H1, and GFX 50S.

How to take great photos with the Pixel 3, the best camera phone around

You’ve scored yourself a new Google Pixel 3 or Pixel 3 XL, and you want to take advantage of that incredible camera. We’ve got everything you need to know right here about how to snap the best photos with your Pixel 3.
Emerging Tech

DJI Mavic 2 Pro vs Mavic 2 Zoom: What’s the real difference?

DJI's Mavic 2 series drones are ready to fly -- but which one is right for you? The Mavic 2 Pro and Mavic 2 Zoom are nearly identical save for their cameras. Here's what you need to know about these powerful new UAVs.