Computers saw Jesus, graffiti, and selfies in this art, and critics were floored

barnes foundation room
Barnes Foundation

The Barnes Foundation art gallery, located in Philadelphia, boasts a collection of famous work you’d probably recognize, whether that’s Vincent van Gogh’s Portrait of the Postman Joseph Roulin and Paul Cézanne’s The Card Players.

But its founder, Albert C. Barnes, had a more unconventional perspective on how to interpret the art — and that perspective is now being reflected in machine learning, of all places. Rather than getting caught up in the minutiae of dates and movements, Albert C. Barnes extolled the virtues of making visual connections between different artworks. Apparently computers agree with him.

A century after he started assembling his collection, machine learning is helping push the boundaries of his unorthodox ideas about how art should be viewed.

The Gallery Comes to You

The Barnes Foundation has always been an essential stop for any art fans in the area — and now, you can enjoy its contents from anywhere in world.

“Dr. Barnes had a very specific point of view about how you view work.”

Under the supervision of chief experience officer Shelley Bernstein, The Barnes Foundation has embarked upon an effort to make its collection available online. This is far more than just a digitization effort; it’s a technologically enhanced expansion of the ethos that the organization was born out of in the early 20th century.

“I’m a big believer in what I call mission-driven tech — thinking about the goals of the institution, like the fact that this was an early teaching institution,” Bernstein explained when she spoke to Digital Trends about the work she’s doing at the Barnes. “Dr. Barnes had a very specific point of view about how you view work, and that you might not necessarily need to know the history behind it.”

Over a period of years, Barnes amassed a collection of paintings, sculptures, jewelry, and furniture. Each wall of each room is meticulously arranged in order to forge connections between pieces that may have been made years apart, halfway across the world from one another.

Barnes tweaked each of his “ensembles” as he obtained new pieces, but they have remained in the exact same configuration since his death in 1951. It makes sense for the physical space to be preserved in this manner, both for practicality’s sake and as a tribute to the foundation’s founder. However, the website allows individuals to forge new links for themselves.

It’s here where machine learning is being used to foster some unusual responses to great art.

An interface on the website allows visitors to access digitized versions of any of the works in the collection. Clicking on a particular piece offers up the opportunity to see it in context as part of an ensemble. There’s also a method of seeing visually related pieces using a slider toggle that stretches from “more similar” to “more surprising.” It’s here where machine learning is being used to foster some unusual responses to great art.

Several different computer vision and machine learning experts were called upon to help the Barnes feed its collection into several different systems. Each one came up with very different results.

“I remember that a developer who was working on the project came to me and said, ‘Shelley this is insane, it’s all wrong,’” Bernstein remembered. “A curator looked at it and said ‘Actually, no — that’s kind of interesting.’”

Automated Art Critic

The machine learning systems that were being fed the Barnes’ collection made observations that might never have come from humans. Given the organization’s history of encouraging visual connections between otherwise disparate works, this seemed like a huge opportunity.

The computers would label a classical artwork as graffiti. They might see soft nudes by Renoir as teddy bears and stuffed animals. Frequently, they would tag bearded faces and other similar shapes as images of Jesus Christ.

philadelphia art gallery the barnes foundation uses machine learning building
The Barnes Foundation
The Barnes Foundation

It would be easy to write off these observations as limitations of the computer vision software, but time and again, curators found that the computers had a point. They were simply making visual connections without any of the second-guessing that a human mind might introduce to the process.

Berstein’s team uses six different tools to analyze the art. She describes Microsoft Azure as offering up the “coolest” results, referring to the most interesting, surprising categorizations. IBM Watson and Google’s offering were apparently successful as well — but much more conservative.

So while machine learning hasn’t completely replaced human curation from the equation, it does offer another perspective that has proven to be pretty unique.

Open Source Art

Obviously, certain parts of its digitization effort are very closely related to its specific ethos and mandate — but all the code is open source, so other institutions can pick and choose the parts that they can make use of and build from an established foundation.

The computers would label a classical artwork as graffiti.

Of course, the open source nature of the digitization isn’t just for other museums and galleries. Just weeks after the digitized collection went online, there were already some interesting projects at play on Twitter that take the ideas even further.

Andrei Taraschuk describes himself as a software developer by day and an art hacker by night. When he saw the online version of the Barnes collection, he knew that he had to put it to good use. Having already created Twitter bots that share a particular artist’s work, he created a Barnes bot that would post images of artwork from the foundation’s collection.

“It’s so cool,” said Bernstein with a grin when we spoke about the Barnes bot. The account doesn’t work in a vacuum — it retweets images shared by other institutions and artists that are part of his ever-growing bot network, forging links between seemingly disparate pieces. Albert C. Barnes never got to experience Twitter, but it seems fair to assume that he would approve of this kind of usage.

Roughly half of the pieces that comprise the Barnes collection are in the public domain, and the foundation makes high quality TIFF files of those works available online. This makes it easy for Taraschuk’s bots to share them, or for UK-based illustrator Sarah McIntyre to run a ‘portrait challenge’ using a particular painting, where Twitter users are invited to produce their own take on an existing artwork.

At their core, these efforts are all about opening up art to a broader swathe of people. If you’re not close to Philadelphia, you can still enjoy and appreciate the collection. if you’ve ever felt that the traditional gallery experience was too stuffy, maybe having a computer show you which classic works most resemble a selfie, or getting a drip feed of fine art via Twitter will be more palatable.

Perhaps more importantly though, it shows that bringing historical fine art into the digital world fundamentally changes how we view and interpret it. Dr. Barnes just may have been ahead of his time.


Nvidia’s new GPUs look amazing, but that doesn’t mean you should buy one

Nvidia's GeForce 2080 is a powerful graphics card that supports ray tracing to deliver real-time cinematic renderings of shadows, light, and reflection in games, but unless you were already planning on upgrading, you'll probably want to…

A library at your fingertips: The 101 best free Kindle books

You shouldn't always have to pay for a good read. Here are our picks for the best free Kindle books, so you don't have to sort through thousands of titles on Amazon and Google Play. We divided them into various genres, so dive in.

The N64 was an all-star console with a line-up of all-star games. These were the best.

The Nintendo 64 introduced a long list of top-tier games, but which were the iconic platform's best? From Mario Party to Ocarina of Time to NFL Blitz, check out our picks for the best N64 games.

Samsung rebrands Gear app as ‘Galaxy Wearables,’ supports Android 9.0 Pie

Following reports that Samsung Gear owners were experiencing connectivity issues after downloading Android 9.0 Pie, the company released an update to the app. The Samsung Gear app is also now officially known as Galaxy Wearable.

Reluctant to give your email address away? Here's how to make a disposable one

Want to sign up for something without the risk of flooding your inbox with copious amounts of spam and unwanted email? You might want to consider using disposable email addresses with one of these handy services.

Both the Razer Blade and XPS 15 are capable laptops, but which is better?

We pit the latest Dell XPS 15 against the latest Razer Blade 15 to see which machine meets the needs of most people. Both are a fast, attractive, and well-built, but they still appeal to different users.

Logitech’s distinctive new ergonomic mouse looks as good as it feels

Logitech's first true ergonomic mouse sports an interesting tilted design that encourages less muscle strain. We spent some time with the MX Vertical to see how comfortable it is and determine whether or not we'd prefer it to a standard…

Use one of these password managers to stay safe online

The internet can be a scary place, especially if you don't have a proper passcode manager. This guide will show you the best password managers you can get right now, including both premium and free options. Find the right password software…

Airport’s low-tech solution to digital chaos involves the humble whiteboard

A U.K. airport has suffered a major computer error, caused by data connection problems, which has stopped flight boards from showing crucial passenger information. The solution is wonderfully low-tech.

Here’s how to watch Nvidia’s GeForce event at Gamescom

Today is August 20, and that means Nvidia may showcase its GeForce RTX 20 Series of add-in graphics cards for gamers. We’re sticking with that name rather than the previous GTX 11 Series brand due to today’s date.

HTC breaks down VR barriers by bringing Oculus Rift titles to Viveport

HTC's Viveport store and subscription service will be opened to Oculus Rift users in September this year, letting them buy titles directly and take advantage of the monthly game-delivery service.

Dell’s new fast-refresh Freesync display could be your next great gaming screen

Dell has debuted a pair of new gaming TN displays, each offering high refresh rates and fast response times to gamers alongside Freesync technology. There are 24- and 27-inch versions of the new screens available now.

Nvidia’s GeForce RTX 20 Series starts at $500 and features real-time ray tracing

Nvidia revealed its new GeForce RTX 2000 Series of add-in desktop graphics cards for gamers during its pre-show Gamescom press event. The new family is based on Nvidia’s new “Turing” architecture focusing on real-time ray tracing.

Nvidia GeForce RTX GPUs are coming to Alienware and Predator gaming desktops

Dell and Acer have both announced support for Nvidia's new GeForce RTX 2000 graphics cards in refreshed gaming desktops, including Predator Orion series systems and Alienware desktops.