Nvidia’s smart paintbrush tool turns simple doodles into works of art

A new drawing app enhanced with deep learning functions could soon convert your simple sketches and doodles to brilliant and realistic works of art.

Nvidia recently announced the development of GauGAN, a smart drawing app that uses a deep learning model to generate photorealistic versions of simple drawings created by human artists. The name of the app, GauGAN, refers both to the French post-Impressionist painter Paul Gauguin and the app’s use of the generative adversarial networks (GANs) to develop its realistic images.

So how does GauGAN work? According to Nvidia’s announcement, the app is like a “smart paintbrush” that fills in the details of your drawing (referred to as a “segmentation map”). Essentially, you provide the outline of what you want and label each segment, dictating what it’s supposed to be. From there, GauGAN takes over, filling in the extra details to make your drawings more realistic:

“Trained on a million images, the deep learning model then fills in the landscape with showstopping results: Draw in a pond, and nearby elements like trees and rocks will appear as reflections in the water. Swap a segment label from “grass” to “snow” and the entire image changes to a winter scene, with a formerly leafy tree turning barren.”

The GANs used in GauGAN are essentially a partnership between two networks, a generator and a discriminator. Within this partnership, a generator develops a series of possible images and the discriminator acts as a sort of editor that teaches the generator “how to improve the realism of its synthetic images.”

It’s also worth noting that while GauGAN uses its extensive knowledge of other images to create these drawings, these drawings are still considered original works since the app is generating new images.

Nvidia’s new drawing app isn’t just limited to nature scenes or landscapes—it can also add in buildings, roads, or even people. GauGAN also allows artists to add style filters. Such filters including mimicking a specific artist’s style or adjusting the lighting of the image from a day to a night scene.

It’s unclear if GauGAN will ever be released for consumer use, but it is expected to be a particularly helpful tool for architects, urban planners, and even game developers.