MIT and IBM’s new A.I. image-editing tool lets you paint with neurons

Whether it’s automatically tagging objects in pictures or the ability to tweak lighting and separate subjects from their background using the iPhone’s “portrait mode,” there’s no doubt that artificial intelligence is a powerful force in modern photo-editing tools.

But what if it were possible to go one step further, and use the latest cutting-edge technologies to develop what may just be the world’s most ambitious (and, in its own way, imaginative) paint program — one that goes far beyond simply touching up or coldly analyzing your existing pictures?

With such a program, all a person would need to do to remove an unsightly line of cars sullying a picture of their family home would be to pass over it with a brush. As if by magic, the vehicles would be replaced by a photorealistic grassy bank. Want to eliminate that photobomber from one of your vacation snaps? No problem: Just click to select them and they’ll vanish in place of a utility pole that looks like it’s always been there. How about adding an authentically ancient door into a photo of an old church? Click and it’s done. You get the idea.

This is what researchers at Massachusetts Institute of Technology and IBM are working toward with an amazing new tech demonstration they call the “GAN Paint Studio.” Described by its creators as providing the ability to “paint with neurons” — referring to the artificial neurons of a machine learning neural network — it’s one of the most potentially transformative photo-editing tools yet created.

It allows users to upload an image of their choosing and then modify any aspect of it they want, whether that’s changing the size of objects or adding in completely new items and objects. Think of it as Photoshop for the “deepfake” generation, albeit one that’s currently more of a proof-of-concept than a finished product.

The future of creative tools

“What we created with this work is a starting point to show how creative tools in the future could work,” Hendrik Strobelt, a research scientist at the MIT-IBM Watson A.I. Lab, told Digital Trends. “We started from a neural network [called a] GAN that can produce its own images of a certain category — for example, kitchen images — and analyzed which internal parts of the network are responsible for producing which feature. This allowed us to modify the images that the network produced. We ‘drew’ on them. The novelty we added is that you can upload your own image of this category and modify it with brushes that do not just draw strokes, but actually draw semantically meaningful units — such as trees, brick-texture, or domes.”

A GAN, or Generative Adversarial Network, is one of the most powerful tools used in generative artificial intelligence. A GAN pits two artificial neural networks against one another. One network generates new images, while the other attempts to work out which images are computer-generated and which are not. Over time, this generative adversarial process causes the “generator” network to become good enough at creating images that it can successfully fool the “discriminator” every time. A GAN was the technology behind the A.I. artwork that famously sold for big bucks at a Christie’s auction in 2018.

The system developed by the MIT and IBM researchers showcases some neat abilities. A bit like Deep Dream, the trippy image-generating tool developed by Google researchers several years back, it shows an impressive understanding of which images fit together. As a result of being trained on a vast archive of images, it picks up an understanding of the basic rules governing relationships between objects. For instance, ask it to add an object in the sky and it won’t draw a window — since it knows that windows aren’t usually (or ever) found there.

As Strobelt notes, GAN Paint Studio is not quite ready for prime time just yet. Although members of the public can have a go at using it, there’s still more work to be done. Notably, the demonstration version is currently low-resolution. However, it does showcase the immense promise of the technology.

Challenging imagination

“The most fun parts [of the technology] are actually when your imagination is challenged,” Strobelt said. “Try adding a door to the Palazzo Vecchio image; it’s kind of mind-blowing if you know the place. The system is far from perfect, and not every image can be modified equally well. There is still research needed on how to optimize all the parts. For example, when the GAN model tries to represent the input model, it might very well use the wrong semantical units to reproduce features — it [may] just generate a door out of tree units. Figuring out when and how it does do right or wrong is actually very interesting future work.”

“I see this as an advanced tool to help humans who think they are not creative to challenge this thought.”

Just as GANs get better over time, so Strobelt thinks that the applications for GAN Paint Studio will open up. “The obvious first idea would be a photo editor with these semantic brushes and erasers,” he said. “This could help you edit vacation photos, for example. It could also allow architects to quickly create variations on the embedding of their building renderings. Game designers could [also use it to] modify level maps quicker.”

If such technology could be added to video effects, it would also prove immensely powerful. This would allow objects to be placed into shots with just the touch of a button. Should a director realize they’ve forgotten to include a background item that’s crucial to the plot in a completed scene, it could be quickly added in — without the need for the current expensive and time-consuming visual effects processes.

Strobelt is decisive in saying that he doesn’t think GAN Pain Studio is truly, autonomously creative. “No,” he said, decisively. “I see this as an advanced tool to help humans who think they are not creative to challenge this thought.”

Then again, what is creativity? As with many other aspects of our lives, such as the jobs we believe only humans can do, it seems that A.I. is ready to ask the big questions.


It’s 2025. How has 5G changed our lives? We asked experts to predict the future

2025 is the year that mobile carriers say 5G goes mainstream. How will the technology change, revolutionize, or make obsolete the things we do today? We asked futurists to give us an idea of our future 5G connected world.
Small Business

The 15 best tech jobs boast top salaries, high satisfaction, lots of openings

The bonanza of tech jobs just keeps coming. High-paying tech jobs abound at companies where people love to work. If you’re ready to make a change, this is a great time to look for something more fulfilling.   
Emerging Tech

Astro the dog-inspired quadruped robot can sit, lie down, and… learn?

Move over Spot! Researchers from Florida Atlantic University have built a new dog robot called Astro. Thanks to deep learning technology, it promises to be able to learn just like a real dog.
Emerging Tech

Google’s soccer-playing A.I. hopes to master the world’s most popular sport

Think the player A.I. in FIFA ‘19 was something special? You haven’t seen anything yet! That’s because Google is developing its own soccer-playing artificial intelligence. And, if the company’s history with machine intelligence is…
Emerging Tech

Eric Geusz: Apple engineer by day, spaceship designer by night

An Apple software engineer by day, artist Eric Geusz spends his nights drawing everyday household objects as amazing, science fiction-style spaceships. Check out the impressive results.
Emerging Tech

The black hole at the center of our galaxy is flaring and no one knows why

At the heart of our galaxy lies a supermassive black hole, Sagittarius A*. Normally this giant monster is relatively docile, but recently it's been a hotbed of unexpected activity, rapidly glowing 75 times brighter than normal.
Emerging Tech

Wreckage, reefs, and robots: The high-tech quest to find Amelia Earhart’s plane

Over 80 years ago, American aviation pioneer Amelia Earhart disappeared while trying to fly around the world. Now an autonomous water-based robot could help find some answers. Here's how.
Emerging Tech

SpaceIL’s crashed lander may have sent thousands of tardigrades to the moon

When the SpaceIL craft Beresheet crashed into the moon earlier this year, it left more than just an impact mark. Thousands of micro-animals called tardigrades were along for the ride and may have survived the crash.
Emerging Tech

NASA’s satellite projects will study the sun using solar sailing

Small satellites can be used for all sorts of purposes, and NASA has been searching for ideas to push ahead the capabilities of the hardware. The agency has announced two new projects to demonstrate the potential of small satellites.
Emerging Tech

Hubble captures a beautiful cosmic jellyfish made of glowing gas

A new image from Hubble might look like a deep-space jellyfish, but it's not a sign of extraterrestrial life - in fact, it's a planetary nebula called NGC 2022, located in the constellation of Orion (The Hunter).
Emerging Tech

Parker Solar Probe makes a second orbit of the sun, captures solar wind on video

The Parker Solar Probe, launched last year, has completed its second orbit around the sun. To celebrate, the team responsible for the probe has released a video showing solar winds in action.

Starman on Tesla Roadster makes first orbit around sun, braces for loneliness

Starman and his Tesla Roadster, sent by SpaceX to outer space last year, have completed their first orbit around the sun. The people on Earth may be able to catch a glimpse of the cherry-red electric vehicle on November 2020.
Emerging Tech

Bernie Sanders calls for a ban on police use of facial recognition

Democratic presidential hopeful Sen. Bernie Sanders is calling for a complete ban on the use of facial recognition by law enforcement as part of the new criminal justice reform plan he introduced this weekend.
Emerging Tech

On the fence about buying solar panels? Tesla now offers them for rent

With solar rental Tesla says “customers get the best from solar power — clean, cheap energy to power homes and vehicles — without upfront costs or decades-long agreements. In fact, customers can get solar power with one click, instead…