Editing images with neural networks
The system, called “GANpaint Studio,” allows a user to upload an image of their choosing and modify multiple aspects of its appearance, from changing the size of objects to adding completely new items like trees and buildings. In addition to helping artists and designers make quick adjustments to visuals, say the researchers, the system may help computer scientists identify “fake” images.
Adapting the system to video clips could enable computer-graphics editors to quickly compose specific arrangements of objects needed for a particular shot. GANpaint Studio, say the researchers, could also be used to improve and debug other GANs that are being developed, by analyzing them for “artifact” units that need to be removed and help researchers better understand neural networks and their underlying structures.
“Right now, machine learning systems are these black boxes that we don’t always know how to improve, kind of like those old TV sets that you have to fix by hitting them on the side, says David Bau, a PhD student at MIT’s Computer Science and Artificial Intelligence Lab (CSAIL)and lead author on a related paper. “This research suggests that, while it might be scary to open up the TV and take a look at all the wires, there’s going to be a lot of meaningful information in there.”
An unexpected discovery, say the researchers, is that the system seems to have learned some simple rules about the relationships between objects – it knows not to put something somewhere it doesn’t belong, like a window in the sky, and it also creates different visuals in different contexts. For example, if there are two different buildings in an image and the system is asked to add doors to both, it doesn’t simply add identical doors – they may ultimately look quite different from each other.
“All drawing apps will follow user instructions, but ours might decide not to draw anything if the user commands to put an object in an impossible location,” says MIT professor Antonio Torralba. “It’s a drawing tool with a strong personality, and it opens a window that allows us to understand how GANs learn to represent the visual world.”
GANs are sets of neural networks developed to compete against each other – in the case of GANpaint, one network is a generator focused on creating realistic images while a second network is a discriminator whose goal is to not be fooled by the generator. Every time the discriminator “catches” the generator, it has to expose the internal reasoning for the decision, which allows the generator to continuously get better.
The researchers say their goal has been to give people more control over GAN networks, but with the recognition that with increased power comes the potential for abuse – like using such technologies to doctor photos. Co-author Jun-Yan Zhu, a postdoc at CSAIL, says that he believes that better understanding GANs — and the kinds of mistakes they make — will help researchers be able to better stamp out fakery.
“You need to know your opponent before you can defend against it,” says Zhu. “This understanding may potentially help us detect fake images more easily.”
For more, see “Semantic Photo Manipulation with a Generative Image Prior.”
Related articles:
AI breakthrough renders interactive 3D environments
New AI method generates realistic human faces
AI generates synthetic MRIs in medical research advance
If you enjoyed this article, you will like the following ones: don't miss them by subscribing to :
