Study guide · Neural networks · 2014

Generative Adversarial Networks

Ian Goodfellow et al. · NeurIPS 2014

How do you teach a network to produce realistic new images when you cannot write down a formula for "realistic"? Goodfellow and colleagues train two networks against each other: a generator that turns random noise into samples, and a discriminator that tries to tell those samples from real data. Each improves by exploiting the other's weaknesses, and in the ideal limit the generator's samples become indistinguishable from the real thing.

The paper in four sentences

A generator network takes random numbers and turns them into a sample, such as an image. A discriminator network looks at a sample and says how likely it is to be real rather than generated.

The discriminator is trained to catch fakes; the generator is trained, by backpropagating through the discriminator, to make samples the discriminator calls real.

The paper proves that if both networks were flexible enough and trained perfectly, this contest has exactly one end point: the generator produces the data distribution itself, and the discriminator can do no better than guessing, 50/50.

In practice the contest can be unstable, the generator can get stuck producing only a few kinds of output, and there is no easy way to measure how good the model is. Later work spent years fixing these problems.

Idea · §1–2

The problem: making new data

By 2014, neural networks were very good at recognising things: given a photo, say "cat". The reverse, generating a new, realistic photo of a cat, was much harder. The approaches of the time mostly tried to write down a probability for every possible image and then adjust it to fit the data. For rich models those probabilities involve sums over astronomically many possibilities, so the methods relied on slow approximations, often long chains of random sampling.

This paper sidesteps all of that. It never writes down the probability of an image. It only needs a network that can produce samples, plus a way to tell it whether its samples are any good. The "way to tell it" is a second network.

Everyday pictureThe paper's own analogy: the generator is a team of counterfeiters trying to print convincing banknotes, and the discriminator is the police trying to spot fakes. Competition drives both to improve until the fakes cannot be told from genuine notes.

TakeawayInstead of writing down how probable each image is, train a sample-maker and judge its output with a learned critic.

Idea · §3 · Fig. 1 · Algorithm 1

Counterfeiter against detective

Training alternates two moves. First, show the discriminator a handful of real examples and a handful of generated ones, and nudge it to score real ones near 1 and fakes near 0. Second, generate a fresh batch of fakes and nudge the generator so the discriminator scores them higher. The generator never sees real data directly; everything it learns comes through the discriminator's judgement, passed back by backpropagation.

Below is a complete GAN in one dimension, small enough to watch. Real data are numbers clustered around 1.5 (the dashed bump). The generator takes a random number and stretches and shifts it, so it can learn where to put its output and how spread out to make it. The discriminator's opinion is the amber line: high where it thinks "real", low where it thinks "fake". This mirrors the paper's Fig. 1.

Dashed: where real data lie. Blue bars: a histogram of 2,000 generator samples. Amber: the discriminator's output D(x), from 0 (sure it is fake) to 1 (sure it is real); the dotted line marks 0.5. Each round is one discriminator update and one generator update, on batches of 128. Notice the generator first squeezes its output into a narrow spike as it races toward the data, then widens to match the spread: a brief, small-scale version of the "collapse" problem discussed below. Illustrative toy: a two-parameter generator and a discriminator that can only draw bell-shaped boundaries.

TakeawayAlternate: train the detective on real versus fake, then train the counterfeiter to fool the current detective. The detective's verdict is the only teaching signal the generator gets.

Check yourself
How does the generator know which direction to change its output?
It backpropagates through the discriminator: the slope of the discriminator's output with respect to the sample tells it which way would look "more real".
Why does the amber line flatten out near 0.5 when training goes well?
When fakes and real data overlap completely, no rule can tell them apart, so the best the discriminator can say anywhere is "50/50".

Idea · §4.1 · Proposition 1, Theorem 1

Where the contest ends

The paper's theory asks: suppose the discriminator were perfect for whatever the generator is currently doing. What would it say? The answer is simple. At each possible sample, compare how common it is among real data with how common it is among fakes. The ideal discriminator outputs real ÷ (real + fake). Where only real data appear, it says 1; where only fakes appear, 0; where both are equally common, ½.

Then they show that the generator's best response to such a discriminator is to make its distribution identical to the real data. At that point the detective is reduced to coin-flipping, and nothing either player can change will help it. That is the game's unique ideal end point.

Drag the fake distribution around below and watch what the ideal detective would do.

Dashed: real data (centre 0, spread 1). Blue: the generator's distribution. Amber: the ideal discriminator real ÷ (real + fake), on a 0 to 1 scale. "Best possible accuracy" is how often that ideal detective would be right on a 50/50 mix of real and fake samples. Set centre 0 and spread 1 to reach the end point.

TakeawayThe perfect detective compares how common a sample is in real versus fake data; the only way to beat it is to make those two the same everywhere.

Check yourself
If fakes are twice as common as real data at some point, what does the ideal detective say there?
real ÷ (real + fake) = 1 ÷ (1 + 2) = ⅓.

Idea · §3, §6 · Table 2

What can go wrong

The proof assumes infinitely flexible networks and a discriminator retrained to perfection at every step. Real training takes small alternating steps, and the authors list the consequences honestly:

  • The pair must stay in step. If the generator trains too long against a stale discriminator, it can learn to send many different random inputs to the same output. The paper calls this the "Helvetica scenario"; it is now known as mode collapse: a face generator that only makes a few faces.
  • Weak early signal. At the start, fakes are obviously fake and the discriminator rejects them confidently, which in the original formulation gives the generator almost no gradient. The paper's fix is to train the generator to make the discriminator say "real" rather than to stop it saying "fake"; the end point is the same but the early push is much stronger.
  • No score. A GAN never computes the probability of an example, so it is hard to measure progress or compare models.

TakeawayThe game has the right end point in theory, but in practice the two players can chase each other, collapse to a few outputs, or stall; and you cannot easily score the result.

Check yourself
What is mode collapse?
The generator maps many different noise inputs to the same or very similar outputs, covering only part of the variety in the real data.

Idea · §5, §7 · Fig. 2

Small pictures, big afterlife

The 2014 samples are small: handwritten digits, low-resolution faces and blurry CIFAR-10 objects (Fig. 2). The authors show the nearest real training image beside each sample, to demonstrate the network is not just copying. On a likelihood-style score (a rough one, they admit) the GAN was competitive with the best methods of the day.

The conclusion sketches extensions that all happened: conditional GANs (generate a picture of a particular class), learned inference, and semi-supervised learning.

Later refinementConvolutional GANs (DCGAN, 2015), the Wasserstein GAN (2017) for steadier training, and StyleGAN (2019) produced photorealistic faces. GANs led image generation for about five years until diffusion models overtook them on image quality and variety: "Diffusion Models Beat GANs" (2021). The adversarial idea lives on in many places, for example as an extra loss that sharpens the auto-encoders inside latent diffusion.

TakeawayGANs showed that a learned critic can replace an explicit probability model; they dominated image synthesis until diffusion models.

Follow the idea

Led to: Diffusion beats GANs (2021)