Why GANs are hard to train

The original objective#

Goodfellow et al. (2014) define the minimax game:

$$ \min_G \max_D \; V(D, G) = \mathbb{E}{x \sim p{\text{data}}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log (1 - D(G(z)))] $$

$D$ is the discriminator, mapping an image to a probability. $G$ is the generator, mapping a noise vector $z$ to an image. $p_{\text{data}}$ is the data distribution, and $p_z$ the prior on $z$, conventionally $\mathcal{N}(0, I)$.

For a fixed $G$ the optimal discriminator is:

$$ D^*(x) = \frac{p_{\text{data}}(x)}{p_{\text{data}}(x) + p_g(x)} $$

where $p_g$ is the generator's implied distribution. Substituting it back gives:

$$ V(D^*, G) = 2 \cdot \mathrm{JSD}(p_{\text{data}} \,|\, p_g) - 2\log 2 $$

$\mathrm{JSD}$ is the Jensen-Shannon divergence. At the optimum the generator is minimising JSD.

Why that result is the problem, not the solution#

If $p_{\text{data}}$ and $p_g$ have disjoint supports, $\mathrm{JSD} = \log 2$ everywhere and its gradient vanishes. Natural images lie on a low-dimensional manifold in pixel space. So early in training the supports really are close to disjoint. See Arjovsky and Bottou (2017), Towards Principled Methods for Training GANs.

The saturating generator loss $\log(1 - D(G(z)))$ therefore has vanishing gradient exactly when $D$ is confident. The non-saturating alternative $-\log D(G(z))$ shares the fixed point but has a gradient that grows as $D(G(z)) \to 0$. Arjovsky and Bottou show it is equivalent to minimising $\mathrm{KL}(p_g | p_{\text{data}}) - 2\,\mathrm{JSD}(p_{\text{data}} | p_g)$. The second term carries the wrong sign, and rewards mode-seeking behaviour. Non-saturating loss trades vanishing gradients for a bias toward mode collapse.

Convergence is not guaranteed by the objective#

The game is not a convex-concave saddle-point problem in parameter space. Mescheder et al. (2018) ask Which Training Methods for GANs do actually Converge?. They show unregularised GAN training need not converge locally, even on an absolutely continuous data distribution. Eigenvalues of the Jacobian at equilibrium can have zero real part, producing orbits rather than convergence.

Their fix is the R1 penalty, a zero-centred gradient penalty applied on real data only:

$$ R_1(\psi) = \frac{\gamma}{2}\, \mathbb{E}{x \sim p{\text{data}}}!\left[|\nabla_x D_\psi(x)|^2\right] $$

$\psi$ are the discriminator parameters and $\gamma$ the penalty weight. They prove local convergence for the regularised game.

The main stabilisers, and what each buys#

WGAN has the strongest theoretical motivation. The Wasserstein-1 distance stays continuous and yields useful gradients even for disjoint supports:

$$ W_1(p_{\text{data}}, p_g) = \sup_{|f|L \le 1} \; \mathbb{E}{x \sim p_{\text{data}}}[f(x)] - \mathbb{E}_{x \sim p_g}[f(x)] $$

The supremum runs over 1-Lipschitz functions $f$. Enforcing that constraint is the entire practical difficulty. Weight clipping (the original proposal) distorts the critic; the gradient penalty of Gulrajani et al. instead penalises $(|\nabla_{\hat{x}} f(\hat{x})|_2 - 1)^2$ at points $\hat{x}$ interpolated between real and fake samples.

Mode collapse, precisely#

Mode collapse is the case where $p_g$ concentrates on a strict subset of the support of $p_{\text{data}}$. It does not show up in the value function. A generator covering one mode perfectly can look fine to a discriminator that has collapsed to the same region.

Measurement needs a separate instrument. On synthetic mixtures, count recovered modes and reverse KL, as Metz et al. (2017) do for unrolled GANs. On real images the standard pair is:

• FID (Heusel et al., 2017): Fréchet distance between Gaussians fitted to Inception features. Sensitive to fidelity and diversity together, which is also its weakness, and biased by sample count.
• Precision and recall for generative models, from Sajjadi et al. (2018) and Kynkäänniemi et al. (2019). This separates fidelity (precision) from coverage (recall). Mode collapse appears as high precision with low recall, which FID alone cannot reveal.

Report both. A paper reporting only FID has not shown that its model covers the data.

Where GAN training stands now#

Diffusion models displaced GANs for open-domain text-to-image generation, but the stability story did not end there.

Huang et al. (2024) derive a regularised relativistic loss carrying both R1 and R2 penalties. They prove local convergence for it. That lets them discard the accumulated bag of tricks. Their paper is The GAN is dead; long live the GAN! A Modern GAN Baseline, NeurIPS 2024. Their R3GAN uses a plain ResNet-and-ConvNeXt style backbone, with no StyleGAN-specific machinery. It surpasses StyleGAN2 on FFHQ, CIFAR, ImageNet and Stacked MNIST.

The claim worth taking from that work is narrow and important. Much of the historic instability was an artefact of a badly conditioned objective. It was not inherent to adversarial training.

GANs remain the practical choice wherever a single forward pass matters. They still dominate real-time super-resolution and on-device image enhancement.

Papers#

• Goodfellow et al., Generative Adversarial Networks, 2014 — arxiv.org/abs/1406.2661
• Arjovsky and Bottou, Towards Principled Methods for Training GANs, 2017 — arxiv.org/abs/1701.04862
• Arjovsky et al., Wasserstein GAN, 2017 — arxiv.org/abs/1701.07875
• Gulrajani et al., Improved Training of Wasserstein GANs, 2017 — arxiv.org/abs/1704.00028
• Heusel et al., GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium, 2017 — arxiv.org/abs/1706.08500
• Miyato et al., Spectral Normalization for Generative Adversarial Networks, 2018 — arxiv.org/abs/1802.05957
• Mescheder et al., Which Training Methods for GANs do actually Converge?, 2018 — arxiv.org/abs/1801.04406
• Kynkäänniemi et al., Improved Precision and Recall Metric for Assessing Generative Models, 2019 — arxiv.org/abs/1904.06991
• Huang et al., The GAN is dead; long live the GAN! A Modern GAN Baseline, NeurIPS 2024 — arxiv.org/abs/2501.05441

What to learn next#

• StyleGAN and latent space — what a stabilised GAN can do once it trains.
• GANs — the architecture from the beginning, if this page moved too fast.
• Diffusion models — the family that replaced GANs for text-to-image.