GANs

The minimax objective#

Goodfellow et al. (2014) define a two-player zero-sum game between a generator G: Z → X and a discriminator D: X → (0,1):

min_G  max_D  V(D, G)  =  E_{x ~ p_data} [ log D(x) ]  +  E_{z ~ p_z} [ log(1 − D(G(z))) ]

• p_data — the true data distribution over X
• p_z — a fixed prior over the latent space, typically N(0, I)
• p_g — the distribution induced on X by pushing p_z through G
• D(x) — the probability the discriminator assigns to x being real

The optimal discriminator, and what the generator is really minimising#

For fixed G, the inner maximisation has a closed-form solution obtained pointwise:

D*_G(x)  =  p_data(x) / ( p_data(x) + p_g(x) )

Substituting back gives the generator's effective objective:

C(G)  =  2 · JSD( p_data ‖ p_g )  −  log 4

where JSD is the Jensen-Shannon divergence. It is non-negative and zero only when p_data = p_g, so the global minimum is C(G) = −log 4 ≈ −1.386, attained uniquely at p_g = p_data, where D*(x) = 1/2 everywhere.

The Developer section's D(real) → 0.50 and D(fake) → 0.50 is this fixed point observed numerically. It is also why the losses carry no quality information: at equilibrium the discriminator loss is 2 log 2 ≈ 1.386 whether the game converged to a good G or is oscillating around a bad one.

The convergence proof in the paper holds in function space, assuming D is optimised to convergence at each step. Neither assumption is met in practice, and the gap between the two is where GAN training pathology lives.

Why the objective fails in practice#

Vanishing gradients. Early in training D(G(z)) ≈ 0, and ∇_G log(1 − D(G(z))) is then near zero. The paper's own remedy is the non-saturating loss: maximise log D(G(z)) instead. Same fixed point, very different gradient magnitude when the generator is losing.

Disjoint supports. Arjovsky & Bottou (2017) show the deeper problem. If p_data is supported on a low-dimensional manifold in R^n — which is the standing assumption for natural images — and p_g is supported on another such manifold, then with probability one the two are disjoint or intersect in a measure-zero set. A perfect discriminator then exists, JSD is constant at log 2, and its gradient with respect to G is zero almost everywhere.

So the theory that justifies the objective breaks precisely under the conditions the method is applied in. This result reframed the whole field.

Wasserstein GANs#

Arjovsky, Chintala & Bottou (2017) replace JSD with the Earth Mover distance, which stays finite and differentiable for disjoint supports. Under the Kantorovich-Rubinstein duality:

W(p_data, p_g)  =  sup_{‖f‖_L ≤ 1}  E_{x~p_data}[ f(x) ]  −  E_{x~p_g}[ f(x) ]

f is a 1-Lipschitz critic producing an unbounded real score rather than a probability. Enforcing the Lipschitz constraint is the entire engineering problem:

• Weight clipping (original WGAN). Works, but biases the critic towards simple functions and is sensitive to the clip value.
• Gradient penalty, WGAN-GP (Gulrajani et al., 2017): add λ E[ (‖∇_x̂ f(x̂)‖₂ − 1)² ] at points x̂ interpolated between real and fake samples. The standard choice.
• Spectral normalisation (Miyato et al., 2018): divide each weight matrix by its largest singular value, estimated by one power iteration per step. Cheap, stable, and now the most widely used constraint.

The claimed benefit — that the critic loss correlates with sample quality — holds partially. It tracks progress within a single run and does not compare across architectures.

Mode collapse#

The Developer section shows a generator abandoning half its target. Formally, p_g concentrates on a subset of supp(p_data), and no term in the objective penalises this. The reverse-KL-like behaviour of the practical non-saturating loss is mode-seeking rather than mode-covering, which is the same reason a variational autoencoder blurs while a GAN sharpens: the two sit on opposite sides of the same trade.

Interventions, roughly in order of adoption:

• Minibatch discrimination (Salimans et al., 2016) gives D features computed across the batch, so a batch of near-identical samples is detectable. Minibatch standard deviation, its cheap descendant, is standard in the StyleGAN line.
• Unrolled GANs (Metz et al., 2017) differentiate the generator update through k steps of the discriminator's future updates, so G cannot exploit a D that is about to adapt.
• Two time-scale update rule (Heusel et al., 2017) uses different learning rates for G and D, with a proof of convergence to a local Nash equilibrium under conditions.
• PacGAN (Lin et al., 2018) feeds D several samples jointly, making a collapsed generator detectable by construction.

Evaluation#

Inception Score (Salimans et al., 2016) — exp( E_x KL( p(y|x) ‖ p(y) ) ). Rewards confident, diverse ImageNet class predictions. Insensitive to intra-class collapse and to matching the real data at all, since it never looks at real images. Barratt & Sharma (2018) document its failure modes; it should not be used alone.

Fréchet Inception Distance (Heusel et al., 2017) — fit Gaussians to Inception pool3 features of real and generated sets and compute

FID = ‖μ_r − μ_g‖²  +  Tr( Σ_r + Σ_g − 2 (Σ_r Σ_g)^{1/2} )

The standard metric, and it is biased upward at small sample sizes, sensitive to the resizing and JPEG pipeline, and reliant on a Gaussian assumption that Inception features do not satisfy. Kynkäänniemi et al. (2023) show FID's ranking can disagree with human judgement on modern generators. KID (Bińkowski et al., 2018) uses an unbiased MMD estimator and behaves better on small samples.

Precision and recall for generative models (Sajjadi et al., 2018; Kynkäänniemi et al., 2019) separate fidelity from coverage, which a single scalar cannot. This is the right instrument for measuring mode collapse, and it is under-used.

Architecture lineage#

• DCGAN (Radford, Metz & Chintala, 2015) — the convolutional recipe that made GANs trainable: strided convolutions instead of pooling, batch norm, no fully-connected hidden layers.
• Progressive growing (Karras et al., 2018) — start at 4×4 and add layers, reaching 1024×1024 faces.
• StyleGAN / StyleGAN2 / StyleGAN3 (Karras et al., 2019, 2020, 2021) — a mapping network to an intermediate latent space W, per-layer style modulation, and in v3 a redesign to make generation equivariant to translation and rotation, removing texture "sticking" in video.
• BigGAN (Brock et al., 2019) — class-conditional at scale, with the truncation trick trading diversity for fidelity at sampling time.
• Conditional GANs (Mirza & Osindero, 2014), pix2pix (Isola et al., 2017) for paired translation, and CycleGAN (Zhu et al., 2017) for unpaired translation via a cycle-consistency loss.

Where GANs stand now#

Diffusion models overtook GANs on image synthesis (Dhariwal & Nichol, 2021) and the gap widened with latent diffusion. The reasons are structural: a diffusion model optimises a stable likelihood-like objective with no adversarial equilibrium to balance, and mode coverage follows from the training objective rather than needing to be defended.

Three roles remain, all genuine:

One-step generation. A GAN samples in a single forward pass. This is why diffusion distillation — turning a many-step sampler into a one-step generator — reaches for adversarial objectives: Adversarial Diffusion Distillation (Sauer et al., 2023) is a GAN loss wrapped around a distilled diffusion student. GigaGAN (Kang et al., 2023) scales a GAN to text-to-image at competitive quality with far faster sampling.

Perceptual super-resolution. SRGAN (Ledig et al., 2017), ESRGAN and Real-ESRGAN remain the practical choice for upscaling, where the adversarial term supplies plausible high-frequency detail that a reconstruction loss averages away.

Neural vocoders. HiFi-GAN (Kong, Kim & Bae, 2020) converts mel-spectrograms to waveforms in real time and is deployed widely in text-to-speech.

As a loss term inside other models. The autoencoder in VQGAN (Esser, Rombach & Ommer, 2021) and in latent diffusion is trained with a patch-based adversarial loss alongside reconstruction. Without it the decoder produces blurred output. So the adversarial objective survives inside the very systems that displaced GANs as standalone generators.

Papers#

• Goodfellow, I. et al. (2014). Generative Adversarial Nets. NeurIPS — arxiv.org/abs/1406.2661
• Radford, A., Metz, L. & Chintala, S. (2015). Unsupervised Representation Learning with Deep Convolutional GANs. — arxiv.org/abs/1511.06434
• Salimans, T. et al. (2016). Improved Techniques for Training GANs. NeurIPS — arxiv.org/abs/1606.03498
• Arjovsky, M. & Bottou, L. (2017). Towards Principled Methods for Training GANs. ICLR — arxiv.org/abs/1701.04862
• Arjovsky, M., Chintala, S. & Bottou, L. (2017). Wasserstein GAN. ICML — arxiv.org/abs/1701.07875
• Gulrajani, I. et al. (2017). Improved Training of Wasserstein GANs. NeurIPS — arxiv.org/abs/1704.00028
• Heusel, M. et al. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS — arxiv.org/abs/1706.08500
• Miyato, T. et al. (2018). Spectral Normalization for GANs. ICLR — arxiv.org/abs/1802.05957
• Karras, T. et al. (2020). Analyzing and Improving the Image Quality of StyleGAN. CVPR — arxiv.org/abs/1912.04958
• Dhariwal, P. & Nichol, A. (2021). Diffusion Models Beat GANs on Image Synthesis. NeurIPS — arxiv.org/abs/2105.05233
• Kang, M. et al. (2023). Scaling up GANs for Text-to-Image Synthesis. CVPR — arxiv.org/abs/2303.05511
• Sauer, A. et al. (2023). Adversarial Diffusion Distillation. — arxiv.org/abs/2311.17042

What to learn next#

• Autoencoders — the other route to generating data, and why its output is smoother.
• Diffusion models — the approach that replaced GANs for image generation.
• Loss functions — why a GAN's loss tells you so little.