Classifier-free guidance

Classifier guidance, first#

Dhariwal and Nichol (2021) introduce guidance by adding the gradient of a separately trained classifier $p_\phi(y \mid \mathbf{x}_t)$ to the score:

$$ \hat{\boldsymbol{\epsilon}}(\mathbf{x}t, t, y) = \boldsymbol{\epsilon}\theta(\mathbf{x}_t, t) - s\sqrt{1 - \bar{\alpha}t}\, \nabla{\mathbf{x}t} \log p\phi(y \mid \mathbf{x}_t) $$

$s$ is the guidance scale. This follows from $\nabla \log p(\mathbf{x}_t \mid y) = \nabla \log p(\mathbf{x}_t) + \nabla \log p(y \mid \mathbf{x}_t)$, with $s$ inserted as a free exponent on the likelihood term. Two costs follow. A classifier must be trained on noisy images at every timestep. The label set is fixed at classifier-training time.

Classifier-free guidance#

Ho and Salimans (2022), Classifier-Free Diffusion Guidance, eliminate the classifier. Train a single conditional model $\boldsymbol{\epsilon}_\theta(\mathbf{x}t, t, \mathbf{c})$. The condition $\mathbf{c}$ is replaced with a learned null token $\varnothing$ with probability $p{\text{uncond}}$, typically 0.1 to 0.2. At sampling:

$$ \tilde{\boldsymbol{\epsilon}}_\theta(\mathbf{x}t, t, \mathbf{c}) = \boldsymbol{\epsilon}\theta(\mathbf{x}t, t, \varnothing) + w\left(\boldsymbol{\epsilon}\theta(\mathbf{x}t, t, \mathbf{c}) - \boldsymbol{\epsilon}\theta(\mathbf{x}_t, t, \varnothing)\right) $$

Two conventions for $w$ circulate and confusing them is a common source of bugs. In the form above (used in Stable Diffusion), $w = 1$ is plain conditional sampling and $w = 0$ is unconditional. The original paper writes $(1+w)\boldsymbol{\epsilon}c - w\boldsymbol{\epsilon}\varnothing$, in which $w = 0$ is plain conditional. Always check which one a codebase means.

Why it samples from a sharpened distribution#

The implied score is:

$$ \nabla_{\mathbf{x}_t} \log \tilde{p}(\mathbf{x}t \mid \mathbf{c}) = \nabla{\mathbf{x}_t} \log p(\mathbf{x}t) + w \nabla{\mathbf{x}_t} \log p(\mathbf{c} \mid \mathbf{x}_t) $$

which corresponds to sampling from $p(\mathbf{x}_t) \, p(\mathbf{c} \mid \mathbf{x}_t)^{w}$. Raising the likelihood to a power greater than one concentrates mass where the condition is most strongly satisfied. It is a temperature on the conditional term. The resulting distribution is not the true conditional for any $w \ne 1$.

Karras et al. (2024), Guiding a Diffusion Model with a Bad Version of Itself, sharpen this analysis. Much of what CFG does at high scale is truncation toward high-density regions. It conflates two effects: improving image quality, and reducing variation. They propose autoguidance. It replaces the unconditional model with a smaller or less-trained version of the same conditional model. Quality improves without the same diversity collapse. This is the cleanest current account of what CFG is doing and what it costs.

The measured trade-off#

The finding that reproduces everywhere: FID is U-shaped in $w$, Inception Score and human preference are monotonic. Guidance improves per-sample fidelity and prompt adherence while shrinking coverage, and FID is sensitive to both.

Reporting practice follows from this. A guidance sweep with FID and CLIP-score plotted against each other, the standard Pareto curve, is the honest presentation. A single FID at an unstated guidance scale is not comparable to anything.

Known failure modes and their fixes#

Oversaturation and burnt contrast at high $w$. Caused by the guided prediction leaving the valid range. Lin et al. (2023) prescribe rescaling the guided output, toward the standard deviation of the conditional prediction. Their paper is Common Diffusion Noise Schedules and Sample Steps are Flawed. diffusers exposes the fix as guidance_rescale, and 0.7 is a common value.

Zero terminal SNR. The same paper shows most schedules do not reach pure noise at the final step. The model always sees a residual mean brightness. It therefore cannot produce very dark or very bright images. Fixes are rescale_betas_zero_snr=True with timestep_spacing="trailing", and a model trained with $\mathbf{v}$-prediction.

Doubled compute. Addressed by distillation. Meng et al. (2023), On Distillation of Guided Diffusion Models, train a student to match the guided output in one pass. FLUX.1 dev ships a guidance-distilled model, which is why applying external CFG to it is a mistake.

Interval guidance. Kynkäänniemi et al. (2024) show guidance is harmful at very high and very low noise levels. It is beneficial only in the middle. Restricting CFG to a timestep interval improves FID substantially, at the same prompt adherence.

Papers#

• Dhariwal and Nichol, Diffusion Models Beat GANs on Image Synthesis, 2021 — arxiv.org/abs/2105.05233
• Ho and Salimans, Classifier-Free Diffusion Guidance, 2022 — arxiv.org/abs/2207.12598
• Lin et al., Common Diffusion Noise Schedules and Sample Steps are Flawed, 2023 — arxiv.org/abs/2305.08891
• Meng et al., On Distillation of Guided Diffusion Models, 2023 — arxiv.org/abs/2210.03142
• Kynkäänniemi et al., Applying Guidance in a Limited Interval Improves Sample and Distribution Quality in Diffusion Models, 2024 — arxiv.org/abs/2404.07724
• Karras et al., Guiding a Diffusion Model with a Bad Version of Itself, 2024 — arxiv.org/abs/2406.02507

What to learn next#

• Diffusion samplers and step counts — the other dial that decides what your picture looks like.
• Prompt engineering — writing the condition that guidance amplifies.
• Temperature and sampling — the same obedience-versus-variety trade, in text.