Diffusion samplers and step counts

Sampling as solving a differential equation#

Song et al. (2021) show the forward noising process is a stochastic differential equation. It has an associated deterministic probability flow ODE, with the same marginals at every $t$:

$$ \mathrm{d}\mathbf{x} = \left[\mathbf{f}(\mathbf{x}, t) - \tfrac{1}{2} g(t)^2 \nabla_{\mathbf{x}} \log p_t(\mathbf{x})\right]\mathrm{d}t $$

$\mathbf{f}$ is the drift, $g$ the diffusion coefficient, and $\nabla_{\mathbf{x}} \log p_t(\mathbf{x})$ the score, which the trained network supplies. Generation is integrating this backwards from $t = T$ to $t = 0$. Every named sampler is a numerical scheme for that integral. The number of function evaluations (NFE) is the honest cost axis.

DDIM and the eta parameter#

Song et al. (2021), Denoising Diffusion Implicit Models, generalise DDPM to a family of non-Markovian processes sharing the same training objective:

$$ \mathbf{x}{t-1} = \sqrt{\bar{\alpha}{t-1}} \underbrace{\left(\frac{\mathbf{x}_t - \sqrt{1 - \bar{\alpha}t}\,\boldsymbol{\epsilon}\theta}{\sqrt{\bar{\alpha}t}}\right)}{\text{predicted } \mathbf{x}0} + \underbrace{\sqrt{1 - \bar{\alpha}{t-1} - \sigma_t^2}\, \boldsymbol{\epsilon}\theta}{\text{direction to } \mathbf{x}_t} + \sigma_t \boldsymbol{\epsilon} $$

with

$$ \sigma_t = \eta \sqrt{\frac{1 - \bar{\alpha}_{t-1}}{1 - \bar{\alpha}_t}} \sqrt{1 - \frac{\bar{\alpha}t}{\bar{\alpha}{t-1}}} $$

$\eta = 1$ recovers DDPM ancestral sampling; $\eta = 0$ is fully deterministic and is an Euler discretisation of the probability flow ODE. The key structural property is this. The objective does not depend on the forward process being Markovian. A model trained as DDPM can therefore be sampled as DDIM, with no retraining.

Determinism also gives DDIM an approximate inverse, called DDIM inversion. It maps a real image to the latent that reproduces it. That is the basis of most training-free editing methods.

Higher-order solvers#

Lu et al. (2022), DPM-Solver, exploit the semi-linear structure of the probability flow ODE. Changing variables to $\lambda_t = \log(\alpha_t / \sigma_t)$, the half-log-SNR, the linear part integrates exactly and only a nonlinear term needs approximating:

$$ \mathbf{x}_t = \frac{\alpha_t}{\alpha_s}\mathbf{x}_s - \alpha_t \int_{\lambda_s}^{\lambda_t} e^{-\lambda} \hat{\boldsymbol{\epsilon}}\theta(\hat{\mathbf{x}}\lambda, \lambda) \, \mathrm{d}\lambda $$

Taylor expanding the integrand to order $k$ gives DPM-Solver-$k$. The exact treatment of the linear part buys the order-of-magnitude reduction in NFE. That is why these solvers dominate the field. DPM-Solver++ (2022) adapts this to the guided, $\mathbf{x}_0$-prediction setting where high guidance scales made the original unstable. UniPC (Zhao et al., 2023) adds a corrector step in a unified predictor-corrector framework.

Karras et al. (2022) separate the design into independent choices. Those are preconditioning, noise schedule, time discretisation, and solver. Their paper is Elucidating the Design Space of Diffusion-Based Generative Models, known as EDM. Their contributions that entered every codebase:

• The Karras sigma schedule, $\sigma_i = \left(\sigma_{\max}^{1/\rho} + \frac{i}{N-1}\left(\sigma_{\min}^{1/\rho} - \sigma_{\max}^{1/\rho}\right)\right)^{\rho}$ with $\rho = 7$, which places steps where curvature is highest.
• Heun's second-order method as the default deterministic solver.
• An analysis of stochasticity. Added noise corrects accumulated discretisation and score-estimation error, the effect visible in the developer table above. Too much of it over-smooths.

Few-step generation#

Below roughly ten steps, better solvers stop being enough and the model itself must change.

• Progressive distillation (Salimans and Ho, 2022). A student learns to take one step where the teacher took two, halving NFE per round.
• Consistency models (Song et al., 2023). Train $f_\theta(\mathbf{x}_t, t)$ to map any point on a trajectory directly to its origin. Self-consistency is enforced along the trajectory. One to four steps.
• Latent Consistency Models (Luo et al., 2023). Consistency distillation applied in latent space. An LCM-LoRA converts existing checkpoints.
• Adversarial Diffusion Distillation (Sauer et al., 2023, SDXL-Turbo): adds an adversarial loss to distillation, reaching one to four steps.
• Rectified flow (Liu et al., 2023) and the reflow procedure straighten the ODE trajectories at training time. SD 3 therefore needs fewer steps by construction, not by distillation.

The consistent finding across all of these: quality per step improves, and sample diversity falls. Distilled few-step models are measurably less varied than their teachers. That is the same trade guidance makes, reached from a different direction.

Papers#

• Song et al., Denoising Diffusion Implicit Models, 2021 — arxiv.org/abs/2010.02502
• Song et al., Score-Based Generative Modeling through SDEs, 2021 — arxiv.org/abs/2011.13456
• Lu et al., DPM-Solver, 2022 — arxiv.org/abs/2206.00927
• Lu et al., DPM-Solver++, 2022 — arxiv.org/abs/2211.01095
• Karras et al., Elucidating the Design Space of Diffusion-Based Generative Models, 2022 — arxiv.org/abs/2206.00364
• Zhao et al., UniPC, 2023 — arxiv.org/abs/2302.04867
• Song et al., Consistency Models, 2023 — arxiv.org/abs/2303.01469
• Liu et al., Flow Straight and Fast (Rectified Flow), 2023 — arxiv.org/abs/2209.03003
• Sauer et al., Adversarial Diffusion Distillation, 2023 — arxiv.org/abs/2311.17042

What to learn next#

• ControlNet — steering the same sampling loop with a picture instead of words.
• Classifier-free guidance — the other dial, and why it doubles your compute.
• Diffusion models — the process all of this is solving.