Latent diffusion

The two-stage formulation#

Rombach et al. (2022), High-Resolution Image Synthesis with Latent Diffusion Models (CVPR 2022), split generation into perceptual compression and semantic generation.

Stage one trains an autoencoder $\mathcal{E}, \mathcal{D}$ with $z = \mathcal{E}(x)$ and $\tilde{x} = \mathcal{D}(z)$. Here $x \in \mathbb{R}^{H \times W \times 3}$ and $z \in \mathbb{R}^{h \times w \times c}$, with $h = H/f$ and $w = W/f$. The objective combines a perceptual loss, a patch-based adversarial loss, and one of two regularisers:

• KL-reg: a small KL penalty toward $\mathcal{N}(0, I)$. The weight is low enough that the model is nearer a deterministic autoencoder than a VAE. This is what Stable Diffusion uses.
• VQ-reg: a vector quantisation layer inside the decoder.

Stage two trains a diffusion model on $z$ with the standard objective:

$$ L_{\text{LDM}} = \mathbb{E}{\mathcal{E}(x), \boldsymbol{\epsilon} \sim \mathcal{N}(0,1), t}\left[\left|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}\theta(z_t, t)\right|_2^2\right] $$

The stages are trained separately. The autoencoder is frozen for stage two, which is what makes it reusable across many diffusion models.

Choosing $f$#

The paper's ablation over $f \in {1, 2, 4, 8, 16, 32}$ produces the central empirical result. Small $f$ leaves the diffusion model doing perceptual compression, which it does inefficiently. Large $f$ throws away information the diffusion model then cannot recover. $f = 4$ and $f = 8$ occupy the sweet spot, with $f = 8$ the deployed choice.

The framing worth remembering: the latent space should retain the semantic variation and discard the perceptual variation. That boundary is empirical, not principled. The whole 4-channel-to-16-channel migration admits it was drawn too aggressively.

Conditioning#

Conditioning enters through cross-attention in the denoising U-Net. A domain-specific encoder $\tau_\theta$ maps the condition $y$ to $\tau_\theta(y) \in \mathbb{R}^{M \times d_\tau}$, and each attention layer computes:

$$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}!\left(\frac{QK^{\top}}{\sqrt{d}}\right) V $$

with $Q = W_Q^{(i)} \varphi_i(z_t)$, $K = W_K^{(i)} \tau_\theta(y)$, $V = W_V^{(i)} \tau_\theta(y)$, where $\varphi_i(z_t)$ is the flattened intermediate feature map. Image features query, condition features supply. The same mechanism carries text, semantic maps and layout. That is why one architecture covered every task in the paper.

The reconstruction ceiling, quantified#

The autoencoder's reconstruction quality is an upper bound. It caps the generative model's fidelity in any metric the decoder affects. Published SD-VAE reconstruction figures on COCO validation sit near 24 to 26 dB PSNR, at $f=8$ with 4 channels. The round trip alone measurably degrades an image before generation begins.

Three consequences that matter in practice:

• FID measured on generated images includes the autoencoder's error. A reported FID for a latent diffusion model is not comparable to a pixel-space model's FID. The autoencoder's reconstruction FID has to be reported too.
• Small high-frequency structure is the systematic failure. Text glyphs, distant faces, thin lines, moiré fabric. These are the observable symptoms of the same cause.
• Adding latent channels is the direct fix and it is not free. Sixteen channels at $f=8$ is a 12x compression instead of 48x, so the denoiser handles four times the data.

Lineage#

Every entry keeps the two-stage latent structure. The backbone changed from convolutional to transformer, and the compression got gentler. The core decision has held since 2022, across images, video and audio. Diffusion runs on a learned compressed representation, not on pixels.

Papers#

• Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, 2022 — arxiv.org/abs/2112.10752
• Esser et al., Taming Transformers for High-Resolution Image Synthesis (VQGAN), 2021 — arxiv.org/abs/2012.09841
• Podell et al., SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, 2023 — arxiv.org/abs/2307.01952
• Esser et al., Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, 2024 — arxiv.org/abs/2403.03206
• Blattmann et al., Stable Video Diffusion, 2023 — arxiv.org/abs/2311.15127

What to learn next#

• Classifier-free guidance — how the prompt actually steers the result.
• Autoencoders — the compression half of this system, from first principles.
• Text to image — the full pipeline these pieces assemble into.