StyleGAN and latent space

Architecture#

Karras et al. (2019), A Style-Based Generator Architecture for GANs (StyleGAN), restructure the generator into three parts.

Mapping network. An eight-layer MLP $f: \mathcal{Z} \to \mathcal{W}$, both $\mathbb{R}^{512}$. Its purpose is disentanglement. $\mathcal{Z}$ is constrained to a fixed Gaussian, so the generator must warp it to match the data. A learned $\mathcal{W}$ need not be Gaussian and can be less curved.

Synthesis network. Starts from a learned constant $4 \times 4 \times 512$ tensor. Style vectors enter at every resolution through learned affine maps $\mathbf{y} = A(\mathbf{w})$.

Noise inputs. Per-pixel Gaussian noise added after each convolution, scaled by learned per-channel weights. It supplies stochastic detail (hair strands, freckles) that the latent should not have to encode.

The StyleGAN1 style operation is AdaIN:

$$ \mathrm{AdaIN}(\mathbf{x}i, \mathbf{y}) = \mathbf{y}{s,i} \frac{\mathbf{x}_i - \mu(\mathbf{x}_i)}{\sigma(\mathbf{x}i)} + \mathbf{y}{b,i} $$

$\mathbf{x}i$ is feature map $i$, with $\mu$ and $\sigma$ its spatial mean and standard deviation. $\mathbf{y}{s,i}$ and $\mathbf{y}_{b,i}$ are the style scale and bias for that channel.

Weight demodulation#

Karras et al. (2020) published Analyzing and Improving the Image Quality of StyleGAN, known as StyleGAN2. They trace the droplet artefacts to AdaIN destroying information carried in feature magnitudes. Their replacement modulates the weights:

$$ w'_{ijk} = s_i \cdot w_{ijk}, \qquad w''{ijk} = \frac{w'{ijk}}{\sqrt{\sum_{i,k} {w'_{ijk}}^2 + \epsilon}} $$

$i$ indexes input channels, $j$ output channels, and $k$ spatial positions. $s_i$ is the style scale for input channel $i$. The normalisation assumes unit-variance i.i.d. inputs, which makes it a statistical assumption rather than an exact operation on the actual activations. That is what removes the artefact. The network can no longer smuggle a signal into feature magnitudes for AdaIN to amplify.

The same paper adds path length regularisation. A fixed-size step in $\mathcal{W}$ should produce a fixed-size change in the image:

$$ \mathbb{E}{\mathbf{w}, \mathbf{y} \sim \mathcal{N}(0,I)}\left(|\mathbf{J}{\mathbf{w}}^{\top} \mathbf{y}|_2 - a\right)^2 $$

$\mathbf{J}_{\mathbf{w}} = \partial g(\mathbf{w}) / \partial \mathbf{w}$ is the generator Jacobian and $a$ a running average of the observed lengths. This makes the latent space better conditioned and measurably easier to invert.

Aliasing and StyleGAN3#

Karras et al. (2021), Alias-Free Generative Adversarial Networks (StyleGAN3), identify "texture sticking". Fine detail in StyleGAN2 adheres to pixel coordinates rather than to the object. A translating face therefore drags a static texture. The cause is aliasing from pointwise nonlinearities and naive upsampling, which violate the sampling theorem.

Their fix treats every feature map as a continuous signal. It band-limits before each nonlinearity, and uses ideal low-pass filters for resampling. The result is exact translation and rotation equivariance, at higher compute cost. StyleGAN3 is the correct choice for video and animation, and StyleGAN2 usually remains better on still-image FID.

Latent space geometry#

Perceptual path length measures how much the image changes for a small latent step:

$$ \mathrm{PPL} = \mathbb{E}\left[\frac{1}{\epsilon^2} \, d\big(G(\mathrm{lerp}(z_1, z_2; t)),\, G(\mathrm{lerp}(z_1, z_2; t + \epsilon))\big)\right] $$

$d$ is a perceptual distance (LPIPS), $t \sim U(0,1)$ and $\epsilon = 10^{-4}$. Lower PPL indicates a smoother, better-behaved latent space. It is consistently lower in $\mathcal{W}$ than in $\mathcal{Z}$, which is the quantitative justification for the mapping network.

Linear separability measures disentanglement. Fit a linear SVM per attribute in each space. Then compute the conditional entropy of the true label, given the classifier's side of the hyperplane. Again $\mathcal{W}$ wins.

Editing and inversion#

InterFaceGAN (Shen et al., 2020) treats each binary attribute as a hyperplane in $\mathcal{W}$. Its unit normal is $\mathbf{n}$. Editing is $\mathbf{w}' = \mathbf{w} + \alpha \mathbf{n}$. For two entangled attributes with normals $\mathbf{n}_1, \mathbf{n}_2$, conditional manipulation projects:

$$ \mathbf{n}_1^{\perp} = \frac{\mathbf{n}_1 - (\mathbf{n}_1^{\top} \mathbf{n}_2)\mathbf{n}_2}{|\mathbf{n}_1 - (\mathbf{n}_1^{\top} \mathbf{n}_2)\mathbf{n}_2|} $$

GANSpace (Härkönen et al., 2020) finds directions unsupervised, by PCA on sampled $\mathbf{w}$ vectors. SeFa (Shen and Zhou, 2021) factorises the first affine layer's weights directly, needing no sampling at all. StyleCLIP (Patashnik et al., 2021) drives the search with a CLIP text encoder. The attribute is named rather than labelled.

Inversion, mapping a real photograph to a latent, is the practical bottleneck. Optimising directly in $\mathcal{W}$ reconstructs poorly. Optimising in $\mathcal{W}+$ reconstructs well, but lands off the learned manifold, so subsequent edits degrade. $\mathcal{W}+$ is a separate $\mathbf{w}$ per layer, $18 \times 512$ for a $1024^2$ generator. Tov et al. (2021), Designing an Encoder for StyleGAN Image Manipulation, formalise this as a distortion-editability trade-off. They train an encoder that targets a chosen point on it.

Honest limits#

The attribute directions inherit every correlation in the training set. On FFHQ, "smile" correlates with age and gender presentation, and edits along a naive smile direction shift both. Projection removes only the correlations you thought to name.

That makes latent editing a fairness problem rather than a purely technical one. Karras et al. (2020), Training Generative Adversarial Networks with Limited Data, is also relevant here. Adaptive discriminator augmentation makes StyleGAN trainable on a few thousand images. That puts these systems within reach of small, unaudited datasets.

Papers#

• Karras et al., A Style-Based Generator Architecture for GANs, 2019 — arxiv.org/abs/1812.04948
• Karras et al., Analyzing and Improving the Image Quality of StyleGAN, 2020 — arxiv.org/abs/1912.04958
• Karras et al., Training Generative Adversarial Networks with Limited Data, 2020 — arxiv.org/abs/2006.06676
• Karras et al., Alias-Free Generative Adversarial Networks, 2021 — arxiv.org/abs/2106.12423
• Shen et al., InterFaceGAN, 2020 — arxiv.org/abs/2005.09635
• Härkönen et al., GANSpace, 2020 — arxiv.org/abs/2004.02546
• Tov et al., Designing an Encoder for StyleGAN Image Manipulation, 2021 — arxiv.org/abs/2102.02766
• Patashnik et al., StyleCLIP, 2021 — arxiv.org/abs/2103.17249

What to learn next#

• Image-to-image translation — generating a picture from another picture instead of from noise.
• Autoencoders — the other way to build a latent space, and how it differs.
• Embeddings — the same "directions mean things" idea, in text.