Gaussian splatting

The representation#

Kerbl et al. (SIGGRAPH 2023), 3D Gaussian splatting for real-time radiance field rendering, ACM TOG 42(4). A scene is a set of anisotropic 3D Gaussians

$$ G_i(\mathbf{x}) = \exp!\left(-\tfrac{1}{2}(\mathbf{x} - \boldsymbol{\mu}_i)^\top \Sigma_i^{-1} (\mathbf{x} - \boldsymbol{\mu}_i)\right) $$

Optimising $\Sigma$ directly would require keeping it positive semi-definite, so it is parameterised as

$$ \Sigma = R S S^\top R^\top $$

with $S$ a diagonal scale matrix and $R$ a rotation from a normalised quaternion. Any gradient step then produces a valid covariance by construction. The parameters stored are the scale logs and the quaternion, not $\Sigma$.

Projection#

A 3D Gaussian projects to a 2D Gaussian on the image plane. The projected covariance is, following Zwicker et al. (2001), EWA volume splatting:

$$ \Sigma' = J W \Sigma W^\top J^\top $$

Where $W$ is the world-to-camera rotation and $J$ the Jacobian of the perspective projection at the Gaussian's centre. $J$ is a local affine approximation to a nonlinear map, so the projection is exact only for small Gaussians near the optical axis. Large Gaussians toward the image edges are drawn slightly wrong, and this is a real, known artefact rather than an implementation detail.

The 2D covariance is then dilated slightly, adding a small multiple of the identity, to guarantee a minimum screen-space footprint of roughly one pixel. Without it, sub-pixel Gaussians alias badly.

Rendering#

Pixel colour is exactly the alpha compositing in the code above:

$$ C = \sum_{i \in \mathcal{N}} c_i \alpha_i \prod_{j=1}^{i-1}(1 - \alpha_j), \qquad \alpha_i = o_i \, G'_i(\mathbf{p}) $$

with $\mathcal{N}$ the depth-sorted Gaussians overlapping the pixel, $o_i$ the learned opacity, and $G'_i$ the projected 2D Gaussian. This is the same equation as the NeRF quadrature. The difference is entirely in how the samples along the ray are obtained: queried from a network, or read from sorted explicit primitives.

The rasteriser is what makes it fast:

1. Split the screen into 16x16 tiles. Assign each Gaussian to every tile its 99% confidence ellipse touches.
2. Build (tile, depth) keys and radix sort globally once per frame, giving a per-tile depth order without per-pixel sorting.
3. Each tile is one CUDA block; threads cooperatively load Gaussians into shared memory and blend front to back, terminating a pixel once transmittance falls below a threshold.
4. The backward pass traverses back to front, recovering the required transmittance by division rather than storing per-pixel lists.

The global sort is an approximation: Gaussians are sorted by centre depth, not per-pixel, so two intersecting Gaussians can be blended in the wrong order for some pixels. It is visually acceptable and is the source of some popping artefacts under camera motion.

Adaptive density control#

Initialised from the SfM point cloud, then every 100 iterations:

• Densify Gaussians whose average positional gradient magnitude exceeds a threshold — a signal that the region is not yet well reconstructed. Small ones are cloned and shifted along the gradient; large ones are split into two with scale divided by roughly 1.6.
• Prune Gaussians with opacity below a threshold, and those with excessive screen-space extent.
• Reset opacity toward zero periodically, forcing everything to re-earn its place. This is the main mechanism against floaters near the cameras.

This is the piece that lets the model choose its own capacity, and it has no equivalent in a fixed-size MLP.

Trade-offs against NeRF#

Notable follow-ups#

• Mip-Splatting (Yu et al., CVPR 2024) adds a 3D smoothing filter bounding each Gaussian's maximum frequency by the training sampling rate, plus a 2D Mip filter replacing the dilation. It fixes the erosion and dilation artefacts that appear when rendering at a different resolution or distance from training.
• 2D Gaussian splatting (Huang et al., SIGGRAPH 2024) replaces 3D ellipsoids with oriented planar discs, giving view-consistent geometry and considerably better surface extraction. The right choice when you need a mesh.
• Deformable and 4D variants extend the representation to dynamic scenes by adding a time-conditioned deformation field or time-varying primitives.
• gsplat (Ye et al., 2024, arXiv 2409.06765) is the open reference rasteriser, used by nerfstudio's Splatfacto, with continuing work through 2026 on faster rasterisation paths and broader camera and sensor models including fisheye and lidar.
• Compression. Vector quantisation of SH coefficients, opacity-based pruning and entropy coding routinely cut file sizes by an order of magnitude with small quality loss.

The honest limits#

It is not geometry. Photometric loss constrains what the scene looks like, not what it is. Gaussians drift off surfaces, become large and semi-transparent in ambiguous regions, and adopt shapes that render correctly and mean nothing. Any downstream use assuming a surface — collision, physics, printing, measurement — needs a variant designed for it.

Baked appearance. Lighting is inseparable from albedo. Relighting requires inverse rendering and remains open.

The sorting approximation. Per-Gaussian rather than per-fragment depth ordering causes popping. Exact solutions cost the performance that motivated the method.

Benchmarks are narrow. Mip-NeRF 360, Tanks and Temples, and Deep Blending are small, static, well-lit and densely captured. Sparse views, motion, changing lighting and reflective surfaces are all substantially harder, and headline PSNR figures say little about them.

What to learn next#

• Neural radiance fields — the same rendering equation, solved the other way.
• Structure from motion — the step every splatting capture starts with.
• Diffusion models — the generative side now feeding 3D scene creation.