Neural radiance fields

The formulation#

Mildenhall et al. (ECCV 2020), NeRF: representing scenes as neural radiance fields for view synthesis. A scene is a continuous function

$$ F_\Theta : (\mathbf{x}, \mathbf{d}) \in \mathbb{R}^3 \times \mathbb{S}^2 \;\mapsto\; (\sigma, \mathbf{c}) \in \mathbb{R}^+ \times \mathbb{R}^3 $$

Where $\mathbf{x}$ is a 3D position, $\mathbf{d}$ a unit viewing direction, $\sigma$ the volume density, and $\mathbf{c}$ the emitted radiance.

The expected colour of a ray $\mathbf{r}(t) = \mathbf{o} + t\mathbf{d}$ between near and far bounds is the classical volume rendering integral (Kajiya and Von Herzen, 1984):

$$ C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t), \mathbf{d})\, dt, \qquad T(t) = \exp!\left(-\int_{t_n}^{t} \sigma(\mathbf{r}(s))\, ds\right) $$

$T(t)$ is the transmittance: the probability that the ray travels from $t_n$ to $t$ without being absorbed.

The quadrature used in practice, and implemented in the code above, is:

$$ \hat{C}(\mathbf{r}) = \sum_{i=1}^{N} T_i \bigl(1 - e^{-\sigma_i \delta_i}\bigr) \mathbf{c}_i, \qquad T_i = \exp!\left(-\sum_{j<i}\sigma_j \delta_j\right) $$

with $\delta_i = t_{i+1} - t_i$. This is exactly alpha compositing with $\alpha_i = 1 - e^{-\sigma_i \delta_i}$, and it is differentiable in every term — which is the whole reason the scheme is trainable.

The weights $w_i = T_i \alpha_i$ form a distribution over "where along the ray the ray terminated", normalised only when the ray is fully opaque. Expected depth is $\sum_i w_i t_i$, and the weight entropy is a useful measure of how confident the model is about a surface.

Positional encoding#

Tancik et al. (NeurIPS 2020), Fourier features let networks learn high frequency functions in low dimensional domains, gives the theory the empirical result above demonstrates.

An MLP with ReLU activations has a neural tangent kernel that, on low-dimensional inputs, decays rapidly with frequency. The network is therefore heavily biased toward low-frequency functions — spectral bias — and fitting fine detail requires impractically long training.

Mapping the input through

$$ \gamma(p) = \bigl[\sin(2^0\pi p), \cos(2^0\pi p), \dots, \sin(2^{L-1}\pi p), \cos(2^{L-1}\pi p)\bigr] $$

makes the composed kernel stationary and widens its spectrum, so high frequencies become learnable. Tancik et al. show that a random Gaussian Fourier mapping with a tuned bandwidth generalises this, and that the bandwidth — not the number of features — is the parameter that matters.

The 8-band regression in the output is the predicted over-bandwidth failure: frequencies above the signal's own content add variance without signal.

NeRF uses $L = 10$ for position and $L = 4$ for direction.

Hierarchical sampling#

Uniform sampling wastes almost every sample in empty space. NeRF trains two networks: a coarse one sampled uniformly, then a fine one sampled from the piecewise-constant PDF given by the coarse network's weights $w_i$, via inverse transform sampling.

This is importance sampling of the rendering integral, and it typically improves quality more than doubling the sample count would. Its descendants — proposal networks in Mip-NeRF 360 and nerfacto — do the same job with a small dedicated density network.

The efficiency lineage#

The 122-million-evaluation figure explains the whole research programme.

• Instant-NGP (Müller et al., SIGGRAPH 2022) replaces the frequency encoding with a multi-resolution hash grid of learnable feature vectors. Most of the capacity moves from network weights into the grid, so the MLP shrinks to two small layers. Training drops from days to seconds, with a fused CUDA implementation.
• Plenoxels (Fridovich-Keil et al., CVPR 2022) removes the network entirely: a sparse voxel grid of spherical-harmonic coefficients, optimised directly. It establishes that the neural network was never the essential ingredient — differentiable volume rendering was.
• TensoRF factorises the volume into low-rank tensor components, reducing memory sharply.

Quality, and the 360 problem#

Mip-NeRF (Barron et al., ICCV 2021) replaces point samples with integrated positional encoding over a conical frustum, giving anti-aliasing and multi-scale consistency. Mip-NeRF 360 (CVPR 2022) adds a space contraction for unbounded scenes plus a distortion regulariser that suppresses the floating artefacts typical of outdoor captures. Zip-NeRF (ICCV 2023) combines that with hash grids.

Unbounded outdoor scenes remain the hard case, and the failure mode is characteristic: correct-looking training views plus "floaters", small clouds of density that satisfy the training views and appear from nowhere in new ones. They are an under-constrained-region symptom, not a bug.

Evaluation#

PSNR, SSIM and LPIPS on held-out views, typically on Blender synthetic objects, LLFF forward-facing scenes, and Mip-NeRF 360 outdoor scenes.

Two cautions. PSNR is dominated by low frequencies and rewards blur, so a method can gain PSNR while looking worse; LPIPS (Zhang et al., CVPR 2018) tracks human judgement better and should be reported alongside. And held-out views in these datasets are interpolations between training views — genuine extrapolation away from the capture trajectory is much harder, and rarely measured.

Where NeRF stands after gaussian splatting#

3D Gaussian splatting (2023) overtook NeRF on the metric that mattered commercially: real-time rendering. It is covered in the next lesson.

NeRF-style implicit fields remain preferable in several places, and it is worth being precise about which:

• Surface extraction. Density fields convert to meshes via marching cubes; SDF-based variants (VolSDF, NeuS, Neuralangelo) give clean geometry that point-based representations struggle with.
• Compactness. A hash-grid NeRF is often smaller on disk than the millions of Gaussians for the same scene.
• Continuity. A continuous field can be queried anywhere and regularised analytically, which matters for physics, inverse rendering and relighting.

The deeper point is that both share the same rendering equation. Gaussian splatting replaces "query a network along a ray" with "sort and blend primitives", and the alpha-compositing arithmetic in the code above is unchanged.

What to learn next#

• Gaussian splatting — the same rendering maths, made real-time.
• Structure from motion — where the camera poses come from.
• Diffusion models — the other family now generating 3D scenes.