Monocular depth estimation

Why it is ill-posed, precisely#

The projection $\pi(\mathbf{X}) = K\mathbf{X}/Z$ satisfies $\pi(s\mathbf{X}) = \pi(\mathbf{X})$ for any $s > 0$. Scene and scale are therefore unidentifiable from a single view: the likelihood is flat along the scale direction.

Monocular depth estimation is consequently not inference but prior-dominated regression. The model learns $p(Z \mid I)$ under the empirical distribution of scenes it was trained on, and its output is the conditional mean of that prior, not a measurement. Everything about its failure modes follows from that sentence.

The scale-and-shift-invariant loss#

Ranftl et al. (TPAMI 2020), Towards robust monocular depth estimation: mixing datasets for zero-shot cross-dataset transfer (MiDaS), made large-scale training possible by removing the incompatibility between datasets. Different sources give depth in metres, in disparity, or up to an unknown scale, so a common target is needed.

Work in disparity space $d = 1/Z$ and align prediction to ground truth per image before computing the loss:

$$ \mathcal{L} = \frac{1}{M}\sum_{i} \rho\bigl(s\,\hat{d}_i + t - d_i^{}\bigr), \qquad (s, t) = \arg\min_{s,t} \sum_i \bigl(s\hat{d}_i + t - d^{}_i\bigr)^2 $$

The alignment $(s, t)$ has a closed form, so the loss remains cheap. Because the loss cannot see scale or shift, the network is never trained to produce them — which is exactly why the code above had to fit a and b.

Disparity space rather than depth space is deliberate: it compresses the far field, matching both the $1/Z$ error behaviour of stereo supervision and the fact that relative error matters more than absolute error at distance.

Architectures#

MiDaS (2019-2022) established the recipe: a strong pretrained encoder, a dense prediction decoder, and mixed multi-dataset training with the affine-invariant loss.

DPT (Ranftl et al., ICCV 2021), Vision transformers for dense prediction, replaced the convolutional encoder with a ViT and reassembled tokens into multi-scale feature maps. The constant global receptive field at every stage is what improved long-range consistency, which convolutional decoders struggle with.

Depth Anything (Yang et al., CVPR 2024) scaled the data instead of the architecture: 1.5M labelled images plus 62M unlabelled ones, pseudo-labelled by a teacher, with strong perturbations applied to the student and an auxiliary loss keeping features aligned to DINOv2 semantics. V2 (NeurIPS 2024) switched the labelled portion to synthetic data for cleaner boundaries, then distilled through a large teacher into pseudo-labelled real images.

Depth Anything 3 (ByteDance Seed, arXiv 2511.10647, released November 2025) generalises the problem to any number of input views, with or without known poses, predicting spatially consistent geometry plus camera parameters. It ships any-view models from 0.08B to 1.15B parameters, plus metric and monocular specialisations. Licensing differs by size — Apache 2.0 for the smaller models, CC BY-NC 4.0 for the larger ones — which matters for commercial use.

Metric depth#

To output metres, a model must break the scale ambiguity with extra information. Two routes:

• Condition on intrinsics. Metric3D (Yin et al., ICCV 2023) and Metric3Dv2 canonicalise the input to a reference camera using the known focal length, so apparent size becomes informative. Accuracy then depends on the intrinsics being right.
• Fine-tune per domain. Train the affine-invariant backbone, then fine-tune a metric head on a single dataset such as NYUv2 or KITTI. Accurate in-domain, and the scale prior does not transfer.

Depth Anything 3's metric variant documents a conversion of the form metric_depth = focal * output / c for a model-specific constant, again making the focal length the carrier of scale.

Evaluation, and how it misleads#

Standard metrics on NYUv2 and KITTI: AbsRel $\frac{1}{N}\sum |Z_i - Z^_i|/Z^_i$, RMSE, and threshold accuracy $\delta_k$, the fraction of pixels with $\max(Z/Z^, Z^/Z) < 1.25^k$.

Three cautions that change how published numbers should be read.

• Median scaling. Most zero-shot evaluations align the prediction to ground truth by median ratio, or by full least-squares affine fit, before scoring. Those numbers therefore describe relative accuracy and say nothing about metric accuracy. The alignment method used is frequently reported only in the appendix, and results are not comparable across different choices.
• Sparse ground truth. KITTI's lidar covers the lower part of the frame and misses sky, glass and thin structures. Errors in the uncovered regions are unmeasured.
• Benchmark saturation and leakage. NYUv2 and KITTI are small and old, and modern models train on tens of millions of images from web sources. Independent evaluation on your own domain is not optional.

Where the failures come from#

The prior-dominated framing predicts them all.

• Mirrors and glass are read as the reflected or transmitted geometry, because that is what the pixel statistics look like.
• Printed images of scenes — posters, screens, painted murals — get depth appropriate to the depicted scene.
• Unusual scale breaks it: a doll's house, a scale model, an extreme close-up.
• Absolute scale drifts across frames, since each image is scaled independently.

None of these are fixed by more data. They are consequences of estimating a scene from a prior over scenes, and the fix is a second view or a second sensor — which is what structure from motion and stereo provide.

What to learn next#

• Structure from motion — recovering real geometry from a moving camera.
• Stereo depth from two cameras — measurement rather than inference.
• Vision transformers — the backbone these models are built on.