Camera models and intrinsics

The model#

A point $\mathbf{X}_w \in \mathbb{R}^3$ in world coordinates maps to a pixel $\mathbf{x}$ by

$$ s \begin{bmatrix} u \ v \ 1 \end{bmatrix} = K \begin{bmatrix} R & \mathbf{t} \end{bmatrix} \begin{bmatrix} X_w \ Y_w \ Z_w \ 1 \end{bmatrix} $$

Where $s$ is an arbitrary positive scalar (the depth in camera coordinates), $R \in SO(3)$ and $\mathbf{t} \in \mathbb{R}^3$ are the extrinsics, and

$$ K = \begin{bmatrix} f_x & \gamma & c_x \ 0 & f_y & c_y \ 0 & 0 & 1 \end{bmatrix} $$

$f_x, f_y$ are focal lengths in pixel units, $c_x, c_y$ the principal point, and $\gamma$ the skew, which accounts for non-perpendicular sensor axes and is zero for every camera manufactured in the last several decades. Most calibration routines fix it at zero, and OpenCV's does.

$K$ has 4 degrees of freedom, $[R|\mathbf{t}]$ has 6, and $P = K[R|\mathbf{t}]$ has 11 (a 3x4 matrix defined up to scale).

Why this is a projective map#

The equality holds only up to scale, which is what makes projection a map from $\mathbb{P}^3$ to $\mathbb{P}^2$ rather than a linear map between vector spaces. Three consequences follow, and they explain most of what is strange about multi-view geometry.

• The fibre of a pixel is a ray. $P^{-1}$ does not exist; the preimage of a pixel is the one-parameter family ${\mathbf{C} + \lambda \, \mathbf{d}}$ where $\mathbf{C}$ is the camera centre and $\mathbf{d} = R^{\top} K^{-1} \mathbf{x}$.
• Straight lines stay straight, ratios do not. Projective transformations preserve incidence and cross-ratio; they do not preserve length, angle, or the midpoint of a segment.
• Points at infinity are ordinary. A direction $[\mathbf{d}^\top, 0]^\top$ projects to a finite pixel — the vanishing point. This is not a degenerate case but a useful one, and it is how horizon-based calibration works.

Normalised coordinates and distortion#

Define normalised image coordinates $\tilde{\mathbf{x}} = (X/Z, Y/Z)$, so that $\mathbf{x} = K [\tilde{\mathbf{x}}^\top, 1]^\top$ in the distortion-free case. The Brown-Conrady model applies distortion in normalised coordinates, before $K$:

$$ \tilde{x}_d = \tilde{x}\,\frac{1 + k_1 r^2 + k_2 r^4 + k_3 r^6}{1 + k_4 r^2 + k_5 r^4 + k_6 r^6} + 2p_1 \tilde{x}\tilde{y} + p_2(r^2 + 2\tilde{x}^2) $$

with $r^2 = \tilde{x}^2 + \tilde{y}^2$, and symmetrically for $\tilde{y}_d$. The rational denominator terms $k_4, k_5, k_6$ are OpenCV's extension for wide-angle lenses and are off by default.

The order matters and is a frequent bug: distortion is applied to normalised coordinates, not to pixels. Undistorting requires an iterative inverse, since the polynomial is not analytically invertible — cv2.undistortPoints runs a fixed-point iteration.

For fields of view beyond roughly 120 degrees the polynomial fits poorly and the correct model is the equidistant or Kannala-Brandt fisheye model, $r_d = \theta(1 + k_1\theta^2 + k_2\theta^4 + \dots)$ with $\theta$ the incidence angle. Fitting a fisheye with the pinhole-plus-polynomial model produces a plausible reprojection error and badly wrong geometry at the periphery.

Degrees of freedom and what is recoverable#

From a single image of an unknown scene, $K$ is not recoverable. From a single image with known structure — a plane, a calibration target, three orthogonal vanishing points — it is.

The image of the absolute conic $\omega = (K K^\top)^{-1}$ is the object that makes this precise. It is a fixed conic under any camera rotation, so anything that constrains $\omega$ constrains $K$. Two orthogonal vanishing points $\mathbf{v}_1, \mathbf{v}_2$ give one linear constraint $\mathbf{v}_1^\top \omega \, \mathbf{v}_2 = 0$; three mutually orthogonal directions give three, enough for $K$ with zero skew and unit aspect ratio. This is the theory behind chessboard calibration and behind single-image calibration from architectural scenes.

Hartley and Zisserman (2004), Multiple View Geometry in Computer Vision, 2nd edition, is the standard reference for all of this, and chapter 6 covers the camera model in full.

Calibrated versus uncalibrated#

The distinction shapes the whole field.

• With $K$ known, image points can be converted to rays and the geometry becomes metric up to a global scale. The essential matrix applies, and pose decomposes into a rotation and a unit translation.
• With $K$ unknown, only projective reconstruction is possible. The fundamental matrix applies, and the reconstruction is correct only up to an arbitrary 3D projective transformation — parallel lines need not be parallel, angles are meaningless. Upgrading to metric requires additional constraints, a process called auto-calibration.

Modern feed-forward reconstruction models such as DUSt3R and VGGT blur this line by predicting point maps directly and recovering intrinsics as a by-product, learned from data rather than derived from constraints. The classical distinction still governs what information is present in the images, which is why it remains worth knowing.

What to learn next#

• Calibrating a camera with a chessboard — measuring these four numbers for a real camera.
• Stereo depth from two cameras — the direct way to get the lost depth back.
• How images are stored — what a pixel coordinate actually indexes.