Optical flow

The brightness constancy equation#

Assume a point's intensity is unchanged as it moves:

$$ I(x, y, t) = I(x + \Delta x,\; y + \Delta y,\; t + \Delta t) $$

A first-order Taylor expansion of the right-hand side gives the optical flow constraint equation:

$$ I_x u + I_y v + I_t = 0 $$

Where $I_x, I_y$ are spatial image gradients, $I_t$ is the temporal gradient, and $(u, v)$ is the flow vector at that pixel.

One equation, two unknowns. The system is underdetermined at every pixel, which is the aperture problem stated algebraically. Only the flow component along the image gradient is observable. Every classical method is a different choice of extra constraint.

The three classical answers#

Lucas and Kanade (1981) assume flow is constant over a window $\Omega$ and solve by least squares:

$$ \begin{bmatrix} u \ v \end{bmatrix} = \left( \sum_{\Omega} \begin{bmatrix} I_x^2 & I_x I_y \ I_x I_y & I_y^2 \end{bmatrix} \right)^{-1} \left( -\sum_{\Omega} \begin{bmatrix} I_x I_t \ I_y I_t \end{bmatrix} \right) $$

That matrix is the structure tensor. Its eigenvalues diagnose the pixel: two large eigenvalues means a corner and a well-posed solve; one large means an edge and the aperture problem; two small means flat texture and no solution. This is exactly why goodFeaturesToTrack, which is the Shi-Tomasi corner detector, is the right companion to Lucas-Kanade.

Horn and Schunck (1981) instead add a global smoothness penalty and minimise:

$$ E = \iint \left[ (I_x u + I_y v + I_t)^2 + \alpha^2 \left( |\nabla u|^2 + |\nabla v|^2 \right) \right] dx\, dy $$

Where $\alpha$ trades data fidelity against smoothness. The result is dense but over-smoothed across motion boundaries, which the modern total-variation family (Zach et al., 2007) fixes with an $L^1$ penalty that permits discontinuities.

Farnebäck (2003) approximates each neighbourhood by a quadratic polynomial and derives displacement from how the polynomial coefficients change. This is what calcOpticalFlowFarneback implements, and it is why the method is comparatively robust to a global illumination scaling: a gain change alters the polynomial coefficients in a way partially absorbed by the estimation.

Coarse-to-fine, and its ceiling#

All three methods linearise, so they are only valid for displacements smaller than roughly the window size. The standard remedy is a Gaussian pyramid: estimate at scale $2^{-L}$, upsample, warp, and refine.

The range this buys is approximately $w \cdot 2^{L}$ pixels for window size $w$. The table in the developer section is that formula made visible.

The ceiling is structural, not a tuning problem. Small fast-moving objects vanish at coarse pyramid levels before their motion can be estimated. Brox and Malik (2011) address it by injecting descriptor matches at full resolution; RAFT sidesteps it by never linearising in the first place.

RAFT and what changed#

Teed and Deng (2020), RAFT: Recurrent All-Pairs Field Transforms, restructured the problem:

1. Encode both frames into features at $1/8$ resolution.
2. Build an all-pairs correlation volume: the inner product of every feature in frame 1 with every feature in frame 2, giving an $H/8 \times W/8 \times H/8 \times W/8$ tensor, pooled into a 4-level pyramid.
3. Initialise flow at zero and run a GRU that repeatedly looks up correlation values at the current flow estimate and emits an update.

Two properties matter. The correlation volume is built once, at full spatial resolution in the feature space, so small fast objects are not destroyed by downsampling. And the iterative updates share weights, so more iterations at test time is a cost-accuracy dial rather than a retraining decision. The torchvision model returning a list of 12 flow fields is this loop made visible.

RAFT won the ECCV 2020 best paper award and reduced Sintel and KITTI errors substantially over the preceding PWC-Net family. Successors worth knowing: GMA (Jiang et al., 2021) adds global motion aggregation for occluded regions; FlowFormer (Huang et al., 2022) replaces the GRU with a transformer over the cost volume; SEA-RAFT (Wang et al., ECCV 2024) simplifies the architecture and adds a mixture-of-Laplace loss, reaching strong Spring and Sintel results at higher throughput.

Metrics and benchmarks#

• EPE (endpoint error): mean Euclidean distance between predicted and ground-truth flow vectors. The default on Sintel.
• Fl-all: percentage of pixels with EPE above 3 px and above 5% of the ground-truth magnitude. The default on KITTI, and more informative than EPE because it is scale-aware.
• Sintel splits results into clean and final passes; final adds motion blur and atmospheric effects and is the harder, more realistic number.

Ground truth for real footage is close to impossible to obtain, so the field trains on synthetic data (FlyingChairs, FlyingThings3D, Sintel) and evaluates zero-shot generalisation to KITTI and Spring. Any claim about flow accuracy on real video is a claim about synthetic-to-real transfer.

Flow in video recognition: the honest position#

Two-stream networks (Simonyan and Zisserman, 2014) fed RGB and stacked flow to separate CNNs and won large gains. I3D inherited the design, and for several years pre-computed flow was standard.

That has largely reversed. Flow is expensive to compute, cannot be trained end to end when pre-computed, and roughly doubles storage. Modern 3D convolutional and transformer models learn motion representations directly from RGB and close most of the gap. Where flow still earns its keep:

• Tracking and video object segmentation, where explicit correspondence is the task rather than a feature.
• Frame interpolation and video compression, where the flow field is the output.
• Small-data regimes, where flow is a strong hand-designed prior that a network would need far more data to learn.

References#

• Lucas and Kanade, An Iterative Image Registration Technique, IJCAI 1981.
• Horn and Schunck, Determining Optical Flow, Artificial Intelligence, 1981.
• Farnebäck, Two-Frame Motion Estimation Based on Polynomial Expansion, SCIA 2003.
• Simonyan and Zisserman, Two-Stream Convolutional Networks, NeurIPS 2014 — arxiv.org/abs/1406.2199
• Teed and Deng, RAFT, ECCV 2020 — arxiv.org/abs/2003.12039
• Jiang et al., Learning to Estimate Hidden Motions with GMA, ICCV 2021 — arxiv.org/abs/2104.02409
• Wang et al., SEA-RAFT, ECCV 2024 — arxiv.org/abs/2405.14793

What to learn next#

• 3D convolutions for video — learning motion instead of measuring it.
• Tracking by detection — where explicit correspondence still wins.
• Video object segmentation — pushing a mask forward using motion.