Face detection models

The single-shot formulation#

Face detection is class-agnostic object detection with one class and a strong scale prior. Given an image $I$, a network produces for each prior $a_i = (a_x, a_y, a_w, a_h)$ a tuple $(p_i, \delta_i, \ell_i)$: a face probability, four box deltas, and ten landmark offsets.

Decoding follows the SSD parameterisation (Liu et al., 2016):

$$ \hat{x} = a_x + \sigma_c\, \delta_x\, a_w, \qquad \hat{w} = a_w \exp(\sigma_s\, \delta_w) $$

$\sigma_c$ and $\sigma_s$ are the centre and size variances, conventionally $0.1$ and $0.2$. They rescale the regression targets to roughly unit variance; they are a normalisation constant, not a learned quantity.

Training uses a multi-task loss. RetinaFace (Deng et al., CVPR 2020, arxiv.org/abs/1905.00641) writes it as:

$$ L = L_{\text{cls}} + \lambda_1 L_{\text{box}} + \lambda_2 L_{\text{pts}} + \lambda_3 L_{\text{pixel}} $$

$L_{\text{cls}}$ is softmax or focal loss over face/background, $L_{\text{box}}$ a smooth-$L_1$ on the four deltas, $L_{\text{pts}}$ a smooth-$L_1$ on the five landmarks, and $L_{\text{pixel}}$ a self-supervised 3D mesh decoder branch. The paper's ablation shows the landmark branch improves the box AP as well — a supervision signal, not only an output.

Positive assignment and the small-face problem#

A prior is positive when its IoU with a ground-truth box exceeds $0.5$ (typically $0.35$ for faces, which are small). WIDER FACE (Yang et al., CVPR 2016) splits its validation set into Easy, Medium and Hard, where Hard is dominated by faces under 20 pixels.

The core difficulty is statistical. Tiny faces match very few priors, so they contribute almost nothing to the loss. Two responses dominate:

• Sample redistribution. SCRFD (Guo et al., 2021, arxiv.org/abs/2105.04714) reallocates positive samples toward shallow, high-resolution stages, and searches how to redistribute compute across backbone, neck and head under a fixed FLOP budget.
• Denser priors at fine strides. Stride-8 or even stride-4 levels, at the cost of a much larger prior count.

Cost and accuracy, roughly#

Reported latencies are hardware- and batch-dependent; treat the ordering as reliable and the absolute numbers as indicative. YuNet is documented in Wu, Peng et al., YuNet: A Tiny Millisecond-level Face Detector, Machine Intelligence Research, 2023.

BlazeFace (Bazarevsky et al., 2019, arxiv.org/abs/1907.05047) takes a different route again: a near-frontal, single-face-dominant model designed for mobile GPU, with a tie-resolution strategy replacing NMS to avoid jitter between video frames. It is the detector inside the MediaPipe face pipeline.

NMS and its alternatives#

Greedy NMS is $O(n^2)$ in the worst case and non-differentiable. Two variants matter in practice:

• Soft-NMS (Bodla et al., ICCV 2017) decays overlapping scores rather than deleting them, which helps in crowds where two real faces genuinely overlap.
• Anchor-free heads (FCOS-style, and YOLO-family face variants such as YOLO5Face, Qi et al., 2021) predict distances to box edges from each location, cutting prior tuning entirely. They still need NMS.

Evaluation caveats#

WIDER FACE AP is computed at IoU $0.5$ and rewards recall of very small faces heavily. Two consequences follow.

First, a detector tuned for WIDER Hard may return many low-confidence boxes that are useless downstream. Second, AP says nothing about landmark quality, which is what actually limits recognition accuracy in a full pipeline. Measure the metric your system depends on, which is usually alignment error, not box AP.

Where detection stops#

A detector is scope-limiting infrastructure. Detection alone supports counting, autofocus, cropping and redaction, and produces no biometric identifier. Adding an embedding model changes the legal character of the system entirely, bringing biometric-data obligations that detection alone does not trigger — see responsible deployment.

Papers#

• Yang et al., WIDER FACE: A Face Detection Benchmark, CVPR 2016
• Zhang et al., Joint Face Detection and Alignment using Multi-task Cascaded CNNs, IEEE SPL 2016 — arxiv.org/abs/1604.02878
• Deng et al., RetinaFace: Single-stage Dense Face Localisation in the Wild, CVPR 2020 — arxiv.org/abs/1905.00641
• Guo et al., Sample and Computation Redistribution for Efficient Face Detection, 2021 — arxiv.org/abs/2105.04714
• Bazarevsky et al., BlazeFace, 2019 — arxiv.org/abs/1907.05047

What to learn next#

• Object detection — the general machinery this is a special case of.
• YOLO — anchors, NMS and single-shot heads in their most-used form.
• OpenCV — the library that ships the detector you will reach for first.