Keypoint metrics: OKS and PCK

Object keypoint similarity#

For a ground-truth person with keypoints $g_i$, visibility flags $v_i$ and area $s$, and predicted keypoints $p_i$:

$$ \mathrm{OKS} = \frac{\sum_i \exp!\left(-\dfrac{d_i^2}{2 s^2 \kappa_i^2}\right)\delta(v_i > 0)}{\sum_i \delta(v_i > 0)} $$

with $d_i = \lVert p_i - g_i \rVert_2$ the Euclidean error for joint $i$, $s$ the object area in pixels squared, and $\kappa_i = 2\sigma_i$ the per-joint falloff. $\delta(\cdot)$ is one when the condition holds and zero otherwise.

The $\sigma_i$ were estimated by COCO from the standard deviation of redundant human annotations of the same images, normalised by object scale. Reported values, in the canonical joint order:

$$ \sigma = 0.1 \cdot [\,0.26, 0.25, 0.25, 0.35, 0.35, 0.79, 0.79, 0.72, 0.72, 0.62, 0.62, 1.07, 1.07, 0.87, 0.87, 0.89, 0.89\,] $$

The ordering is informative on its own. Facial landmarks are three to four times tighter than hips. Any model or loss weighting joints uniformly is fighting the metric.

Substituting $d_i = 2\sigma_i s^{1/2}$ gives a per-joint score of $e^{-1/2} \approx 0.607$, which is the sense in which $2\sigma_i$ is "the tolerance".

PCK and its variants#

$$ \mathrm{PCK}@\alpha = \frac{1}{K}\sum_{i=1}^{K} \delta!\left(\lVert p_i - g_i\rVert_2 \le \alpha \cdot L\right) $$

$L$ is a normalising length. The literature uses at least four:

PCK is a thresholded step function, so it is not differentiable, discards all information about how wrong a miss is, and saturates. Yang and Ramanan (2013) introduced PCP; MPII's switch to PCKh was motivated by torso length being unstable under foreshortening.

Face alignment's NME has its own trap: inter-ocular normalisation collapses for profile faces, which is why Sagonas et al. (2016) and later work report the area under the cumulative error curve rather than a single NME.

Keypoint AP#

COCO applies the detection AP machinery with OKS replacing IoU, sweeping $\mathrm{OKS} \in {0.50, 0.55, \dots, 0.95}$ and reporting $\mathrm{AP}$, $\mathrm{AP}^{50}$, $\mathrm{AP}^{75}$, $\mathrm{AP}^{M}$ and $\mathrm{AP}^{L}$. maxDets is $[20]$ and there is no small-object bucket, since keypoints are not annotated on small people.

A subtlety with real consequences: OKS depends on the ground-truth area, so a detection cannot be scored without being matched first, and the matching uses the score being computed. cocoeval.computeOks resolves this by evaluating all pairs and letting the greedy assignment pick, which means an over-confident prediction can consume a ground truth that a better prediction would have matched.

Top-down versus bottom-up, and what the metric rewards#

Top-down methods (Mask R-CNN keypoints, HRNet, ViTPose) detect people, crop, and regress joints in the crop. Accuracy is high because the crop normalises scale, which is exactly the quantity OKS divides by. Cost grows linearly with the number of people.

Bottom-up methods (OpenPose, Associative Embedding, HigherHRNet) detect all joints then group them. Constant cost in the number of people, weaker on small people, and grouping failures produce the left-right and person-mixing errors that OKS penalises heavily.

Sun et al. (2019), HRNet, is the architectural turning point: maintaining high-resolution representations throughout, rather than recovering resolution by upsampling, directly attacks the localisation precision that the strict end of the OKS sweep measures.

Known problems with the metric#

Sigma values are dataset-specific and rarely re-estimated. They were measured on COCO images with COCO annotators. Applying them to infants, to animals, or to overhead camera views is an unexamined assumption. Cao et al. (2019) and the AP-10K animal-pose work both had to derive their own.

Area normalisation conflates scale with difficulty. A large but heavily occluded person gets a generous tolerance because they are large. Occlusion, not size, is what drives error.

OKS has no notion of anatomical plausibility. A prediction with the elbow beyond the wrist can score well if the individual distances are small. Bone-length and joint-angle consistency checks are a separate, and worthwhile, evaluation.

Aggregate scores conceal per-joint collapse. Reporting per-joint AP is cheap and is what actually tells you whether the model has learned wrists. Most papers include the table; most readers skip it.

Papers#

• Lin et al., Microsoft COCO, ECCV 2014 — arxiv.org/abs/1405.0312
• Andriluka et al., 2D Human Pose Estimation: New Benchmark and State of the Art Analysis, CVPR 2014 (MPII, PCKh)
• Yang and Ramanan, Articulated Human Detection with Flexible Mixtures of Parts, TPAMI 2013
• Cao et al., OpenPose, TPAMI 2019 — arxiv.org/abs/1812.08008
• Sun et al., Deep High-Resolution Representation Learning for Human Pose Estimation, CVPR 2019 — arxiv.org/abs/1902.09212
• Ruggero Ronchi and Perona, Benchmarking and Error Diagnosis in Multi-Instance Pose Estimation, ICCV 2017 — arxiv.org/abs/1707.05388

What to learn next#

• Tracking metrics: MOTA and HOTA — scoring things that move across frames.
• Mean average precision — the ranked-list machinery that sits on top of OKS.
• Convolutional neural networks — the backbone that produces the heatmaps these joints are read from.