Heatmaps for keypoints

The representation#

For keypoint $k$ at true location $(x, y)$ in image coordinates, and network stride $s$, the target heatmap on the $H \times W$ grid is

$$ G_k(u, v) = \exp\left(-\frac{(u - x/s)^2 + (v - y/s)^2}{2\sigma^2}\right) $$

$u, v$ index heatmap cells, $\sigma$ is the blob width in heatmap units. Training minimises mean squared error between predicted and target heatmaps, typically masked to visible joints.

The formulation is from Tompson et al. (NeurIPS 2014) and the stacked hourglass architecture of Newell et al. (ECCV 2016), and it displaced direct coordinate regression (DeepPose, Toshev and Szegedy, CVPR 2014) almost completely.

The quantisation error, quantified#

Standard decoding takes $\hat{p} = \arg\max_{u,v} \hat{G}(u,v)$ and maps back as $s \cdot \hat{p}$. The error from rounding alone, for a target uniformly distributed within a cell, has expectation

$$ \mathbb{E}\big[\lVert \epsilon \rVert\big] = s \cdot \mathbb{E}\big[\sqrt{\delta_x^2 + \delta_y^2}\big], \qquad \delta \sim \mathcal{U}[-\tfrac12, \tfrac12]^2 $$

which evaluates to roughly $0.3826\,s$. The measured $0.3833$ heatmap pixels above matches this to three decimal places, confirming the experiment is measuring the quantisation term and nothing else.

The empirical quarter offset shifts the estimate by $0.25$ toward the larger of the two neighbours:

$$ \hat{p} \leftarrow \hat{p} + 0.25 \cdot \operatorname{sign}\big(\nabla \hat{G}(\hat{p})\big) $$

It halves the error and is unprincipled — the shift magnitude is a constant, unrelated to the actual blob shape.

DARK#

Zhang et al. (CVPR 2020), Distribution-Aware Coordinate Representation for Human Pose Estimation (arxiv.org/abs/1910.06278), treat the heatmap as a sampled Gaussian and solve for its continuous peak.

Take $\mathcal{P} = \log \hat{G}$. For a Gaussian, $\mathcal{P}$ is a quadratic, so a second-order Taylor expansion at the discrete maximum $m$ is exact:

$$ \mathcal{P}(p) = \mathcal{P}(m) + \mathcal{D}(m)^\top (p - m) + \tfrac{1}{2}(p - m)^\top \mathcal{D}''(m)(p - m) $$

Setting the derivative to zero gives

$$ \hat{p} = m - \big(\mathcal{D}''(m)\big)^{-1} \mathcal{D}(m) $$

$\mathcal{D}(m)$ is the first derivative and $\mathcal{D}''(m)$ the Hessian of the log-heatmap at $m$, both estimated by finite differences. The code above uses the diagonal approximation; the full paper uses the complete $2 \times 2$ Hessian.

DARK also corrects the encoding side: the standard practice of centring the Gaussian on a rounded integer coordinate injects a bias before training even starts. Generating unbiased targets at the true sub-pixel location removes it. The paper reports both halves as necessary, and it is model-agnostic — reported gains of roughly 1 to 2 AP on COCO across several backbones.

UDP (Huang et al., CVPR 2020), The Devil is in the Details, addresses a related bias: the conventional size = (w - 1) versus w ambiguity in the image-to-heatmap affine transform introduces a systematic flip-test misalignment. Both papers are about arithmetic in the data pipeline, not architecture, and both produce gains comparable to a substantial model change.

Resolution and architecture#

Quantisation error scales linearly with stride, which is a direct architectural constraint.

• Hourglass (Newell et al., 2016) — symmetric downsample/upsample with skip connections, repeated.
• Simple Baseline (Xiao et al., ECCV 2018) — ResNet plus three transposed convolutions; a strong and deliberately plain baseline.
• HRNet (Sun et al., CVPR 2019, arxiv.org/abs/1902.09212) — maintains a high-resolution branch throughout, fusing repeatedly across resolutions rather than recovering resolution at the end. HRNet-W48 reaches 76.3 AP on COCO val at 384x288 input.
• ViTPose (Xu et al., NeurIPS 2022, arxiv.org/abs/2204.12484) — a plain ViT backbone with a lightweight decoder, reaching 80.9 AP on COCO test-dev at the largest scale.

Alternatives to argmax decoding#

Integral / soft-argmax (Sun et al., ECCV 2018) replaces the discrete maximum with a spatial expectation over the softmax-normalised heatmap:

$$ \hat{p} = \sum_{u,v} (u, v) \cdot \frac{\exp(\hat{G}(u,v)/T)}{\sum_{u',v'} \exp(\hat{G}(u',v')/T)} $$

This is differentiable end to end, removing the need for an explicit heatmap loss. The drawback is a bias induced by background activation across the whole map, and a documented sensitivity to the temperature $T$.

SimCC (Li et al., ECCV 2022) reformulates the problem as two 1D classifications, over $x$ and over $y$ separately, at sub-pixel bin resolution. This removes 2D quantisation entirely and is what RTMPose (Jiang et al., 2023, arxiv.org/abs/2303.07399) builds on to reach 75.8 AP on COCO at over 90 FPS on a desktop CPU.

Regression, revisited. RLE (Li et al., ICCV 2021), Residual Log-likelihood Estimation, makes direct coordinate regression competitive by learning the output distribution with a normalising flow instead of assuming a fixed Laplacian. Regression is cheaper at inference and is the right choice for tight latency budgets.

Evaluation#

COCO uses OKS — object keypoint similarity:

$$ \text{OKS} = \frac{\sum_i \exp\left(-d_i^2 / (2 s^2 \kappa_i^2)\right) \cdot \mathbb{1}[v_i > 0]}{\sum_i \mathbb{1}[v_i > 0]} $$

$d_i$ is the distance between predicted and true keypoint $i$, $s$ the square root of the person's segment area, $\kappa_i$ a per-keypoint constant reflecting annotator disagreement, and $v_i$ the visibility flag. AP is then averaged over OKS thresholds from 0.50 to 0.95.

The $\kappa_i$ values matter and are frequently ignored: eyes have small $\kappa$ and hips large, so a hip error of the same magnitude costs far less OKS than an eye error. MPII instead uses PCKh, thresholding at a fraction of head segment length.

Papers#

• Toshev and Szegedy, DeepPose, CVPR 2014 — arxiv.org/abs/1312.4659
• Tompson et al., Joint Training of a CNN and a Graphical Model for Human Pose Estimation, NeurIPS 2014
• Newell et al., Stacked Hourglass Networks, ECCV 2016 — arxiv.org/abs/1603.06937
• Xiao et al., Simple Baselines for Human Pose Estimation and Tracking, ECCV 2018 — arxiv.org/abs/1804.06208
• Sun et al., Deep High-Resolution Representation Learning (HRNet), CVPR 2019 — arxiv.org/abs/1902.09212
• Zhang et al., Distribution-Aware Coordinate Representation (DARK), CVPR 2020 — arxiv.org/abs/1910.06278
• Huang et al., The Devil is in the Details (UDP), CVPR 2020 — arxiv.org/abs/1911.07524
• Li et al., Human Pose Regression with Residual Log-likelihood Estimation, ICCV 2021 — arxiv.org/abs/2107.11291
• Li et al., SimCC: a Simple Coordinate Classification Perspective, ECCV 2022 — arxiv.org/abs/2107.03332

What to learn next#

• CNN — the grid-preserving architecture that makes heatmaps natural.
• Image segmentation — the other task that outputs a picture rather than a label.
• Object detection — where keypoint models get their person boxes from.