Segmentation metrics

Definitions#

For a class $c$ with prediction set $P$ and ground-truth set $G$ over pixels:

$$ \mathrm{IoU}_c = \frac{|P \cap G|}{|P \cup G|} = \frac{\mathrm{TP}}{\mathrm{TP} + \mathrm{FP} + \mathrm{FN}} $$

$$ \mathrm{Dice}_c = \frac{2|P \cap G|}{|P| + |G|} = \frac{2\,\mathrm{TP}}{2\,\mathrm{TP} + \mathrm{FP} + \mathrm{FN}} $$

Dice is the $F_1$ score computed over pixels. The two are related by a strictly increasing bijection on $[0,1]$:

$$ \mathrm{Dice} = \frac{2\,\mathrm{IoU}}{1 + \mathrm{IoU}}, \qquad \mathrm{IoU} = \frac{\mathrm{Dice}}{2 - \mathrm{Dice}} $$

So they induce identical rankings over models. They differ in aggregation: averaging IoU over images and averaging Dice over images give different orderings, because the mean of a nonlinear function is not that function of the mean.

Aggregation choices, and why they matter more than the metric#

$$ \mathrm{mIoU} = \frac{1}{|C|}\sum_{c} \mathrm{IoU}_c, \qquad \mathrm{fwIoU} = \sum_c \frac{|G_c|}{\sum_{c'} |G_{c'}|}\,\mathrm{IoU}_c $$

A third option, sometimes called dataset-level or aggregate IoU, pools TP, FP and FN across all images before dividing. Cityscapes reports this. Per-image averaging is a fourth. On long-tailed data these four can differ by ten points or more on the same predictions, which is why cross-paper comparison requires checking the evaluation script, not the metric name.

Panoptic quality#

For panoptic segmentation, Kirillov et al. (2019) define

$$ \mathrm{PQ} = \underbrace{\frac{\sum_{(p,g) \in \mathrm{TP}} \mathrm{IoU}(p,g)}{|\mathrm{TP}|}}{\text{segmentation quality}} \times \underbrace{\frac{|\mathrm{TP}|}{|\mathrm{TP}| + \frac{1}{2}|\mathrm{FP}| + \frac{1}{2}|\mathrm{FN}|}}{\text{recognition quality}} $$

Segments match when IoU exceeds 0.5, which makes the matching unique. The factorisation is the useful part: it separates did you find the object from did you outline it well, which mIoU cannot.

Boundary-aware metrics#

Boundary IoU (Cheng et al., 2021) computes IoU restricted to a band of width $d$ around each mask's contour:

$$ \mathrm{BIoU}_d = \frac{|(G_d \cap G) \cap (P_d \cap P)|}{|(G_d \cap G) \cup (P_d \cap P)|} $$

where $X_d$ denotes the set of pixels within distance $d$ of $\partial X$. Unlike Trimap IoU it is sensitive to boundary errors on large objects, and unlike the earlier F-measure of Perazzi et al. (2016) it does not saturate.

The related Hausdorff distance and its 95th-percentile variant HD95 are standard in medical imaging:

$$ d_H(P, G) = \max\left{ \sup_{p \in P} \inf_{g \in G} |p - g|,\; \sup_{g \in G} \inf_{p \in P} |p - g| \right} $$

HD95 exists because the plain supremum is decided by a single outlier pixel and is unusable in practice.

Losses versus metrics#

Soft Dice loss (Milletari et al., 2016, V-Net) relaxes the sets to probabilities:

$$ \mathcal{L}_{\text{Dice}} = 1 - \frac{2\sum_i p_i g_i + \epsilon}{\sum_i p_i + \sum_i g_i + \epsilon} $$

It is differentiable and handles foreground-background imbalance far better than pixel-wise cross-entropy. Three cautions carry into evaluation. It is unstable when the ground truth for a class is empty in a batch, and the $\epsilon$ chosen then determines the loss value. Its gradients are non-local, so a single confident wrong pixel moves the whole map. And Bertels et al. (2019) show that optimising soft Dice yields systematically miscalibrated probabilities, which matters if anything downstream consumes the confidence rather than the argmax.

Tversky loss (Salehi et al., 2017) generalises Dice with asymmetric FP and FN weights $\alpha, \beta$, and is the standard tool when a missed lesion costs far more than a false alarm.

The annotation ceiling#

Inter-annotator Dice is the honest upper bound. Published figures for organ segmentation commonly sit in the 0.85 to 0.95 range depending on structure and modality, and for lesions considerably lower. Joskowicz et al. (2019) and the various MICCAI challenge reports document this directly. Any model result within the inter-observer band is a measurement of the annotation protocol as much as of the model, and the correct response is to report the variability alongside the score rather than to claim superiority.

Papers#

• Milletari et al., V-Net, 3DV 2016 — arxiv.org/abs/1606.04797
• Salehi et al., Tversky Loss Function, MLMI 2017 — arxiv.org/abs/1706.05721
• Kirillov et al., Panoptic Segmentation, CVPR 2019 — arxiv.org/abs/1801.00868
• Bertels et al., Optimizing the Dice Score and Jaccard Index for Medical Image Segmentation, MICCAI 2019 — arxiv.org/abs/1911.01685
• Cheng et al., Boundary IoU, CVPR 2021 — arxiv.org/abs/2103.16562
• Perazzi et al., A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation, CVPR 2016

What to learn next#

• Image segmentation — the models these numbers score.
• Keypoint metrics: OKS and PCK — the same problem for points instead of regions.
• Mean average precision — how instance segmentation borrows the detection metric.