Intersection over union

Definition#

For boxes $A$ and $B$ as subsets of the image plane:

$$ \mathrm{IoU}(A, B) = \frac{|A \cap B|}{|A \cup B|} = \frac{|A \cap B|}{|A| + |B| - |A \cap B|} $$

Where $|\cdot|$ is area. This is the Jaccard index. It is scale-invariant and lies in $[0, 1]$. It is also a proper metric: $1 - \mathrm{IoU}$ satisfies the triangle inequality (Kosub, 2019).

Scale invariance is the property that makes it usable across a benchmark containing both birds and buses. No pixel-distance measure has it.

Why it fails as a loss#

Two failures, both structural.

Zero gradient on disjoint boxes. If $A \cap B = \emptyset$ then $\mathrm{IoU} = 0$ for every disjoint configuration, so $\nabla \mathrm{IoU} = 0$ everywhere in that region. Early in training most predictions are disjoint from most targets.

Plateaus at equal overlap. Many distinct configurations share an IoU value, so the loss surface has large flat regions.

Generalized IoU#

Rezatofighi et al. (2019) add a term for the smallest axis-aligned box $C$ enclosing both:

$$ \mathrm{GIoU}(A, B) = \mathrm{IoU}(A, B) - \frac{|C \setminus (A \cup B)|}{|C|} $$

The second term is the fraction of the enclosing box that neither input occupies. Range is $(-1, 1]$. It is $-1$ in the limit as the boxes separate to infinity, and equals IoU when the union fills $C$.

Its known degeneracy: when $A \subseteq B$, $C = B$, so $|C \setminus (A \cup B)| = 0$ and $\mathrm{GIoU} = \mathrm{IoU}$. Enclosure carries no extra signal. The output block above is exactly this case.

Distance and Complete IoU#

Zheng et al. (2020) replace the enclosure term with a normalised centre distance:

$$ \mathcal{L}_{\mathrm{DIoU}} = 1 - \mathrm{IoU} + \frac{\rho^2(\mathbf{b}, \mathbf{b}^{gt})}{c^2} $$

Where $\rho(\cdot,\cdot)$ is Euclidean distance between box centres and $c$ is the diagonal length of the smallest enclosing box $C$. The ratio is bounded in $[0, 1)$, so the loss stays comparable across scales.

CIoU adds an aspect-ratio term:

$$ \mathcal{L}{\mathrm{CIoU}} = \mathcal{L}{\mathrm{DIoU}} + \alpha v, \qquad v = \frac{4}{\pi^2}\left(\arctan\frac{w^{gt}}{h^{gt}} - \arctan\frac{w}{h}\right)^2, \qquad \alpha = \frac{v}{(1 - \mathrm{IoU}) + v} $$

Where $w, h$ are the predicted width and height and $w^{gt}, h^{gt}$ the target's. $v$ measures aspect-ratio disagreement. $\alpha$ is a trade-off weight. It switches the aspect term off when overlap is poor, so the model fixes position before shape. $\alpha$ is treated as a constant during backpropagation.

DIoU also gives DIoU-NMS, which subtracts the centre-distance term from the suppression criterion. Two distinct objects that overlap heavily but have well-separated centres survive, which helps in crowded scenes.

Later variants#

• EIoU (Zhang et al., 2022) replaces the aspect-ratio term with separate width and height penalties. Their argument is that $v$ is zero whenever the ratio matches, even if both edges are wrong.
• SIoU (Gevorgyan, 2022) adds an angle term steering the prediction onto the nearest axis before closing the distance.
• alpha-IoU (He et al., 2021) raises IoU to a power, $\mathcal{L} = 1 - \mathrm{IoU}^{\alpha}$, sharpening the gradient on high-quality boxes.
• Wise-IoU (Tong et al., 2023) attenuates the loss on low-quality examples, which matters when labels are noisy.

Gains between these are small and dataset-dependent. Treat the choice as a hyperparameter. Be sceptical of any paper claiming a large mAP jump from a loss term alone.

Cost#

box_iou on $N$ predictions and $M$ targets is $O(NM)$ time and memory. With $N = 100{,}000$ anchors and $M = 50$ targets that is five million pairs, which is fine. With $N = M = 100{,}000$ it is $10^{10}$ entries, which is not. Chunk the computation, or restrict candidate pairs by centre distance first.

Papers#

• Rezatofighi et al., Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression, CVPR 2019 — arxiv.org/abs/1902.09630
• Zheng et al., Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression, AAAI 2020 — arxiv.org/abs/1911.08287
• Zhang et al., Focal and Efficient IOU Loss for Accurate Bounding Box Regression, 2022 — arxiv.org/abs/2101.08158
• He et al., Alpha-IoU: A Family of Power Intersection over Union Losses, NeurIPS 2021 — arxiv.org/abs/2110.13675

What to learn next#

• Anchor boxes — the first big consumer of the IoU matrix.
• Non-maximum suppression — using IoU to delete duplicate detections.
• Model evaluation — where precision, recall and thresholds come from.