Bounding box formats

The representation, formally#

A box is a point in a four-dimensional space, but the parameterisation you choose changes the loss surface. Let a box be $b = (x_1, y_1, x_2, y_2)$ with $x_1 < x_2$ and $y_1 < y_2$.

The centre-size form is a smooth bijection on the interior of the valid region:

$$ c_x = \frac{x_1 + x_2}{2}, \quad c_y = \frac{y_1 + y_2}{2}, \quad w = x_2 - x_1, \quad h = y_2 - y_1 $$

The constraint $x_1 < x_2$ becomes $w > 0$. This matters because a network predicting $w$ directly can emit a negative number, whereas a network predicting $\log w$ cannot. Every anchor-based detector since Girshick et al. (2014) therefore regresses in log-space:

$$ t_x = \frac{c_x - c_x^a}{w^a}, \quad t_y = \frac{c_y - c_y^a}{h^a}, \quad t_w = \log \frac{w}{w^a}, \quad t_h = \log \frac{h}{h^a} $$

Where the superscript $a$ denotes the anchor. Dividing the translation terms by the anchor size makes the targets scale-invariant. A 10-pixel error on a 400-pixel truck and the same error on a 40-pixel bird stop being equally bad.

Why the parameterisation changes what the model learns#

Regressing $(x_1, y_1, x_2, y_2)$ with an $L_1$ loss treats the four coordinates as independent. They are not. Two of them jointly determine width. An error in $x_1$ is a location error or a size error, depending on what $x_2$ did.

This is a large part of why IoU-family losses replaced coordinate losses; see intersection over union. Rezatofighi et al. (2019) make the point directly. Minimising an $L_n$ coordinate loss is not equivalent to maximising IoU. Two pairs of boxes with the same $L_2$ error can have very different IoU.

Discretisation and the half-pixel question#

A pixel is an area, not a point. Where the centre of the top-left pixel sits decides where a box edge falls. Libraries disagree about it.

• COCO uses 0-indexed continuous coordinates; box area is $w \cdot h$.
• Pascal VOC XML is 1-indexed and treats coordinates as inclusive pixel indices, so the width is $x_2 - x_1 + 1$.
• torchvision uses 0-indexed continuous coordinates with width $x_2 - x_1$.

The $+1$ convention was standard in early detection code and survives in some evaluation scripts. On a 500-pixel object the difference is a fifth of one percent of the area. On an 8-pixel object it is around a quarter of the area. That can move the IoU across a match threshold. The same half-pixel reasoning drives aligned=True in roi_align; see RoI pooling and RoI align.

Rotated and beyond-rectangle representations#

The axis-aligned rectangle is a modelling assumption, and a poor one for aerial imagery, text and shelf products.

• Oriented boxes add an angle, $(c_x, c_y, w, h, \theta)$. The angle is periodic, so a naive $L_1$ loss has a discontinuity at the wrap-around. Fixes include the Gaussian-Wasserstein-distance loss of Yang et al. (2021) and representing the box by its corner set.
• Quadrilaterals store eight numbers and avoid the periodicity problem, at the cost of an ordering ambiguity among the corners.
• Distance-to-edge, $(l, t, r, b)$ from a point, is what anchor-free detectors regress. It has no negative-size failure mode when passed through a non-negative activation. See anchor-free detection.
• Masks discard the rectangle entirely. See image segmentation.

Papers#

• Girshick et al., Rich feature hierarchies for accurate object detection and semantic segmentation (R-CNN), 2014 — arxiv.org/abs/1311.2524
• Rezatofighi et al., Generalized Intersection over Union, 2019 — arxiv.org/abs/1902.09630
• Lin et al., Microsoft COCO: Common Objects in Context, 2014 — arxiv.org/abs/1405.0312
• Yang et al., Rethinking Rotated Object Detection with Gaussian Wasserstein Distance Loss, 2021 — arxiv.org/abs/2101.11952

What to learn next#

• Intersection over union — the number that decides whether two boxes are the same box.
• Object detection — the wider picture these four numbers sit inside.
• How images are stored — why pixel coordinates start where they do.