Mean average precision

Setup#

pip install numpy

That is all this needs. The point of writing it out is that mAP stops being a mystery once you have seen the fifty lines it takes.

The whole metric, from scratch#

import numpy as np

# Ground truth: (image_id, box). Boxes are [x1, y1, x2, y2].
GT = [
    (1, [10, 10, 50, 50]),
    (1, [60, 60, 90, 90]),
    (2, [20, 20, 60, 60]),
    (2, [70, 20, 95, 45]),      # nothing will ever detect this one
]

# Predictions: (image_id, box, confidence score).
PRED = [
    (1, [12, 12, 52, 52], 0.95),
    (2, [22, 18, 58, 62], 0.90),
    (1, [58, 58, 88, 88], 0.80),
    (1, [70, 10, 95, 40], 0.70),   # nothing is there: a false positive
    (2, [21, 19, 59, 61], 0.60),   # second box on an already-detected object
    (1, [11, 11, 49, 49], 0.50),   # ditto
]


def iou(a, b):
    ix1, iy1 = max(a[0], b[0]), max(a[1], b[1])
    ix2, iy2 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, ix2 - ix1) * max(0, iy2 - iy1)
    area_a = (a[2] - a[0]) * (a[3] - a[1])
    area_b = (b[2] - b[0]) * (b[3] - b[1])
    return inter / (area_a + area_b - inter)


def match(preds, gts, thr):
    """Greedy matching, highest confidence first. Returns 1 for TP, 0 for FP."""
    order = sorted(range(len(preds)), key=lambda i: -preds[i][2])
    taken = set()                       # each ground-truth box can be used once
    flags = []
    for i in order:
        img, box, _ = preds[i]
        best, best_g = 0.0, None
        for g, (gimg, gbox) in enumerate(gts):
            if gimg != img or g in taken:
                continue
            v = iou(box, gbox)
            if v > best:
                best, best_g = v, g
        if best >= thr:
            taken.add(best_g)
            flags.append(1)
        else:
            flags.append(0)
    return np.array(flags), [preds[i][2] for i in order]


def pr_curve(flags, n_gt):
    tp = np.cumsum(flags)
    fp = np.cumsum(1 - flags)
    recall = tp / n_gt
    precision = tp / (tp + fp)
    return precision, recall


def ap_101(precision, recall):
    """COCO's 101-point interpolated average precision."""
    rec_thrs = np.linspace(0.0, 1.0, 101)
    # for each recall level, the best precision achieved at that recall or beyond
    p_interp = [precision[recall >= r].max() if (recall >= r).any() else 0.0
                for r in rec_thrs]
    return float(np.mean(p_interp))


flags, scores = match(PRED, GT, 0.5)
prec, rec = pr_curve(flags, len(GT))

print("detections sorted by confidence, at IoU 0.50")
print("score  TP?  precision  recall")
for s, f, p, r in zip(scores, flags, prec, rec):
    print(f"{s:.2f}   {f}    {p:9.3f}  {r:6.3f}")

print(f"\nAP@0.50 = {ap_101(prec, rec):.4f}")

print("\nAP at each COCO IoU threshold:")
aps = []
for thr in np.arange(0.5, 1.0, 0.05):
    f, _ = match(PRED, GT, thr)
    p, r = pr_curve(f, len(GT))
    a = ap_101(p, r)
    aps.append(a)
    print(f"  IoU {thr:.2f} -> AP {a:.4f}")
print(f"\nmAP@[.50:.95] = {np.mean(aps):.4f}")

# Now move the false positive to the top of the ranking and nothing else.
LOUD = [(i, b, 0.99 if s == 0.70 else s) for i, b, s in PRED]
f, _ = match(LOUD, GT, 0.5)
p, r = pr_curve(f, len(GT))
print(f"\nsame boxes, the false positive now scores 0.99: AP@0.50 = {ap_101(p, r):.4f}")

detections sorted by confidence, at IoU 0.50
score  TP?  precision  recall
0.95   1        1.000   0.250
0.90   1        1.000   0.500
0.80   1        1.000   0.750
0.70   0        0.750   0.750
0.60   0        0.600   0.750
0.50   0        0.500   0.750

AP@0.50 = 0.7525

AP at each COCO IoU threshold:
  IoU 0.50 -> AP 0.7525
  IoU 0.55 -> AP 0.7525
  IoU 0.60 -> AP 0.7525
  IoU 0.65 -> AP 0.7525
  IoU 0.70 -> AP 0.7525
  IoU 0.75 -> AP 0.7525
  IoU 0.80 -> AP 0.5050
  IoU 0.85 -> AP 0.1683
  IoU 0.90 -> AP 0.1683
  IoU 0.95 -> AP 0.0000

mAP@[.50:.95] = 0.5356

same boxes, the false positive now scores 0.99: AP@0.50 = 0.5644

Reading that output properly#

The ceiling is set by what you missed. One of the four ground-truth boxes is never detected. Recall stops at 0.750 and stays there. Since 76 of the 101 recall points sit at or below 0.75, and precision is a perfect 1.000 at each of them, AP lands at 76/101 = 0.7525. Missing a quarter of the objects caps you near three quarters. No confidence tuning recovers it.

Low-confidence false positives were free. Three of the six predictions were wrong. They still cost nothing, because every real object had already been found by the time they appeared. This is the single most misread property of AP. Lowering your confidence threshold to dump more boxes into the output can raise AP even while making the model useless to a human reading its output.

Move one false positive to the top and AP drops by 0.19. Same boxes. Same count of errors. Only the ranking changed, from 0.7525 to 0.5644. AP is a measure of ordering, not of counts.

The IoU sweep is where sloppy boxes get caught. AP is flat at 0.7525 up to IoU 0.75, then falls off a cliff. Our boxes are two pixels out. At the strict end that stops being close enough, and the model that looked fine scores zero. mAP@[.50:.95] of 0.5356 is the honest summary.

The parts that trip people up#

taken = set() is the reason a duplicate box is a false positive. Each ground-truth object may be claimed once. Remove that line and a detector spraying ten boxes on one car scores ten true positives.

Matching walks the predictions in confidence order, not in file order. A greedy match by confidence is what COCO does, and it is why a confident-but-slightly-worse box can steal the ground truth from a less confident, better-fitting one.

precision[recall >= r].max() is the interpolation step. Raw precision-recall curves are saw-toothed; each new true positive bumps precision up. COCO smooths this by taking, at each recall level, the best precision available at that recall or higher. The 101 points come straight from cocoeval.py, where recThrs = np.linspace(0, 1, 101).

Real COCO evaluation adds two things this code leaves out. It caps detections per image at 100, and it reports AP separately for small, medium and large objects, split at 32 by 32 and 96 by 96 pixels. Small-object AP is usually where the pain is.

Use the real thing in production#

pip install pycocotools

from pycocotools.coco import COCO
from pycocotools.cocoeval import COCOeval

coco_gt = COCO("instances_val.json")
coco_dt = coco_gt.loadRes("detections.json")
e = COCOeval(coco_gt, coco_dt, iouType="bbox")
e.evaluate(); e.accumulate(); e.summarize()

No output block for this one. It depends entirely on your annotation file and your model, and an invented table would teach you to expect numbers that will not appear.

Two practical notes. detections.json must be a flat list of {"image_id", "category_id", "bbox", "score"}, with bbox in [x, y, width, height] — not the corner format used above. And category_id must match the ground-truth file's ids, not your model's class indices. Silently mismatched ids are the most common cause of a suspiciously low mAP.

Common mistakes#

Running non-maximum suppression too aggressively before evaluating. NMS removes overlapping boxes. Set the threshold too tight and you delete real detections of two people standing close together, which shows up as poor recall you will blame on the model.

Comparing an mAP@0.5 number to a mAP@[.50:.95] number. They are different metrics. Pascal VOC style mAP@0.5 numbers look far higher. Papers quote both, and press releases quote whichever is bigger.

Filtering low-confidence detections before submitting. Since low-ranked false positives are nearly free, cutting them off can only lose you recall. Submit everything and let the metric sort the ranking out. Filter for humans, not for the metric.

Averaging per-image mAP. mAP is computed once over the whole dataset, pooling every detection. Averaging a per-image score gives a different, wrong number, and images containing no objects of a class break it entirely.

Try it yourself#

Add a fifth ground-truth box in image 1 and a matching high-confidence prediction. Predict what happens to AP before you run it. Then shrink every predicted box by four pixels on each side and watch the IoU sweep collapse while AP@0.50 barely moves.

What to learn next#

• Detection error analysis — splitting a low mAP into the mistakes that caused it.
• Object detection — the task this metric scores.
• Model evaluation — precision and recall in the simpler classification setting.