Video Understanding and Tracking
Tracking by detection
Run a detector on every frame independently, then solve a small matching puzzle per frame to decide which new box is which old object.
- 16 min read
- 3 reading levels
- Updated
Read these first
On this page 8
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Tracking by detection means finding objects fresh in every frame. Then you work out which new box is which old object.
Picture a school photograph taken every second in a busy playground. Each photo tells you where the children are. No photo tells you which child is which.
To follow one child you compare consecutive photos and reason. This child stands almost where that one stood a second ago. So they are the same child.
That reasoning step is the whole subject. The detector does the seeing. You do the matching.
Why it exists
Detectors already work well. You met one in object detection and another in YOLO.
They have one gap. A detector treats every frame as a fresh picture, so it hands you boxes with no names. Frame ten gives you three boxes. Frame eleven gives you three boxes. Nothing says which is which.
Almost every useful question needs the names. How many separate people entered the shop. How long did this car wait at the signal. Did that person walk in a circle.
So the design splits in two. Let the detector do what it is good at, and add a small matching step on top.
How it works
frame 10 frame 11
┌──────────────┐ ┌──────────────┐
│ [A] [B] │ │ [?] [?] │
└──────────────┘ └──────────────┘
known: track 1, track 2 new: two boxes, no names
matching step
│
which new box overlaps which old one?
│
▼
┌──────────────┐
│ [1] [2] │ names carried forward
└──────────────┘The usual measure of sameness is overlap — how much of the same area two boxes cover. In tracking this is called IoU, meaning intersection over union.
Two boxes that sit almost on top of each other score close to one. Two boxes far apart score zero. Between frames a fraction of a second apart, the same object barely moves, so overlap works remarkably well.
The part that is easy to get wrong
The first idea most people have is greedy matching. Take the best-overlapping pair, lock it in, then take the next best.
Greedy fails whenever two objects are close together. It locks in a pair that looks good alone, then forces a bad pairing on whoever is left.
The fix is to score the whole arrangement rather than one pair at a time. Choose the set of pairings with the best total, even if no individual pairing is the single best available. There is a classic algorithm for this. The code below shows greedy getting it wrong on two people.
What happens when things go missing
Detectors blink. An object is behind a pole for a moment, or the detector's confidence dips, and the box vanishes.
Delete the track immediately and the object gets a new name when it comes back. That is called an ID switch, and it is the main error these systems make.
So tracks are kept alive for a few frames after their box disappears. They are only reported after appearing several times in a row. Both rules exist because detectors are unreliable and you should not trust a single frame.
Where you have already seen this
- Shops counting how many people came in today.
- Traffic cameras counting vehicles by type.
- Sports broadcasts putting a name tag above a player.
- Warehouse robots keeping out of a moving person's way.
What is honestly hard here
Everything hard about tracking happens when objects touch.
Two people walk past each other and their boxes overlap. Overlap alone can no longer tell them apart. The tracker guesses, and if it guesses wrong the names are swapped for the rest of the video.
Getting this right needs more than box positions. It needs a memory of what each object looks like, which is the subject of a later lesson.
Remember this
- Detect independently per frame, then match boxes to existing tracks.
- Match by overlap, and score the whole arrangement rather than one pair at a time.
- Keep tracks alive across a few missing frames, or you will collect ID switches.
What to learn next
- SORT, DeepSORT and ByteTrack — the three trackers everyone actually runs.
- Object detection — the component doing the seeing.
- YOLO — the detector most tracking pipelines are built on.
Developer — Code and libraries.
Setup
pip install numpy scipyRun against numpy 1.26.4 and scipy 1.14.1. No detector needed: we work directly on boxes so the association logic is visible on its own.
Greedy matching, and exactly how it fails
import numpy as np
from scipy.optimize import linear_sum_assignment
def iou(a, b):
"""Boxes are (x1, y1, x2, y2). Returns overlap over union, between 0 and 1."""
x1, y1 = max(a[0], b[0]), max(a[1], b[1])
x2, y2 = min(a[2], b[2]), min(a[3], b[3])
inter = max(0., x2 - x1) * max(0., y2 - y1)
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
return inter / (area_a + area_b - inter)
# Two people walking right. Track 2 is in front and moving slightly faster.
tracks = {1: (10, 10, 30, 50), 2: (19, 10, 39, 50)}
dets = {"A": (15, 10, 35, 50), "B": (26, 10, 46, 50)}
truth = {1: "A", 2: "B"} # detection A really belongs to track 1
names_t, names_d = list(tracks), list(dets)
M = np.array([[iou(tracks[t], dets[d]) for d in names_d] for t in names_t])
print("IoU matrix A B")
for i, t in enumerate(names_t):
print(f"track {t} " + " ".join(f"{v:.3f}" for v in M[i]))
GATE = 0.1
# Greedy: repeatedly take the single best remaining pair.
used_t, used_d, greedy = set(), set(), {}
for ti, di in sorted(np.ndindex(M.shape), key=lambda p: -M[p]):
if ti in used_t or di in used_d or M[ti, di] < GATE:
continue
greedy[names_t[ti]] = names_d[di]
used_t.add(ti)
used_d.add(di)
# Hungarian: choose the set of pairs with the best TOTAL, not the best single pair.
rows, cols = linear_sum_assignment(-M) # negate, because it minimises
hung = {names_t[r]: names_d[c] for r, c in zip(rows, cols) if M[r, c] >= GATE}
print("\nground truth :", truth)
print("greedy :", greedy, " switches:", sum(greedy.get(k) != v for k, v in truth.items()))
print("Hungarian :", hung, " switches:", sum(hung.get(k) != v for k, v in truth.items()))
print(f"\ntotal IoU greedy {sum(iou(tracks[k], dets[v]) for k, v in greedy.items()):.3f}"
f" Hungarian {sum(iou(tracks[k], dets[v]) for k, v in hung.items()):.3f}")IoU matrix A B
track 1 0.600 0.111
track 2 0.667 0.481
ground truth : {1: 'A', 2: 'B'}
greedy : {2: 'A', 1: 'B'} switches: 2
Hungarian : {1: 'A', 2: 'B'} switches: 0
total IoU greedy 0.778 Hungarian 1.081Trace the failure by hand, it is worth it
The largest single entry in the matrix is 0.667, at track 2 against detection A. Greedy takes it first.
Track 1 now has only detection B left, scoring 0.111. Above the gate, so it is accepted. Both people have swapped names. Two ID switches, from one frame.
The Hungarian algorithm maximises the total instead. Track 1 to A is 0.600, track 2 to B is 0.481, totalling 1.081. That beats greedy's 0.778, and it happens to be the truth.
This is not a contrived case. It is what happens every time two objects of similar size are close and moving at slightly different speeds — which describes a crowded pavement, a queue, or a group of players. scipy.optimize.linear_sum_assignment solves it in a line and there is no reason to write greedy matching.
The Hungarian algorithm minimises, so pass the negated similarity. Cost matrices in tracker code are usually 1 - IoU for the same reason.
A whole tracker, association only
Matching is one frame. A tracker also has to decide when a track is born, when it is confirmed, and when it dies.
import numpy as np
from scipy.optimize import linear_sum_assignment
def iou(a, b):
x1, y1 = max(a[0], b[0]), max(a[1], b[1])
x2, y2 = min(a[2], b[2]), min(a[3], b[3])
inter = max(0., x2 - x1) * max(0., y2 - y1)
return inter / ((a[2]-a[0])*(a[3]-a[1]) + (b[2]-b[0])*(b[3]-b[1]) - inter)
class Tracker:
"""Association only. No motion model yet - that arrives in the next lesson."""
def __init__(self, gate=0.2, max_age=3, min_hits=2):
self.gate, self.max_age, self.min_hits = gate, max_age, min_hits
self.tracks, self.next_id = {}, 1
def update(self, dets):
ids = list(self.tracks)
pairs = []
if ids and dets:
M = np.array([[iou(self.tracks[i]["box"], d) for d in dets] for i in ids])
r_idx, c_idx = linear_sum_assignment(-M)
pairs = [(r, c) for r, c in zip(r_idx, c_idx) if M[r, c] >= self.gate]
matched_t = {r for r, _ in pairs}
matched_d = {c for _, c in pairs}
for r, c in pairs: # matched: refresh the track
t = self.tracks[ids[r]]
t["box"], t["age"], t["hits"] = dets[c], 0, t["hits"] + 1
for r, i in enumerate(ids): # unmatched tracks grow older
if r not in matched_t:
self.tracks[i]["age"] += 1
for c, d in enumerate(dets): # unmatched detections start tracks
if c not in matched_d:
self.tracks[self.next_id] = {"box": d, "age": 0, "hits": 1}
self.next_id += 1
for i in [i for i, t in self.tracks.items() if t["age"] > self.max_age]:
del self.tracks[i] # missing too long: delete it
# A track is only reported once min_hits detections have confirmed it.
return {i: t["box"] for i, t in self.tracks.items()
if t["age"] == 0 and t["hits"] >= self.min_hits}
def box(x, w=20., h=40.):
return (x, 10., x + w, 10. + h)
frames = []
for t in range(9):
dets = [] if t in (3, 4) else [box(10. + 6 * t)] # the detector blinks at t=3,4
if t >= 5:
dets.append(box(90. - 4 * t, w=18.)) # a second person enters
frames.append(dets)
tr = Tracker()
for t, dets in enumerate(frames):
out = tr.update(dets)
print(f"frame {t}: {len(dets)} detections -> confirmed tracks "
+ str({i: f"x={b[0]:.0f}" for i, b in out.items()}))frame 0: 1 detections -> confirmed tracks {}
frame 1: 1 detections -> confirmed tracks {1: 'x=16'}
frame 2: 1 detections -> confirmed tracks {1: 'x=22'}
frame 3: 0 detections -> confirmed tracks {}
frame 4: 0 detections -> confirmed tracks {}
frame 5: 2 detections -> confirmed tracks {}
frame 6: 2 detections -> confirmed tracks {2: 'x=46', 3: 'x=66'}
frame 7: 2 detections -> confirmed tracks {2: 'x=52', 3: 'x=62'}
frame 8: 2 detections -> confirmed tracks {2: 'x=58', 3: 'x=58'}This tracker has a bug you can see in the output
The walker enters as track 1 and comes back as track 2. One person, two identities, one ID switch. The tracker was kept alive across the two missing frames by max_age=3, so why did it fail?
Because the track's stored box is stale. It was last seen at x=22 in frame 2. By frame 5 the person is at x=40. Those two boxes overlap by about 0.05, below the 0.2 gate, so the match is rejected and a new track is born.
The tracker knew the person was moving right at six pixels per frame and did nothing with that knowledge. It compared against where the person was, not where they would be.
That is exactly the gap a motion model fills, and it is the first thing SORT adds — see SORT, DeepSORT and ByteTrack.
min_hits explains the empty frames. Frames 0 and 5 have detections but report nothing, because a track needs two consecutive hits before it is trusted. This suppresses one-frame false detections at the cost of a one-frame reporting delay. Both are real costs and the trade is deliberate.
Common mistakes
Writing greedy matching. See the first output. linear_sum_assignment is one import.
No gate on the cost matrix. The Hungarian algorithm assigns every row it can, including pairs with an IoU of 0.01. Always discard matches below a threshold after solving.
Reporting tracks from their first frame. A single spurious detection becomes a track, gets an ID, and pollutes your counts. Use min_hits.
Deleting a track on the first missed frame. Detectors miss constantly. max_age between 10 and 30 frames is typical for 30-frame-per-second video.
Matching across classes. A person track should not match a car detection. Run separate cost matrices per class, or set cross-class costs to infinity.
Counting unique IDs as a count of objects. Every ID switch inflates the count. If counting is the goal, count line crossings by confirmed tracks rather than counting IDs.
Try it yourself
Give each track a vx field, updated as the change in x between matched frames, and compare detections against box + vx instead of box. Re-run and check whether track 1 survives the blink. That change is most of SORT.
What to learn next
- SORT, DeepSORT and ByteTrack — the three trackers everyone actually runs.
- Object detection — the component doing the seeing.
- YOLO — the detector most tracking pipelines are built on.
Researcher — Mathematics and papers.
The tracking-by-detection paradigm
Multi-object tracking splits into two families.
Tracking by detection runs a detector per frame and solves association afterwards. It inherits every detector improvement for free, which is why it has dominated since detectors became reliable. The tracker is a small, cheap, mostly hand-designed component.
Joint detection and tracking predicts boxes and associations in one network — CenterTrack, FairMOT, TransTrack, MOTR. Elegant, and empirically it has struggled to beat a strong detector plus a simple tracker at equal detector quality. The ByteTrack paper is the sharpest statement of that finding.
The assignment problem
Given $m$ tracks and $n$ detections with cost matrix $C \in \mathbb{R}^{m \times n}$, find a partial permutation minimising total cost:
$$ \min_{X} \sum_{i=1}^{m}\sum_{j=1}^{n} C_{ij} X_{ij} \quad \text{s.t.} \quad \sum_j X_{ij} \le 1,\; \sum_i X_{ij} \le 1,\; X_{ij} \in {0,1} $$
The Hungarian algorithm (Kuhn, 1955; Munkres, 1957) solves this in $O(n^3)$. scipy.optimize.linear_sum_assignment implements the Jonker-Volgenant variant, which is fast enough that association is never the bottleneck — detection is.
Greedy matching is $O(n^2 \log n)$ and has no approximation guarantee on this problem. The example in the developer section shows a case where greedy achieves 0.778 against the optimum of 1.081, and it is the common case in crowds, not a constructed worst case.
Cost functions
IoU is the baseline. It is scale-dependent in an awkward way: small objects moving a few pixels lose all overlap, while large objects retain overlap despite large motion. This makes a single global gate wrong for scenes with mixed object sizes.
GIoU, DIoU, CIoU extend IoU with a penalty on centre distance and aspect ratio, and remain informative when boxes do not overlap at all. DIoU in particular gives a usable gradient for non-overlapping boxes, which plain IoU does not.
Mahalanobis distance between a Kalman filter's predicted state and the detection, using the filter's covariance. This is what DeepSORT uses as a gate: it accounts for the tracker's own uncertainty, so a track that has been coasting through occlusion is given a wider search region than one updated last frame.
Appearance cosine distance between re-identification embeddings. The only cue that survives when boxes are ambiguous — see re-identification embeddings.
Modern trackers combine several. TrackTrack (Shim et al., CVPR 2025), the current default in Ultralytics, combines height-modulated IoU, optional cosine re-identification distance, a confidence-projection distance and a corner-angle distance, solved iteratively with a relaxing threshold.
Track lifecycle
The state machine is nearly universal:
| State | Entered when | Reported? |
|---|---|---|
| Tentative | a detection matches nothing | no |
| Confirmed | min_hits consecutive matches | yes |
| Lost | unmatched, age <= max_age | no |
| Deleted | unmatched, age > max_age | never again |
min_hits trades false-positive tracks against detection latency. max_age trades ID switches against identity contamination — a long max_age lets a track survive occlusion, and also lets it latch onto the wrong object when it reappears.
Track-aware initialisation matters more than it appears. Spawning a new track for every unmatched high-confidence detection produces duplicate identities on the same object when the detector emits two overlapping boxes. TrackTrack's second contribution is suppressing such duplicate spawns before an ID is issued.
Metrics, and why there are three
MOTA (Bernardin and Stiefelhagen, 2008):
$$ \text{MOTA} = 1 - \frac{\sum_t (\text{FN}_t + \text{FP}_t + \text{IDSW}_t)}{\sum_t \text{GT}_t} $$
Where FN is missed ground-truth objects, FP is false positives, IDSW is identity switches, and GT is the number of ground-truth objects. Its defect is that FN and FP are counted per frame while IDSW is a rare event, so MOTA is dominated by detection quality and barely reflects association quality at all.
IDF1 (Ristani et al., 2016) is an F1 score over identity-consistent detections, computed after a global bipartite matching between predicted and ground-truth trajectories. It measures association well and behaves non-intuitively with respect to detection.
HOTA (Luiten et al., IJCV 2021) decomposes tracking into detection, association and localisation, scores each with an IoU formulation, and combines them into one number that balances all three. It further decomposes into sub-metrics isolating five error types. The paper's argument is that MOTA over-weights detection and IDF1 over-weights association, and that HOTA aligns better with human judgement of tracking quality.
Report all three, or report HOTA and say so. A tracker tuned to maximise MOTA is a tracker tuned to maximise its detector.
Benchmarks
- MOT17 / MOT20: pedestrians, static cameras, MOT20 far more crowded. The long-standing standard.
- DanceTrack (Sun et al., CVPR 2022): dancers in near-identical outfits with irregular, non-linear motion. Deliberately breaks both cues — appearance is uninformative because outfits match, and motion is uninformative because dancers move erratically. Scores drop dramatically relative to MOT17, which is what makes it useful.
- BDD100K, KITTI: driving, moving camera, many classes.
- SportsMOT: fast, non-linear motion with frequent occlusion.
DanceTrack is the benchmark that reveals whether a tracker has real association machinery or a linear motion prior that happens to fit pedestrians walking in straight lines.
References
- Kuhn, The Hungarian Method for the Assignment Problem, Naval Research Logistics Quarterly, 1955.
- Bernardin and Stiefelhagen, Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics, EURASIP JIVP, 2008.
- Ristani et al., Performance Measures and a Data Set for Multi-Target, Multi-Camera Tracking, ECCV Workshops 2016 — arxiv.org/abs/1609.01775
- Luiten et al., HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking, IJCV 2021 — arxiv.org/abs/2009.07736
- Sun et al., DanceTrack, CVPR 2022 — arxiv.org/abs/2111.14690
- Shim et al., Focusing on Tracks for Online Multi-Object Tracking (TrackTrack), CVPR 2025.
What to learn next
- SORT, DeepSORT and ByteTrack — the three trackers everyone actually runs.
- Object detection — the component doing the seeing.
- YOLO — the detector most tracking pipelines are built on.