Video object segmentation

Setup#

pip install opencv-python numpy

Run against opencv-python 4.11.0 and numpy 1.26.4. We build the evaluation metrics and two baselines from scratch, because understanding what J&F measures is worth more than calling a library that prints it.

The metrics, implemented#

DAVIS, the standard benchmark, scores with two numbers averaged together.

J is region similarity: intersection over union between the predicted and true masks. F is boundary accuracy: an F1 score between the two outlines, allowing a small tolerance so a one-pixel offset is not punished. The headline number is J&F, their mean.

import cv2
import numpy as np

H = W = 96
N = 12
rng = np.random.default_rng(0)
texture = rng.integers(0, 90, (H, W), dtype=np.uint8)     # a static, textured background


def scene(t):
    """Frame t and its true mask: a bright disc drifting, growing and shrinking."""
    cx, cy = 22 + 2.7 * t, 38 + 1.3 * t
    r = 11 + 2.5 * np.sin(t / 2)
    yy, xx = np.mgrid[0:H, 0:W]
    mask = ((xx - cx) ** 2 + (yy - cy) ** 2) <= r * r
    frame = texture.copy()
    frame[mask] = 230
    return cv2.GaussianBlur(frame, (3, 3), 0.8), mask


def region_j(pred, gt):
    """DAVIS J: intersection over union of the two masks."""
    inter = np.logical_and(pred, gt).sum()
    union = np.logical_or(pred, gt).sum()
    return 1.0 if union == 0 else inter / union


def boundary_f(pred, gt, tol_frac=0.008):
    """DAVIS F: F1 between the two outlines, allowing a small tolerance."""
    def outline(m):
        m8 = m.astype(np.uint8)
        grad = cv2.morphologyEx(m8, cv2.MORPH_GRADIENT, np.ones((3, 3), np.uint8))
        return grad.astype(bool)

    tol = max(1, int(round(tol_frac * np.hypot(H, W))))
    k = np.ones((2 * tol + 1, 2 * tol + 1), np.uint8)
    bp, bg = outline(pred), outline(gt)
    bp_d = cv2.dilate(bp.astype(np.uint8), k).astype(bool)
    bg_d = cv2.dilate(bg.astype(np.uint8), k).astype(bool)
    prec = (bp & bg_d).sum() / max(bp.sum(), 1)
    rec = (bg & bp_d).sum() / max(bg.sum(), 1)
    return 0.0 if prec + rec == 0 else 2 * prec * rec / (prec + rec)


frames, gts = zip(*[scene(t) for t in range(N)])
copy_mask = gts[0].copy()          # baseline 1: never update the mask
flow_mask = gts[0].copy()          # baseline 2: push the mask along the optical flow

print(f"{'frame':>5}{'copy J':>9}{'copy F':>9}{'flow J':>9}{'flow F':>9}")
for t in range(1, N):
    flow = cv2.calcOpticalFlowFarneback(frames[t - 1], frames[t], None,
                                        0.5, 3, 15, 3, 5, 1.2, 0)
    dx = np.median(flow[..., 0][flow_mask]) if flow_mask.any() else 0.0
    dy = np.median(flow[..., 1][flow_mask]) if flow_mask.any() else 0.0
    M = np.float32([[1, 0, dx], [0, 1, dy]])
    flow_mask = cv2.warpAffine(flow_mask.astype(np.uint8), M, (W, H)).astype(bool)
    print(f"{t:5d}{region_j(copy_mask, gts[t]):9.3f}{boundary_f(copy_mask, gts[t]):9.3f}"
          f"{region_j(flow_mask, gts[t]):9.3f}{boundary_f(flow_mask, gts[t]):9.3f}")

frame   copy J   copy F   flow J   flow F
    1    0.702    0.631    0.807    0.952
    2    0.501    0.324    0.698    0.676
    3    0.359    0.209    0.653    0.516
    4    0.241    0.162    0.692    0.613
    5    0.137    0.134    0.791    0.810
    6    0.054    0.162    0.885    0.961
    7    0.000    0.179    0.834    0.959
    8    0.000    0.000    0.682    0.720
    9    0.000    0.000    0.619    0.605
   10    0.000    0.000    0.643    0.643
   11    0.000    0.000    0.702    0.688

What these two baselines prove#

Copying the first mask collapses to zero by frame 7. The object has moved off it entirely. This is the "do nothing" baseline, and any real method must beat it by a wide margin. Publishing a number without this comparison hides how much of the score is from the object barely moving.

Flow propagation tracks position and never fixes shape. The flow J column hovers between 0.6 and 0.9 and does not collapse, because the mask is being carried along by the measured motion. But it never returns to 1.0 either.

The reason is in the scene definition: the disc also grows and shrinks. A translation-only warp cannot represent that. So the mask keeps the wrong size, and J oscillates in step with the radius.

That oscillation is the argument for everything that came after. Propagating a mask by motion is a good start and a permanent ceiling. You need something that re-decides which pixels belong to the object using its appearance, not only where the object went.

J and F disagree, and both matter. At frame 6, flow J is 0.885 while flow F is 0.961. The region overlap is imperfect but the boundary lines up well. At frame 3 the reverse holds. A method can look strong on one and weak on the other, which is why DAVIS reports their mean rather than either alone.

How real systems work#

Modern methods replace the flow warp with a memory read. The pattern, established by STM (Oh et al., 2019) and refined by XMem and Cutie:

1. Encode each past frame together with its mask into key-value pairs, stored in a memory bank.
2. Encode the current frame into queries.
3. For each query position, attend over all memory keys and read back a weighted mix of the values.
4. Decode the result into a mask for this frame.
5. Add this frame and its predicted mask to the memory.

This is cross-attention over time, and step 5 is the drift risk. You are storing your own predictions as if they were ground truth. Every design in this family has a policy limiting what enters memory and for how long, because unbounded memory grows without limit and unfiltered memory poisons itself.

Running one of these needs a GPU and a checkpoint download of several hundred megabytes. Nothing about it is CPU-friendly, and this page will not pretend otherwise. The mechanism is what transfers; the code is a checkout of the authors' repository.

The current practical answer#

Since 2024 the general-purpose route is Meta's Segment Anything family. SAM 2 (Meta, August 2024) extended promptable segmentation to video with a memory attention module and a memory bank holding both past frames and object pointers. SAM 3, released 19 November 2025, added promptable concept segmentation: give a noun phrase like "yellow school bus" and it returns masks and consistent IDs for every matching object across the video, where SAM 1 and 2 handled one object per prompt.

For a working system today, that family is the sensible starting point. It is zero-shot, so it needs no training data of your own, and the Ultralytics and Hugging Face wrappers make it a few lines. It also needs a GPU for anything longer than a short clip, and the licence is Meta's own rather than a standard open-source one, so check it before shipping.

Common mistakes#

Evaluating with J alone. A blobby mask in roughly the right place scores well on J and poorly on F. Report both.

Feeding your own predicted masks back into memory without any filter. This is how drift becomes collapse. Keep the first frame's ground-truth mask permanently in memory, and be selective about what else you store.

Assuming an object that vanishes is gone. Occlusion is temporary; deletion is permanent. A system that deletes on the first missing frame cannot recover, and re-appearance after occlusion is a large fraction of real cases.

Testing on DAVIS 2016 and claiming a general result. It has one object per video, short sequences, and little occlusion. MOSE and LVOS were built specifically because DAVIS was saturated and unrepresentative.

Using a fixed threshold on the mask probability across a whole dataset. Object scale varies enormously. Tune the threshold per sequence, or use the boundary metric to choose it — model evaluation covers why a single global threshold rarely holds.

Try it yourself#

Add a third baseline that also rescales the mask, using the ratio of flow divergence inside the mask to estimate size change. Compare its J against the translation-only version. Then add an occlusion by blanking the disc for two frames, and watch what both baselines do.

What to learn next#

• Tracking by detection — the same identity problem, solved with boxes instead of masks.
• Image segmentation — the single-frame version of this task.
• Optical flow — the motion signal both baselines above depend on.