DETR and set prediction

The set prediction loss#

DETR predicts a fixed set $\hat{y} = {\hat{y}i}{i=1}^{N}$ with $N$ larger than the maximum object count. The ground-truth set $y$ is padded with $\varnothing$ (no-object) to size $N$.

The optimal permutation $\hat{\sigma}$ minimises a total matching cost:

$$ \hat{\sigma} = \arg\min_{\sigma \in \mathfrak{S}N} \sum{i=1}^{N} \mathcal{L}_{\text{match}}\big(y_i, \hat{y}_{\sigma(i)}\big) $$

Where $\mathfrak{S}_N$ is the set of permutations of $N$ elements. The matching cost for a non-empty target with class $c_i$ and box $b_i$ is:

$$ \mathcal{L}{\text{match}} = -\mathbb{1}{{c_i \neq \varnothing}} \hat{p}_{\sigma(i)}(c_i)

• \mathbb{1}_{{c_i \neq \varnothing}} \mathcal{L}_{\text{box}}\big(b_i, \hat{b}_{\sigma(i)}\big) $$

Note it uses the raw probability $\hat p$, not its log. The Hungarian algorithm (Kuhn, 1955) solves this in $O(N^3)$; scipy.optimize.linear_sum_assignment implements a modern variant of it. With $N = 100$ this is negligible next to the forward pass.

The training loss then applies negative log-likelihood over classes and the box loss on matched pairs:

$$ \mathcal{L}{\text{Hungarian}} = \sum{i=1}^{N}\Big[-\log \hat{p}_{\hat\sigma(i)}(c_i)

• \mathbb{1}_{{c_i \neq \varnothing}} \mathcal{L}_{\text{box}}\big(b_i, \hat{b}_{\hat\sigma(i)}\big)\Big] $$

With $\mathcal{L}{\text{box}} = \lambda{L1}|b - \hat b|1 + \lambda{\text{giou}}\mathcal{L}_{\text{giou}}$, weights 5 and 2 in the paper. The log-probability weight for $c_i = \varnothing$ is divided by 10 to counter the class imbalance. The matching cost and the training loss are deliberately different functions.

Why the matching is a permutation, not a soft assignment#

The one-to-one constraint is what removes NMS. Any relaxation that lets two predictions share a target reintroduces duplicates.

This is also the source of the instability that DN-DETR (Li et al., 2022) identified. The optimal permutation can flip between epochs for the same image. A query then receives contradictory targets across steps. Their fix adds noised copies of ground-truth boxes as extra queries. Those have a fixed, known assignment. The model gets a stable denoising signal alongside the unstable matching one.

The convergence problem, diagnosed#

Zhu et al. (2021), Deformable DETR, state the cause directly. DETR "suffers from slow convergence and limited feature spatial resolution". Transformer attention over image features starts nearly uniform, and needs many epochs to become selective. Dense attention over a high-resolution map is also quadratic in cost.

Their fix is multi-scale deformable attention. Each query predicts $K$ sampling offsets per level and attends only to those points:

$$ \mathrm{MSDeformAttn}(z_q, \hat{p}q, {x^l}) = \sum{m=1}^{M} W_m \left[\sum_{l=1}^{L}\sum_{k=1}^{K} A_{mlqk} \cdot W'_m x^l\big(\phi_l(\hat p_q) + \Delta p_{mlqk}\big)\right] $$

Here $z_q$ is the query feature and $\hat p_q$ its normalised reference point. $\phi_l$ rescales that point to level $l$. $\Delta p_{mlqk}$ are learned sampling offsets. $A_{mlqk}$ are learned attention weights that sum to one over $l$ and $k$. The index $m$ runs over attention heads. Cost becomes linear in feature-map size, and multi-scale features become affordable, which is what fixes small-object accuracy.

The paper reports better performance than DETR, especially on small objects, with ten times fewer training epochs.

Query design, the thread that runs through the whole family#

• DETR: queries are free learned embeddings with no spatial meaning.
• Conditional DETR (Meng et al., 2021): decodes a spatial query from the reference point, decoupling content and position.
• DAB-DETR (Liu et al., 2022): queries are 4D boxes $(x, y, w, h)$, refined at every decoder layer, with width and height modulating the attention's spatial extent.
• DINO (Zhang et al., 2022): mixed query selection initialises positional queries from encoder outputs, keeping content queries learnable. It adds contrastive denoising with positive and negative noise levels.

Read them in that order and one idea gets progressively more explicit. It converges on something close to a learned, iteratively-refined proposal set. Sparse R-CNN arrived at a similar place from the two-stage direction.

Where the line stands#

RT-DETR (Zhao et al., 2023) made the family real-time. Its hybrid encoder decouples intra-scale interaction from cross-scale fusion, and query selection becomes IoU-aware. The line continued through RT-DETRv2, D-FINE and DEIM.

RT-DETRv4 (ECCV 2026) reports 55.4 AP at 124 FPS for its L variant. That is against DEIM-L at 54.7 and YOLOv13-L at 53.4. Its X variant reports 57.0 AP. Its contribution is a training-only Deep Semantic Injector. It aligns the deepest feature map with a frozen DINOv3 backbone, so inference cost is unchanged.

That last detail points at the most interesting current trend in detection. Gains come from what supervises the model during training, not from what runs at inference.

Papers#

• Carion et al., End-to-End Object Detection with Transformers, ECCV 2020 — arxiv.org/abs/2005.12872
• Zhu et al., Deformable DETR, ICLR 2021 — arxiv.org/abs/2010.04159
• Liu et al., DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR, ICLR 2022 — arxiv.org/abs/2201.12329
• Li et al., DN-DETR: Accelerate DETR Training by Introducing Query DeNoising, CVPR 2022 — arxiv.org/abs/2203.01305
• Zhang et al., DINO: DETR with Improved DeNoising Anchor Boxes, ICLR 2023 — arxiv.org/abs/2203.03605
• Zhao et al., DETRs Beat YOLOs on Real-time Object Detection, CVPR 2024 — arxiv.org/abs/2304.08069

What to learn next#

• Open-vocabulary detection — attaching language to this architecture.
• Transformers — the encoder and decoder underneath.
• Attention — the mechanism the queries use to find their object.