Learn AI Create the Future — with Pranay
Main site

Object Detection in Depth

Focal loss and RetinaNet

Focal loss quietens the tens of thousands of easy background boxes that drown a one-stage detector, which is what let RetinaNet match two-stage accuracy at one-stage speed.

Read these first

On this page 9
  1. The short answer
  2. The analogy
  3. Why it exists
  4. How it works
  5. What RetinaNet is
  6. The honest caveat
  7. Where you have seen the result
  8. Remember this
  9. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Focal loss turns down the volume on the easy answers, so the model can hear the hard ones.

The analogy

Think about marking ten thousand copies of an easy answer, mixed in with twenty copies of a hard one.

If you spend the same attention on every paper, you spend almost all of your day nodding at easy answers. By evening you have learned nothing about the hard question, because you barely looked at it.

A sensible marker skims the easy ones and slows down on the hard ones. Focal loss does that automatically, inside training.

Why it exists

A one-stage detector scores every candidate box in the picture at once. There can be a hundred thousand of them, and perhaps ten contain an object.

Every candidate contributes to the training signal. The background ones are individually easy: an empty patch of sky is obvious background. Each one contributes a tiny amount of learning pressure.

Tiny, times a hundred thousand, is enormous. The rare objects are drowned out. The model settles into answering "nothing here" to everything, because that is right almost all the time.

Two-stage detectors sidestep this. Their first stage throws away most of the background before the careful classifier ever sees it. One-stage detectors have no such filter, which is why they were less accurate for years.

How it works

   plain loss                        focal loss
   -----------                       -----------
   easy background   -> small        easy background   -> almost nothing
   hard background   -> medium       hard background   -> medium
   object            -> large        object            -> large

   multiply the easy ones by a hundred thousand
   and they still swamp everything

Focal loss adds one factor. The more confident and correct a prediction already is, the more its contribution shrinks.

An easy background box that the model already calls background at high confidence contributes almost nothing. A confusing box contributes its full share. The rare objects keep their voice.

What RetinaNet is

RetinaNet is the detector built to test this idea. A standard backbone, a feature pyramid, a dense head over every position, and this loss.

The result was the point of the paper: a one-stage detector matching the accuracy of two-stage detectors while staying fast. The change was in the loss, not the architecture.

The honest caveat

Focal loss reduces the imbalance. It does not remove it. Even with it, most of the training signal still comes from background. There is still an overwhelming amount of it.

It also adds two settings you have to choose. Choose them badly and training gets worse, not better.

Where you have seen the result

  • Any real-time detection demo running smoothly on video.
  • A camera counting people in a wide, mostly-empty frame.
  • A crack detector on a long stretch of pipeline images.
  • A drone finding a few animals across a large field.

Remember this

  • One-stage detectors score enormous numbers of mostly-empty boxes, so background swamps training.
  • Focal loss shrinks the contribution of predictions the model already gets confidently right.
  • RetinaNet showed this alone was enough to close the accuracy gap with two-stage detectors.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch torchvision

Written against torch 2.5.1 and torchvision 0.20.1. torchvision.ops.sigmoid_focal_loss is the reference implementation.

Measuring who pays the loss

focal.py
import torch
import torch.nn.functional as F
from torchvision.ops import sigmoid_focal_loss

# A realistic anchor batch: 10,000 candidate boxes, 20 of them on real objects.
N, POS = 10_000, 20
targets = torch.zeros(N)
targets[:POS] = 1.0

torch.manual_seed(0)
logits = torch.empty(N)
logits[:POS] = torch.randn(POS) * 0.5 + 0.5      # positives, weakly confident
logits[POS:] = torch.randn(N - POS) * 0.8 - 3.5  # negatives, mostly easy

p = torch.sigmoid(logits)
print(f"{POS} objects against {N - POS:,} background anchors")
print(f"mean predicted probability: objects {p[:POS].mean():.3f}, background {p[POS:].mean():.4f}")

bce = F.binary_cross_entropy_with_logits(logits, targets, reduction="none")
foc = sigmoid_focal_loss(logits, targets, alpha=0.25, gamma=2.0, reduction="none")

print("\nwho pays the loss:")
for name, x in [("plain BCE", bce), ("focal   ", foc)]:
    share = x[POS:].sum() / x.sum() * 100
    ratio = x[:POS].mean() / x[POS:].mean()
    print(f"  {name}  background share {share:5.1f}%   one object anchor is worth "
          f"{ratio:6.1f} background anchors")

print("\none easy background anchor, predicted probability 0.02:")
x, t = torch.tensor([-3.892]), torch.tensor([0.0])
b = F.binary_cross_entropy_with_logits(x, t)
f = sigmoid_focal_loss(x, t, alpha=0.25, gamma=2.0, reduction="mean")
print(f"  BCE   {b.item():.5f}")
print(f"  focal {f.item():.7f}   ({b.item() / f.item():.0f} times smaller)")

print("\none hard object anchor, predicted probability 0.3:")
x, t = torch.tensor([-0.847]), torch.tensor([1.0])
b = F.binary_cross_entropy_with_logits(x, t)
f = sigmoid_focal_loss(x, t, alpha=0.25, gamma=2.0, reduction="mean")
print(f"  BCE   {b.item():.5f}")
print(f"  focal {f.item():.5f}   ({b.item() / f.item():.1f} times smaller)")

print("\nhow gamma reshapes the loss on a background anchor (alpha fixed at 0.25):")
print(f"{'p':>6}{'BCE':>10}{'g=0':>10}{'g=1':>10}{'g=2':>10}{'g=5':>10}")
for prob in [0.01, 0.05, 0.2, 0.5, 0.9]:
    x = torch.logit(torch.tensor([prob]))
    row = [F.binary_cross_entropy_with_logits(x, torch.tensor([0.0])).item()]
    for g in [0.0, 1.0, 2.0, 5.0]:
        row.append(sigmoid_focal_loss(x, torch.tensor([0.0]), alpha=0.25,
                                      gamma=g, reduction="mean").item())
    print(f"{prob:>6}" + "".join(f"{v:>10.5f}" for v in row))

# The prior-bias trick RetinaNet needs to start training at all.
prior = 0.01
bias = -torch.log(torch.tensor((1 - prior) / prior))
print(f"\nRetinaNet sets the final classification bias to {bias:.3f},")
print(f"so every anchor starts at probability {torch.sigmoid(bias):.3f}")
naive = sigmoid_focal_loss(torch.zeros(N), targets, alpha=0.25, gamma=2.0, reduction="sum")
primed = sigmoid_focal_loss(torch.full((N,), bias.item()), targets,
                            alpha=0.25, gamma=2.0, reduction="sum")
print(f"first-step total loss, zero bias  : {naive:.1f}")
print(f"first-step total loss, prior bias : {primed:.1f}")
Output
20 objects against 9,980 background anchors
mean predicted probability: objects 0.653, background 0.0385

who pays the loss:
  plain BCE  background share  97.8%   one object anchor is worth   11.1 background anchors
  focal     background share  84.9%   one object anchor is worth   88.5 background anchors

one easy background anchor, predicted probability 0.02:
  BCE   0.02020
  focal 0.0000061   (3335 times smaller)

one hard object anchor, predicted probability 0.3:
  BCE   1.20376
  focal 0.14743   (8.2 times smaller)

how gamma reshapes the loss on a background anchor (alpha fixed at 0.25):
     p       BCE       g=0       g=1       g=2       g=5
  0.01   0.01005   0.00754   0.00008   0.00000   0.00000
  0.05   0.05129   0.03847   0.00192   0.00010   0.00000
   0.2   0.22314   0.16736   0.03347   0.00669   0.00005
   0.5   0.69315   0.51986   0.25993   0.12997   0.01625
   0.9   2.30258   1.72694   1.55424   1.39882   1.01974

RetinaNet sets the final classification bias to -4.595,
so every anchor starts at probability 0.010
first-step total loss, zero bias  : 1297.9
first-step total loss, prior bias : 22.6

Reading that output, because the interesting numbers are not the obvious ones

Under plain BCE, one object anchor is worth 11 background anchors. Under focal loss it is worth 88. That eight-fold change in relative weight is the actual mechanism. It summarises focal loss better than any single loss value.

Background still pays 84.9 percent of the focal loss. Focal loss did not eliminate the imbalance, it reduced it from 97.8 percent. Anyone claiming focal loss solves class imbalance has not measured it. It shifts the balance enough to train, and that was sufficient.

An easy background anchor gets 3,335 times less loss. A probability of 0.02 on a background anchor is already correct. Multiplying by the modulating factor makes it near-invisible.

A hard object anchor gets only 8.2 times less loss. Both numbers shrank, which surprises people. The point is not the absolute size, it is the ratio between them. Easy examples shrank 400 times more than hard ones did.

Look at the gamma column at p=0.9, on a background anchor. That is a confident wrong answer. Going from gamma 0 to gamma 5 barely moves it: 1.73 down to 1.02. Focal loss deliberately leaves hard, wrong examples nearly intact while crushing easy correct ones. That asymmetry is the design.

The prior bias changes the first step from 1297.9 to 22.6. Without it, an untrained model with zero bias predicts probability 0.5 everywhere. All 10,000 anchors are then "uncertain", and every one contributes full loss. The gradient at step one is enormous and training diverges. RetinaNet initialises the final classification bias so every anchor starts at probability 0.01. This is not a detail, it is required for the model to train at all.

Using it in a real detector

retinanet.py
import torch
from torchvision.models.detection import retinanet_resnet50_fpn

torch.manual_seed(0)
model = retinanet_resnet50_fpn(weights=None, weights_backbone=None, num_classes=21)
model.train()

losses = model([torch.rand(3, 320, 320)],
               [{"boxes": torch.tensor([[50., 60., 180., 240.]]),
                 "labels": torch.tensor([3])}])
print("RetinaNet returns two losses, not four:")
for k, v in losses.items():
    print(f"  {k:<18}{v.item():.4f}")
print("\nparameters:", f"{sum(p.numel() for p in model.parameters()):,}")
print("head is shared across all pyramid levels:",
      type(model.head.classification_head).__name__)
Output
RetinaNet returns two losses, not four:
  classification    1.1848
  bbox_regression   0.5926

parameters: 32,615,969
head is shared across all pyramid levels: RetinaNetClassificationHead

The loss values come from random initialisation with a fixed seed. Treat them as a smoke test rather than a measurement. The structure is the point: two losses instead of Faster R-CNN's four, because there is no proposal stage to supervise.

Choosing alpha and gamma

gamma controls how hard easy examples are suppressed. alpha is a plain class weight applied on top.

  • gamma = 2.0, alpha = 0.25 are the paper's values and a sound default.
  • Raising gamma suppresses easy examples harder. Above about 5 the loss becomes dominated by a handful of hard examples, and label noise starts to steer training.
  • alpha = 0.25 weights the positive class at 0.25 and negatives at 0.75, which looks backwards. It is not. Gamma already suppresses the abundant negatives so aggressively that positives end up over-weighted, and alpha corrects back. The two parameters interact, so tune them together or not at all.

Normalise by the number of positive anchors, not the total. torchvision's retinanet does this internally. Write your own head and normalise by N, and your loss scale changes with image content. Your learning rate then stops meaning anything.

Common mistakes

Using softmax cross-entropy instead of per-class sigmoid. Focal loss as published is binary, applied independently per class. A softmax forces the class scores to compete, which changes the imbalance structure.

Skipping the prior bias initialisation. The most common cause of "my RetinaNet loss is NaN at step 3".

Normalising by total anchors. Divide by the number of positives. Otherwise images with more objects get quieter gradients, which is exactly backwards.

Applying focal loss to a balanced problem. On a two-class problem with a 50-50 split it does nothing useful and makes calibration worse.

Reading focal loss values across runs. The number depends on the positive count, gamma, alpha and the normaliser. Compare AP, not loss.

Try it yourself

Set gamma=0.0 in the first script and re-run. Focal loss becomes weighted BCE, and the "one object anchor is worth" number should collapse back towards the BCE value. Then sweep gamma from 0 to 4 and plot that ratio. You have measured the only thing focal loss actually does.

What to learn next

Researcher — Mathematics and papers.

The loss

Binary cross-entropy for one anchor, written with $p_t$:

$$ p_t = \begin{cases} p & y = 1 \ 1 - p & y = 0 \end{cases}, \qquad \mathrm{CE}(p_t) = -\log(p_t) $$

Where $p$ is the predicted probability for the positive class and $y \in {0, 1}$ the label. Focal loss (Lin et al., 2017) inserts a modulating factor:

$$ \mathrm{FL}(p_t) = -\alpha_t (1 - p_t)^{\gamma} \log(p_t) $$

Where $\gamma \geq 0$ is the focusing parameter and $\alpha_t$ is $\alpha$ for positives and $1 - \alpha$ for negatives. At $\gamma = 0$ this is $\alpha$-weighted cross-entropy.

Two properties drive the behaviour. When $p_t \to 1$ the factor $(1 - p_t)^\gamma \to 0$, so well-classified examples vanish from the loss. When $p_t$ is small the factor is near 1 and the example is untouched.

The gradient with respect to the logit $x$, for $y = 1$:

$$ \frac{\partial \mathrm{FL}}{\partial x} = \alpha (1 - p)^{\gamma} \big(\gamma p \log(p) + p - 1\big) $$

The bracket is bounded, and the prefactor decays as $p \to 1$. This is what prevents the accumulated gradient of $10^5$ easy negatives from dominating.

Why $\alpha = 0.25$ weights positives down

Reading the paper's ablation table matters here. At $\gamma = 0$, the best $\alpha$ is 0.75, weighting the rare positives up, exactly as intuition suggests. At $\gamma = 2$, the best $\alpha$ is 0.25.

The reason is that $\gamma$ has already removed most of the negatives' contribution. Applying a positive-favouring $\alpha$ on top over-corrects. The two parameters are coupled, and the paper says so explicitly.

The prior initialisation

The final classification layer's bias is set to

$$ b = -\log!\left(\frac{1 - \pi}{\pi}\right), \qquad \pi = 0.01 $$

So that every anchor's initial predicted probability is $\pi$. Without it the total first-step loss is dominated by the $\sim 10^5$ anchors sitting at $p = 0.5$, and training is unstable. The paper reports this as necessary, not as a refinement.

Competing accounts of the imbalance

Focal loss is one answer to imbalance. It is not the only one, and the field has not fully settled.

  • OHEM (Shrivastava et al., 2016) keeps only the highest-loss examples. The paper's ablation reports focal loss outperforming OHEM under matched conditions.
  • Gradient Harmonizing Mechanism (Li et al., 2019) reweights by gradient-norm density rather than by confidence. That also handles outliers with very large gradients.
  • Prime Sample Attention (Cao et al., 2020) argues the useful axis is not hard-versus-easy. What matters is which samples determine the ranking that mAP measures.
  • Assignment-based approaches attack the problem upstream: ATSS, SimOTA and task-aligned assignment all reduce the number of ambiguous candidates before any loss is computed. Modern detectors lean on this route more than on the loss.

Generalized Focal Loss

Li et al. (2020) extend focal loss to continuous targets, which lets a single head predict a quality-aware classification score:

$$ \mathrm{QFL}(\sigma) = -|y - \sigma|^{\beta}\big((1-y)\log(1-\sigma) + y\log(\sigma)\big) $$

Here $y \in [0, 1]$ is the IoU of the predicted box with its target, and 0 for negatives. $\sigma$ is the predicted score, and $\beta$ plays the role of $\gamma$. The modulating factor becomes the absolute error $|y - \sigma|$, which is defined for non-binary $y$.

This merges the classification score and the localisation-quality score into one number. It removes the inconsistency where a detector is confident about a poorly-placed box. Its companion, Distribution Focal Loss, is the box representation used in modern YOLO heads; see YOLO versions compared.

RetinaNet as an architecture

ResNet plus FPN over levels P3 to P7, with nine anchors per location. Those nine are three scales at one-third-octave spacing crossed with three aspect ratios. Two subnets of four 3x3 convolutions each follow, one for classification and one for box regression. Both are shared across all pyramid levels.

The classification subnet does not share weights with the box subnet. The paper reports this decoupling as beneficial, and the same finding appears independently in YOLOX and FCOS. A single shared tower for both tasks measurably underperforms. The features useful for "what is it" and for "exactly where is it" differ.

Papers

What to learn next