Feature pyramid networks

Setup#

pip install torch torchvision

Written against torch 2.5.1 and torchvision 0.20.1. torchvision.ops.FeaturePyramidNetwork takes an OrderedDict of feature maps and returns one with the same keys.

Building a pyramid and proving the top-down path works#

import math
from collections import OrderedDict
import torch
from torchvision.ops import FeaturePyramidNetwork

torch.manual_seed(0)

# What a ResNet hands over for a 640x640 image: deep maps are small but rich.
backbone_out = OrderedDict([
    ("c2", torch.rand(1,  256, 160, 160)),   # stride  4
    ("c3", torch.rand(1,  512,  80,  80)),   # stride  8
    ("c4", torch.rand(1, 1024,  40,  40)),   # stride 16
    ("c5", torch.rand(1, 2048,  20,  20)),   # stride 32
])
for k, v in backbone_out.items():
    print(f"{k}: {tuple(v.shape)}  stride {640 // v.shape[-1]}")

fpn = FeaturePyramidNetwork(in_channels_list=[256, 512, 1024, 2048], out_channels=256)
with torch.no_grad():
    pyramid = fpn(backbone_out)

print("\nafter the FPN, every level speaks the same language (256 channels):")
for k, v in pyramid.items():
    print(f"{k}: {tuple(v.shape)}")

print(f"\nFPN parameters: {sum(p.numel() for p in fpn.parameters()):,}"
      "  (a 1x1 lateral conv and a 3x3 output conv per level)")

# Proof that the top-down path really carries deep meaning downwards:
# change ONLY the deepest map and watch every level move, including the shallowest.
with torch.no_grad():
    changed = OrderedDict(backbone_out)
    changed["c5"] = torch.rand(1, 2048, 20, 20)
    pyramid2 = fpn(changed)
print("\nmean change per level after replacing c5 alone:")
for k in pyramid:
    print(f"  {k}: {(pyramid[k] - pyramid2[k]).abs().mean():.4f}")

# Which level should a given object be detected on? Faster R-CNN's FPN rule.
def level_for(w, h, k0=4, canonical=224):
    return int(min(5, max(2, math.floor(k0 + math.log2(math.sqrt(w * h) / canonical + 1e-6)))))

print("\nobject size -> pyramid level (Faster R-CNN FPN assignment):")
for w, h, what in [(20, 20, "a distant bird"), (60, 90, "a helmet"),
                   (150, 300, "a person"), (400, 380, "a lorry")]:
    lvl = level_for(w, h)
    print(f"  {what:<16}{w:>4} x{h:<4} -> P{lvl}, stride {2**lvl}")

print("\ngrid cells a 24x24 object covers at each stride:")
for stride in [4, 8, 16, 32]:
    print(f"  stride {stride:>2}: {(24 / stride) ** 2:6.2f} cells")

c2: (1, 256, 160, 160)  stride 4
c3: (1, 512, 80, 80)  stride 8
c4: (1, 1024, 40, 40)  stride 16
c5: (1, 2048, 20, 20)  stride 32

after the FPN, every level speaks the same language (256 channels):
c2: (1, 256, 160, 160)
c3: (1, 256, 80, 80)
c4: (1, 256, 40, 40)
c5: (1, 256, 20, 20)

FPN parameters: 3,344,384  (a 1x1 lateral conv and a 3x3 output conv per level)

mean change per level after replacing c5 alone:
  c2: 0.3250
  c3: 0.3234
  c4: 0.3201
  c5: 0.3128

object size -> pyramid level (Faster R-CNN FPN assignment):
  a distant bird    20 x20   -> P2, stride 4
  a helmet          60 x90   -> P2, stride 4
  a person         150 x300  -> P3, stride 8
  a lorry          400 x380  -> P4, stride 16

grid cells a 24x24 object covers at each stride:
  stride  4:  36.00 cells
  stride  8:   9.00 cells
  stride 16:   2.25 cells
  stride 32:   0.56 cells

Reading that output#

Channel counts collapse to 256 everywhere. The backbone hands over 256, 512, 1024 and 2048 channels. The FPN's 1x1 lateral convolutions project all of them to a common width. That is what lets one shared detection head run on every level. A head with different weights per level would be a different architecture, and a worse one.

Spatial sizes are untouched. 160, 80, 40 and 20 go in and come out. The FPN changes what each cell means, not where cells are.

The top-down test is the important one. Replacing only c5 changed c2 by 0.3250 on average. c2 never touched c5 directly. The signal travelled: c5 was upsampled and added into c4, which was upsampled and added into c3, and so on. That chain is the entire point of the architecture.

These magnitudes come from an untrained FPN with random weights, seeded at zero for reproducibility. Do not read meaning into the exact values. Read meaning into the fact that none of them is zero.

3.3 million parameters. A ResNet-50 backbone is around 25 million. The pyramid costs about an eighth of the backbone and lifts small-object accuracy substantially. That ratio is why it became standard.

The level assignment rule sends a 20-pixel bird and a 60-by-90 helmet to the same level. The formula floors at P2. Everything smaller than roughly 112 pixels on a side lands on the finest level.

This is a known weakness. The finest level carries most of the work and most of the small objects. Detectors for tiny objects add a P1 level, or tile the image instead.

The last table is the reason all of this exists. A 24-pixel object covers 0.56 cells at stride 32. Less than one cell. There is nothing to classify. At stride 4 it covers 36 cells, which is plenty.

Adding an FPN to a real backbone#

import torch
from torchvision.models import resnet50
from torchvision.models.detection.backbone_utils import BackboneWithFPN

body = resnet50(weights=None)                    # weights=None: nothing is downloaded
backbone = BackboneWithFPN(
    body,
    return_layers={"layer1": "0", "layer2": "1", "layer3": "2", "layer4": "3"},
    in_channels_list=[256, 512, 1024, 2048],
    out_channels=256,
)

with torch.no_grad():
    out = backbone(torch.zeros(1, 3, 640, 640))

for k, v in out.items():
    print(f"level {k}: {tuple(v.shape)}  stride {640 // v.shape[-1]}")
print("\nbackbone+FPN parameters:", f"{sum(p.numel() for p in backbone.parameters()):,}")

level 0: (1, 256, 160, 160)  stride 4
level 1: (1, 256, 80, 80)  stride 8
level 2: (1, 256, 40, 40)  stride 16
level 3: (1, 256, 20, 20)  stride 32
level pool: (1, 256, 10, 10)  stride 64

backbone+FPN parameters: 26,852,416

BackboneWithFPN adds a pool level by max-pooling the deepest output, giving a stride-64 map for very large objects. Detection models in torchvision expect that key to exist. Worth knowing before you write your own backbone wrapper.

The pyramid variants you will meet in configs#

Every one of them is the same idea: move information between resolutions before detecting. They differ in which direction, how many times, and whether the mixing weights are learned.

Common mistakes#

Feeding a plain dict instead of an OrderedDict. Level ordering decides which map is upsampled into which. In Python 3.7 and later a plain dict preserves insertion order, so this usually works. The API asks for OrderedDict, and being explicit costs nothing.

Mismatched in_channels_list. It must line up with the maps you pass, in the same order. A mismatch fails at the first lateral convolution with a shape error that names a layer you did not write.

Detecting from the raw backbone C maps. Those have not been mixed. Use the pyramid outputs, not the backbone outputs.

Assuming more levels is better. Each level costs memory and adds candidate boxes. Levels that produce no positives on your data are pure cost. Print positives per level once and delete the dead ones.

Ignoring where your objects actually live. If everything in your dataset is 20 pixels across, every object lands on one level. The rest of the pyramid is wasted. Consider a finer level or tiling instead.

Try it yourself#

Add a stride-2 level to the first snippet by prepending a (1, 128, 320, 320) map called c1. Print the new parameter count and the new total cell count across the pyramid. That cell count is how many candidate positions your head must score every frame. It is the honest price of detecting smaller things.

What to learn next#

• RoI pooling and RoI align — pulling a fixed-size patch out of a pyramid level.
• Convolutional neural networks — why deep layers lose detail in the first place.
• Detecting very small objects — where the pyramid runs out of road.