Learn AI Create the Future — with Pranay
Main site

Segmentation in Depth

DeepLab and atrous convolution

Atrous convolution spreads a filter's fingers apart so it sees a wide area without shrinking the picture, which is how DeepLab keeps detail and context at once.

Read these first

On this page 8
  1. The short answer
  2. The analogy
  3. Why it exists
  4. How it works
  5. Where you have already seen this
  6. The catch nobody mentions first
  7. Remember this
  8. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

The short answer

Atrous convolution spreads a small filter's fingers apart, so it covers a wide area without the picture being shrunk.

The analogy

Measure a table with your hand. Keep your fingers together and one span covers a small stretch of wood.

Now spread your fingers wide. You still have five fingertips touching, and the same hand. That one span now reaches much further across the table.

You gained reach without gaining fingers. You did skip the wood between your fingertips — that is the honest cost of the trick.

A filter in a network works the same way. Spread its sampling points apart and it sees a wide region using the same small number of weights.

Why it exists

The U-Net answer was to shrink the picture, then grow it back. The detail travelled across sideways. It works, and it is not the only route.

Look at the problem again. A classification network shrinks a photo five times over. A large photo ends up as a tiny grid of scores, and every fine edge has been thrown away.

You could refuse to shrink. Then each filter only ever sees a tiny neighbourhood. The network never gains the context to tell a road from a rooftop.

Atrous convolution breaks that trade. Stop shrinking after a point, and spread the filters instead. Wide view, full-size picture, same number of weights.

The word atrous is French for "with holes", which describes the gaps between the spread-out fingers. Many libraries call the same thing dilation.

How it works

   normal filter, fingers together      atrous filter, fingers spread
   sampling points marked X             sampling points marked X

        X X X                             X . X . X
        X X X                             . . . . .
        X X X                             X . X . X
                                          . . . . .
        covers three across               X . X . X

        nine weights                      covers five across
                                          still nine weights

There is a second idea in DeepLab, and it is as important. A photo contains a lorry that fills the frame and a road sign the size of a thumbnail.

One spread setting cannot suit both. So DeepLab runs several spreads side by side over the same picture, and blends the results.

That parallel bank of filters is called ASPP, short for atrous spatial pyramid pooling. Narrow spreads catch small objects, wide spreads catch large ones, and the network learns how much to trust each.

Where you have already seen this

  • Portrait mode and background replacement on phones, where the outline has to follow hair and shoulders.
  • Map making from satellite photos, shading buildings, water and farmland.
  • Driving assistance systems that mark the drivable road surface ahead.

The catch nobody mentions first

Spreading fingers has a real cost, and it is easy to see. Stack several filters with the same wide spread, and they all land on the same alternating points.

Half the picture is then never touched by that stack. The output picks up a faint checkerboard pattern, and thin things fall in the gaps and vanish.

The fix is not to use one wide spread everywhere. Mix the spreads, which is what ASPP does anyway.

Remember this

  • Atrous spreads a filter's sampling points apart: wider view, same weights.
  • It lets a network keep the picture large instead of shrinking it.
  • ASPP runs several spreads at once, because objects come in many sizes.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install torch torchvision

All of this runs on CPU in seconds. Nothing downloads.

What a dilation rate actually buys

The clean way to measure a receptive field is to push a single spike through the stack and see how far it spreads.

atrous_basics.py
import torch, torch.nn as nn, torch.nn.functional as F

# ---------- 1. what a dilation rate buys you ----------
def support(rates, length=41):
    """Push a single spike through stacked 3-tap filters and see how wide it spreads."""
    x = torch.zeros(1, 1, length); x[0, 0, length // 2] = 1.0
    w = torch.ones(1, 1, 3)
    for r in rates:
        x = F.conv1d(x, w, padding=r, dilation=r)
    return (x[0, 0] > 0)

for rates in [(1, 1, 1), (1, 2, 4), (2, 2, 2)]:
    s = support(rates)
    print(f"rates {str(rates):<10} width {int(s.sum()):>3}  span {int(s.nonzero().max()-s.nonzero().min())+1:>3}  "
          + "".join("#" if v else "." for v in s))

# ---------- 2. same weights, different reach ----------
conv1 = nn.Conv2d(64, 64, 3, padding=1, dilation=1)
conv6 = nn.Conv2d(64, 64, 3, padding=6, dilation=6)
x = torch.zeros(1, 64, 64, 64)
print(f"\ndilation 1 : params {sum(p.numel() for p in conv1.parameters()):,}  out {tuple(conv1(x).shape)}")
print(f"dilation 6 : params {sum(p.numel() for p in conv6.parameters()):,}  out {tuple(conv6(x).shape)}")

# ---------- 3. output stride: the problem DeepLab set out to solve ----------
img = 224
print("\nplain classification backbone, five stride-2 stages:")
size = img
for stage in range(1, 6):
    size //= 2
    print(f"   stage {stage}: {size:>3} x {size:<3}  (output stride {img // size})")
print(f"   a {img}x{img} photo becomes a {size}x{size} score map -> {img // size}x upsample to get it back")

print("\nsame backbone with the last two stages switched to atrous (stride 1, rate 2 then 4):")
size = img
for stage, stride in enumerate([2, 2, 2, 1, 1], start=1):
    size //= stride
    print(f"   stage {stage}: {size:>3} x {size:<3}  (output stride {img // size})")

# ---------- 4. ASPP ----------
class ASPP(nn.Module):
    def __init__(self, cin=512, cout=256, rates=(6, 12, 18)):
        super().__init__()
        self.b0 = nn.Sequential(nn.Conv2d(cin, cout, 1, bias=False), nn.BatchNorm2d(cout), nn.ReLU())
        self.branches = nn.ModuleList([
            nn.Sequential(nn.Conv2d(cin, cout, 3, padding=r, dilation=r, bias=False),
                          nn.BatchNorm2d(cout), nn.ReLU()) for r in rates])
        self.pool = nn.Sequential(nn.AdaptiveAvgPool2d(1),
                                  nn.Conv2d(cin, cout, 1, bias=False), nn.ReLU())
        self.project = nn.Sequential(nn.Conv2d(cout * (len(rates) + 2), cout, 1, bias=False),
                                     nn.BatchNorm2d(cout), nn.ReLU())
    def forward(self, x):
        hw = x.shape[-2:]
        parts = [self.b0(x)] + [b(x) for b in self.branches]
        parts.append(F.interpolate(self.pool(x), size=hw, mode="bilinear", align_corners=False))
        for name, p in zip(["1x1", "rate 6", "rate 12", "rate 18", "image pool"], parts):
            print(f"   {name:<11} {tuple(p.shape)}")
        return self.project(torch.cat(parts, dim=1))

print("\nASPP on a 28x28 feature map (output stride 8 for a 224x224 image):")
aspp = ASPP()
out = aspp(torch.zeros(1, 512, 28, 28))
print(f"   concatenated -> projected {tuple(out.shape)}")
print(f"   ASPP parameters: {sum(p.numel() for p in aspp.parameters()):,}")
Output
rates (1, 1, 1)  width   7  span   7  .................#######.................
rates (1, 2, 4)  width  15  span  15  .............###############.............
rates (2, 2, 2)  width   7  span  13  ..............#.#.#.#.#.#.#..............

dilation 1 : params 36,928  out (1, 64, 64, 64)
dilation 6 : params 36,928  out (1, 64, 64, 64)

plain classification backbone, five stride-2 stages:
   stage 1: 112 x 112  (output stride 2)
   stage 2:  56 x 56   (output stride 4)
   stage 3:  28 x 28   (output stride 8)
   stage 4:  14 x 14   (output stride 16)
   stage 5:   7 x 7    (output stride 32)
   a 224x224 photo becomes a 7x7 score map -> 32x upsample to get it back

same backbone with the last two stages switched to atrous (stride 1, rate 2 then 4):
   stage 1: 112 x 112  (output stride 2)
   stage 2:  56 x 56   (output stride 4)
   stage 3:  28 x 28   (output stride 8)
   stage 4:  28 x 28   (output stride 8)
   stage 5:  28 x 28   (output stride 8)

ASPP on a 28x28 feature map (output stride 8 for a 224x224 image):
   1x1         (1, 256, 28, 28)
   rate 6      (1, 256, 28, 28)
   rate 12     (1, 256, 28, 28)
   rate 18     (1, 256, 28, 28)
   image pool  (1, 256, 28, 28)
   concatenated -> projected (1, 256, 28, 28)
   ASPP parameters: 4,131,328

Reading the output, line by line

The gridding artefact is visible in row three. Rates (2, 2, 2) produce #.#.#.#.#.#.# — a comb. The span is 13, but only 7 positions are ever touched. Six positions inside the receptive field contribute nothing at all.

Compare rates (1, 2, 4): span 15, and every one of those 15 positions is covered. Mixed rates give a solid receptive field; repeated rates give a sieve. This is why DeepLab uses rates like 6, 12, 18 rather than 12, 12, 12. Remember it whenever you set dilation by hand.

Dilation is free in parameters and free in output size. Both convolutions hold 36,928 parameters and both return a 64x64 map. Only the padding changed to keep the size, which is the rule padding = dilation for a 3x3 kernel.

Output stride is the whole motivation. A standard backbone turns 224 pixels into a 7x7 grid. Every value in that grid stands for a 32x32 patch of the original photo. A boundary can be placed to within 32 pixels, and upsampling guesses the rest.

Switching the last two stages to stride 1 with dilation holds the map at 28x28. Each value now stands for an 8x8 patch. That is the trade DeepLab makes.

It is not free. Those last stages now run on a map four times larger in each dimension, so their compute rises roughly sixteen-fold. Output stride 8 is accurate and slow; output stride 16 is the usual production compromise.

ASPP's five branches each return 256 channels at the same size. They concatenate to 1280 channels, then a 1x1 convolution projects back to 256. The image-pooling branch is the one people skip: it averages the entire map to a single value per channel, then broadcasts it back. Without it, no branch ever sees the whole image.

The same thing in torchvision

deeplab_shapes.py
import torch
from torchvision.models.segmentation import deeplabv3_resnet50

m = deeplabv3_resnet50(weights=None, weights_backbone=None, num_classes=21, aux_loss=True).eval()
print("deeplabv3_resnet50 parameters:", f"{sum(p.numel() for p in m.parameters()):,}")

x = torch.zeros(1, 3, 128, 128)
with torch.no_grad():
    out = m(x)
print("output keys:", list(out.keys()))
for k, v in out.items():
    print(f"   {k:5s} {tuple(v.shape)}")

feats = m.backbone(x)                      # what the backbone hands the head
for k, v in feats.items():
    print(f"   backbone[{k}] {tuple(v.shape)}  -> output stride {128 // v.shape[-1]}")
Output
deeplabv3_resnet50 parameters: 42,004,074
output keys: ['out', 'aux']
   out   (1, 21, 128, 128)
   aux   (1, 21, 128, 128)
   backbone[aux] (1, 1024, 16, 16)  -> output stride 8
   backbone[out] (1, 2048, 16, 16)  -> output stride 8

weights=None, weights_backbone=None builds the architecture without downloading anything, which is what you want when you are inspecting shapes. For real predictions, pass weights=DeepLabV3_ResNet50_Weights.DEFAULT, which downloads roughly 160 MB of ImageNet-and-COCO-trained weights.

Note the output is full resolution while the backbone stops at stride 8. The final step is one bilinear upsample by a factor of 8, which is DeepLabv3 exactly. DeepLabv3+ replaces that single jump with a small decoder that first goes to stride 4 and merges a low-level feature map.

Common mistakes

Forgetting padding = dilation. With dilation=6 and no padding change, a 3x3 convolution eats 12 pixels off each dimension. The layer runs, the sizes drift, and a concatenation fails four layers later.

Stacking identical dilation rates. That is the comb in the output above. Alternate rates, or use ASPP.

Using output stride 8 in production without measuring. It roughly quadruples the cost of the deep stages against stride 16, for a gain that is often under one mIoU point. Measure both.

Dilating an early layer. Atrous belongs in the deep stages where the receptive field needs to be large. Applied at stage 1, it skips exactly the fine texture the early layers exist to capture.

Assuming ASPP rates transfer. Rates 6, 12 and 18 were tuned for output stride 16. At stride 8, DeepLabv3 doubles them, because the same rate now covers half as much of the original image.

Try it yourself

Add (1, 3, 9) and (3, 6, 12) to the rate list in the first script. Predict the span before running, using the rule that stacked $3\times3$ layers give a span of one plus twice the sum of the rates. Then check which rate sets leave gaps.

What to learn next

Researcher — Mathematics and papers.

Definition

For a 1D signal $x$, filter $w$ of length $K$ and rate $r$:

$$ y[i] = \sum_{k=0}^{K-1} x[i + r \cdot k] \cdot w[k] $$

With $r = 1$ this is standard convolution. The effective kernel size is $k_e = r(K-1) + 1$, while the parameter count stays $K$. In 2D the same rate applies along both axes unless specified separately.

Output size follows the general formula:

$$ H_{\text{out}} = \left\lfloor \frac{H_{\text{in}} + 2P - r(K-1) - 1}{S} \right\rfloor + 1 $$

so $P = r(K-1)/2$ preserves size at stride 1, giving $P = r$ for $K = 3$.

Receptive field, stacked

For $L$ layers of kernel size $K$ with rates $r_1 \dots r_L$ and all strides 1:

$$ R_L = 1 + (K-1)\sum_{l=1}^{L} r_l $$

But coverage is not the same as span. Wang et al. (2018), Understanding Convolution for Semantic Segmentation, formalise the gridding problem. A stack whose rates share a common divisor $d > 1$ samples only a lattice of spacing $d$. Their hybrid dilated convolution requires the greatest common divisor of the rate group to be 1. A further coverage condition ensures no gaps at the top of the stack. The (2, 2, 2) versus (1, 2, 4) contrast in the developer section is that theorem, made visible.

The DeepLab lineage

VersionYearContribution
DeepLab v12015Atrous convolution for dense prediction; fully connected CRF as post-processing
DeepLab v22016ASPP; ResNet backbone; CRF retained
DeepLab v32017ASPP with batch norm and an image-level pooling branch; CRF dropped
DeepLab v3+2018Encoder-decoder: a decoder recovering stride 4, plus depthwise separable atrous convolution

Chen et al. (2018), Encoder-Decoder with Atrous Separable Convolution, ECCV, report 89.0 percent mIoU on the PASCAL VOC 2012 test set. Cityscapes reaches 82.1 percent, without post-processing. That "without post-processing" clause matters: v1 and v2 depended on a dense CRF, and v3 removed the need for it by improving the network instead.

The CRF's departure is a useful case study. Krähenbühl and Koltun (2011) gave an efficient mean-field inference for a fully connected pairwise CRF. It added several mIoU points to weak segmentation networks. As the networks improved, the gain shrank to noise, and the cost, complexity and extra hyperparameters stopped being worth it. See cleaning up predicted masks for where post-processing still earns its place.

Cost accounting

Let a stage run at output stride $s$ on an $H \times W$ input with $C$ channels and $3\times3$ kernels. FLOPs scale as $\frac{HW}{s^2} \cdot 9C^2 \cdot 2$. Moving that stage from $s=32$ to $s=8$ multiplies its cost by 16 and leaves its parameter count unchanged.

Atrous separable convolution (v3+) factorises each atrous convolution into a depthwise atrous convolution and a $1\times1$ pointwise convolution. That cuts the cost by roughly $\frac{1}{C_{\text{out}}} + \frac{1}{9}$, close to a ninefold saving at typical widths. See CNN for the depthwise separable derivation.

Memory, not FLOPs, is usually the binding constraint. Activation memory at stride 8 is 16 times that at stride 32 for the affected stages. That is why high-resolution segmentation training is memory-bound, and why gradient checkpointing is common.

Where atrous sits now

Transformer-based segmenters reach global context by attention rather than dilation. SegFormer (Xie et al., 2021) drops dilation entirely, using a hierarchical encoder with efficient self-attention and a lightweight all-MLP decoder. Mask2Former reaches context through its transformer decoder.

Atrous convolution has not disappeared, and two properties keep it in use. It adds no parameters, and its cost is predictable and local, which matters for deployment on fixed-function accelerators where attention is awkward. Most edge segmentation models shipping today still use an ASPP-style head, often the lightweight LR-ASPP variant that pairs with MobileNetV3.

Papers

  • Chen, Papandreou, Kokkinos, Murphy, Yuille, Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs (DeepLab v1), ICLR 2015 — arxiv.org/abs/1412.7062
  • Chen et al., DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs (v2), TPAMI 2018 — arxiv.org/abs/1606.00915
  • Chen, Papandreou, Schroff, Adam, Rethinking Atrous Convolution for Semantic Image Segmentation (v3), 2017 — arxiv.org/abs/1706.05587
  • Chen, Zhu, Papandreou, Schroff, Adam, Encoder-Decoder with Atrous Separable Convolution (v3+), ECCV 2018 — arxiv.org/abs/1802.02611
  • Yu and Koltun, Multi-Scale Context Aggregation by Dilated Convolutions, ICLR 2016 — arxiv.org/abs/1511.07122
  • Wang et al., Understanding Convolution for Semantic Segmentation, WACV 2018 — arxiv.org/abs/1702.08502

What to learn next