Swin transformer

The construction#

Liu, Lin, Cao, Hu, Wei, Zhang, Lin and Guo (2021), Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, ICCV, arxiv.org/abs/2103.14030.

Global multi-head self-attention over $h \times w$ tokens of dimension $C$ costs

$$ \Omega(\text{MSA}) = 4 h w C^2 + 2 (hw)^2 C $$

while window-based attention with $M \times M$ windows costs

$$ \Omega(\text{W-MSA}) = 4 h w C^2 + 2 M^2 h w C . $$

The first term, the projections, is identical. The second term is the difference: quadratic in $hw$ against linear in $hw$ with a constant of $M^2$. With $M = 7$ fixed, cost is linear in image area.

Consecutive blocks alternate W-MSA and SW-MSA, the shifted variant, displacing the window grid by $\lfloor M/2 \rfloor$. The efficient implementation is cyclic shift by torch.roll plus a mask that blocks attention between tokens that were not spatially adjacent before the roll, which avoids the padding that a naive implementation would need.

Attention within a window uses a learned relative position bias $B$:

$$ \mathrm{Attention}(Q,K,V) = \mathrm{SoftMax}!\left(\frac{QK^{\top}}{\sqrt{d}} + B\right) V $$

$B$ is indexed from a table of size $(2M-1) \times (2M-1)$, covering every relative offset inside a window. The paper reports this bias as worth a substantial margin over absolute position embeddings, and over no position information at all.

Patch merging concatenates each $2 \times 2$ group of neighbouring tokens along the channel axis, giving $4C$, then applies a linear layer to $2C$. Four stages give strides 4, 8, 16 and 32, matching the ResNet hierarchy.

Reported results: 87.3% ImageNet-1K top-1, 58.7 box AP and 51.1 mask AP on COCO test-dev, and 53.5 mIoU on ADE20K val, with margins of +2.7 box AP, +2.6 mask AP and +3.2 mIoU over the previous state of the art.

What the design is trading#

The shifted window scheme reintroduces two priors that a plain ViT discards: locality, since attention is bounded, and hierarchy, since resolution decreases through stages. Both are the priors that convolution has built in. Swin's position in the literature is therefore a hybrid, not a pure transformer, and the ConvNeXt paper reads it that way explicitly, treating Swin as the target to match with convolutions alone.

The receptive field grows by roughly $M/2$ tokens per shifted layer plus the doubling at each merge, so it is bounded and grows with depth exactly as in a CNN, rather than being global from layer one.

Swin V2#

Liu et al. (2021), Swin Transformer V2: Scaling Up Capacity and Resolution, arxiv.org/abs/2111.09883, scale to 3 billion parameters and images up to 1536x1536 with three changes:

1. Residual post-normalisation with cosine attention. Pre-norm transformers accumulate activation amplitude across depth, which destabilises very large models. Moving the norm after the residual branch, and replacing the dot product with a scaled cosine similarity, bounds the attention logits regardless of activation magnitude.
2. Log-spaced continuous position bias. The bias table is replaced by a small network taking log-spaced relative coordinates, so a model trained at one window size transfers to another by evaluating the network rather than interpolating a table.
3. SimMIM self-supervised pretraining, to reduce the labelled data required at that scale.

Point 1 is the transferable idea. Attention logit growth with scale is a general failure mode, not a Swin-specific one.

Where it stands now#

Swin remains a strong dense-prediction backbone and the reference point for windowed attention. Three qualifications are worth stating.

• At classification scale with matched training, ConvNeXt matches or exceeds it at equal FLOPs, and is faster in wall-clock terms on most hardware.
• Plain ViT backbones became competitive for detection once Li et al. (2022), Exploring Plain Vision Transformer Backbones for Object Detection (ViTDet), showed that a simple feature pyramid built from a single-scale ViT works, removing the assumption that hierarchy must be built into the backbone.
• Attention implementations have improved enough that the practical cost argument has narrowed for moderate resolutions. The argument still holds at large resolutions, where the quadratic term dominates.

The durable lesson is structural: bounding an operator's context and then moving the bounds is a general way to make a quadratic mechanism linear. The same pattern appears in sliding-window and block-sparse attention for language models.

What to learn next#

• Contrastive learning for images — training these backbones without labels.
• Vision transformers — the unwindowed original.
• Attention — the operation being bounded here.