Positions in images and video

The vision transformer baseline#

Dosovitskiy et al. (2021) use a learned absolute embedding $E_{pos} \in \mathbb{R}^{(N+1) \times d}$ for $N = (H/P)(W/P)$ patches plus one class token, added to the patch projections. The paper's own ablation reports that 2-D-aware position embeddings gave no consistent benefit over the flat 1-D learned table at their scale, and attributes this to the model learning the grid structure from data. Later work at other scales and with other objectives does find 2-D structure useful, so this is not settled in one direction.

Fixed 2-D sine-cosine#

The standard fixed construction, used in MAE and in DETR-family detectors, allocates $d/2$ channels to each axis:

$$ PE(r, c) = \big[\, \mathrm{sincos}{d/2}(r) \;\Vert\; \mathrm{sincos}{d/2}(c) \,\big] $$

where $\mathrm{sincos}_m(p)$ is the standard 1-D construction of width $m$. The concatenation makes the inner product separable:

$$ PE(r_1,c_1)^\top PE(r_2,c_2) = f(r_1 - r_2) + g(c_1 - c_2) $$

so grid distance decomposes additively across axes. That separability is the property the developer output verifies, and it is the reason one step right and one step down score identically.

Chu et al. (2023), CPVT, take a different route: generate positions with a depthwise convolution over the token grid, so the encoding is conditioned on content and resolution-agnostic by construction.

Resolution change#

For a source grid $G \times G$ and target $G' \times G'$, treat $E_{pos}$ minus its class row as a $G \times G$ image with $d$ channels and resample bicubically. This is the standard adaptation for fine-tuning at higher resolution, used in ViT and DeiT.

Beyer et al. (2023), FlexiViT, train a single model to accept multiple patch sizes by resizing the patch-embedding kernel and position table during training, so the interpolation is in-distribution rather than a test-time hack. Dehghani et al. (2023), NaViT, take native-resolution training further with sequence packing, avoiding a fixed grid altogether.

RoPE in vision#

Heo et al. (2024), Rotary Position Embedding for Vision Transformer, study axial 2-D RoPE, where head dimensions are partitioned by axis and rotated by $r$ and $c$ separately, and a mixed variant where learned frequencies span both axes. They report gains on classification, detection and segmentation, and particularly on resolution changes at test time, since RoPE requires no table to interpolate.

Axial 2-D RoPE gives

$$ \langle R_{(r_1,c_1)} q,\; R_{(r_2,c_2)} k \rangle = g\big(q, k,\, r_1 - r_2,\, c_1 - c_2\big) $$

with the relative property holding independently per axis, as the developer output shows numerically. Purely axial rotation cannot represent diagonal relative offsets as a single interaction, which is the motivation for the mixed-frequency variant.

M-RoPE for video and multimodal input#

Wang et al. (2024), Qwen2-VL, partition the rotary dimensions into temporal, height and width groups. Position ids are assigned per modality as tabulated in the developer section. Two consequences follow.

Setting $t = h = w$ for text tokens makes M-RoPE numerically identical to 1-D RoPE on text, so no capability is traded away.

Because the maximum id used by an image or video block is bounded by its grid rather than by its token count, a long visual input consumes fewer positional steps than it does sequence steps. The Qwen2-VL report notes this helps extrapolation to longer videos than were seen in training.

Qwen2.5-VL extends the temporal axis to align with absolute timestamps rather than frame indices, so a model can reason about elapsed time rather than frame count, and handles variable frame rates.

Open questions#

• Axial decomposition assumes row and column are independent. Learned mixed frequencies (Heo et al.) relax that and help. How much structure is genuinely two-dimensional rather than separable is unsettled.
• Native-resolution training (NaViT) and flexible patching (FlexiViT) reduce the interpolation problem but complicate batching. The tradeoff against fixed-grid training is workload dependent.
• For video, the right relative weighting between one frame of time and one patch of space is a hyperparameter with little theory behind it.

Papers#

• Dosovitskiy et al., An Image is Worth 16x16 Words (ViT), ICLR 2021 — arxiv.org/abs/2010.11929
• He et al., Masked Autoencoders Are Scalable Vision Learners (MAE), CVPR 2022 — arxiv.org/abs/2111.06377
• Chu et al., Conditional Positional Encodings for Vision Transformers, ICLR 2023 — arxiv.org/abs/2102.10882
• Beyer et al., FlexiViT: One Model for All Patch Sizes, CVPR 2023 — arxiv.org/abs/2212.08013
• Dehghani et al., Patch n' Pack: NaViT, NeurIPS 2023 — arxiv.org/abs/2307.06304
• Heo et al., Rotary Position Embedding for Vision Transformer, ECCV 2024 — arxiv.org/abs/2403.13298
• Wang et al., Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, 2024 — arxiv.org/abs/2409.12191

What to learn next#

• Vision transformers — the architecture these positions feed.
• Vision language models — where text and image positions have to coexist.
• Rotary position embeddings (RoPE) — the one-dimensional version this extends.