Video transformers

Factorisation, formally#

Let a clip give $T$ frames and $S$ spatial patches per frame, so $N = TS$ tokens. Self-attention over all tokens costs

$$ O(N^2 d) = O(T^2 S^2 d) $$

for embedding dimension $d$. The dependence on $T$ is quadratic, which is what makes long clips unreachable.

Divided space-time attention (Bertasius et al., 2021, TimeSformer) applies temporal attention within each spatial position, then spatial attention within each frame:

$$ O(T S^2 d) + O(S T^2 d) = O(TS \cdot (S + T) \cdot d) $$

For $S = 196$ and $T = 8$ this is a 7.7x reduction, matching the developer table. TimeSformer compared five schemes — space-only, joint space-time, divided, sparse local-global, and axial — and found divided attention best on both Kinetics-400 and Something-Something v2. The paper attributes this to divided attention learning separate parameters for the spatial and temporal roles, giving higher modelling capacity than joint attention at lower cost.

ViViT (Arnab et al., 2021) enumerated four models: joint space-time; factorised encoder (a spatial transformer per frame, then a temporal transformer over per-frame representations); factorised self-attention; and factorised dot-product attention. The factorised encoder gives the best complexity-accuracy trade-off, and is effectively late fusion with a learned temporal aggregator.

ViViT also introduced tubelet embedding: a 3D patch of size $t \times h \times w$ instead of a per-frame 2D patch. This fuses temporal information at the tokenisation step and reduces the token count by a factor of $t$. VideoMAE's tubelet_size=2 is this idea, and it is why a 16-frame clip yields 1568 tokens rather than 3136.

Attention with structure#

Two designs reintroduce convolution-like inductive bias rather than removing pairs:

MViT / MViTv2 (Fan et al., ICCV 2021; Li et al., CVPR 2022) build a multiscale feature hierarchy inside the transformer. Pooling attention downsamples keys and values with a strided pooling operator, so resolution drops and channel count rises through the network, exactly as in a CNN. This makes the token count shrink with depth instead of staying constant.

Video Swin (Liu et al., CVPR 2022) restricts attention to non-overlapping local 3D windows, shifting the window partition between blocks so information crosses window boundaries. Attention cost becomes linear in the number of tokens. Structurally this is close to a large separable convolution with content-dependent weights, and the paper argues explicitly that reintroducing locality is why it beats global-attention models at equal compute.

Self-supervised pretraining#

Supervised video pretraining is bounded by label availability. The masked-modelling route removed that bound.

VideoMAE (Tong et al., NeurIPS 2022) masks 90–95% of tubelets and reconstructs the missing pixels. Three findings from the paper:

1. Extremely high masking ratios work on video, higher than on images, because temporal redundancy makes reconstruction too easy at lower ratios.
2. Tube masking — masking the same spatial positions across all frames — is essential. Random per-frame masking lets the model copy from the adjacent frame instead of learning structure.
3. It performs well on datasets of only 3,000 to 4,000 videos with no extra data, and the paper reports data quality mattering more than data quantity.

Reported results with a plain ViT backbone and no extra data: 83.9% on Kinetics-400, 75.3% on Something-Something v2, 90.8% on UCF101, 61.1% on HMDB51.

V-JEPA 2 (Meta, June 2025) predicts in representation space rather than pixel space, trained on over a million hours of video. It reports 77.3% top-1 on Something-Something v2 and 39.7 recall-at-5 on Epic-Kitchens-100 action anticipation, and post-trains into an action-conditioned world model used for zero-shot robot planning. The direction of travel is clear: video pretraining objectives are moving from pixel reconstruction to latent prediction.

What the benchmark numbers do and do not show#

Three cautions when reading video transformer results:

Kinetics rewards scene recognition. Many Kinetics classes are identifiable from a single frame. Strong Kinetics accuracy is weak evidence of temporal modelling. Something-Something v2, whose classes come in direction-reversed pairs, is the discriminating benchmark.

Evaluation protocol is often buried. A 10 x 3 view protocol adds 2 to 4 points of top-1 over 1 x 1 on Kinetics-400 — comparable to the gap between competing architectures. Comparisons across differing protocols are not comparisons.

Pretraining corpus dominates architecture. ViViT and TimeSformer results depend on ImageNet-21k or JFT initialisation. Liu et al. (2022), A ConvNet for the 2020s, made the parallel argument in the image domain: much of the reported transformer advantage came from training recipes rather than architecture. The same caution applies to video, where compute budgets are larger and ablations rarer.

References#

• Bertasius, Wang and Torresani, Is Space-Time Attention All You Need for Video Understanding? (TimeSformer), ICML 2021 — arxiv.org/abs/2102.05095
• Arnab et al., ViViT: A Video Vision Transformer, ICCV 2021 — arxiv.org/abs/2103.15691
• Fan et al., Multiscale Vision Transformers, ICCV 2021 — arxiv.org/abs/2104.11227
• Liu et al., Video Swin Transformer, CVPR 2022 — arxiv.org/abs/2106.13230
• Tong et al., VideoMAE, NeurIPS 2022 — arxiv.org/abs/2203.12602
• Assran et al., V-JEPA 2, 2025 — arxiv.org/abs/2506.09985

What to learn next#

• Action recognition in practice — putting one of these models to work.
• Vision transformers — the image model every video transformer extends.
• Attention — the mechanism itself, at its simplest.