3D convolutions for video

The operation#

For input $X$ with $C_{\text{in}}$ channels and kernel $K$ of size $k_t \times k_h \times k_w$:

$$ Y[c_o, t, i, j] = b[c_o] + \sum_{c=0}^{C_{\text{in}}-1} \sum_{p=0}^{k_t-1} \sum_{m=0}^{k_h-1} \sum_{n=0}^{k_w-1} X[c,\, t + p,\, i + m,\, j + n] \cdot K[c_o, c, p, m, n] $$

Where $c_o$ indexes the output channel, $t$ the temporal position, and $(i, j)$ the spatial position. Compared with 2D convolution, the only change is the extra sum over $p$.

Parameters: $C_{\text{out}} C_{\text{in}} k_t k_h k_w + C_{\text{out}}$, a factor $k_t$ above the 2D case.

FLOPs: $\approx 2 \cdot T_{\text{out}} H_{\text{out}} W_{\text{out}} \cdot C_{\text{out}} C_{\text{in}} k_t k_h k_w$, a factor $k_t \cdot T_{\text{out}}$ above 2D convolution on a single frame.

Activation memory grows linearly in $T$, which is why clip length is usually the first hyperparameter sacrificed.

Factorisations#

Full 3D convolution is rarely used unfactorised in modern architectures. Three decompositions dominate:

R(2+1)D (Tran et al., CVPR 2018) replaces $k_t \times k_h \times k_w$ with $1 \times k_h \times k_w$ followed by $k_t \times 1 \times 1$, with a hidden width

$$ M_i = \left\lfloor \frac{k_t k_h k_w \, C_{i-1} C_i}{k_h k_w \, C_{i-1} + k_t \, C_i} \right\rfloor $$

chosen so parameter count matches the unfactorised layer. The gain is the additional non-linearity between the two, plus a measurably lower training error — the paper's central empirical claim is that R(2+1)D is easier to optimise, not smaller.

P3D (Qiu et al., ICCV 2017) explores three wiring variants of the same split (serial, parallel, and a residual hybrid) inside a ResNet bottleneck.

S3D (Xie et al., ECCV 2018) applies the separation selectively: 2D convolutions in early layers, separable 3D in later layers. It reports higher accuracy than I3D at lower cost, and gives the useful finding that temporal modelling is more valuable in late layers than early ones. Early layers are learning edges, which do not need a temporal extent.

Architectural lineage#

X3D is the one most worth reading if compute is your constraint. It starts from a tiny 2D image classifier and expands one axis at a time — temporal length, frame rate, spatial resolution, width, bottleneck width, depth — greedily selecting the expansion with the best accuracy-per-FLOP at each step. The result reaches I3D-level accuracy at roughly a fifth of the multiply-adds.

Inflation, formally#

Given 2D weights $W^{2D} \in \mathbb{R}^{C_o \times C_i \times k_h \times k_w}$, the inflated kernel is

$$ W^{3D}[c_o, c, p, m, n] = \frac{1}{k_t} W^{2D}[c_o, c, m, n] \quad \forall p $$

The bootstrapping argument: feed a clip made of $k_t$ copies of one image. The inflated 3D layer's response equals the 2D layer's response on that image, so the whole network's behaviour is preserved. The division by $k_t$ is what makes this exact, and omitting it scales activations by $k_t$ at every layer.

I3D reported that inflated Kinetics-pretrained models transferred to UCF-101 and HMDB-51 far better than any from-scratch 3D model, which is what established Kinetics as the standard video pretraining corpus.

Where 3D convolutions stand now#

Transformers overtook 3D CNNs on the large benchmarks from 2021 onward — see video transformers. The honest comparison is narrower than headline numbers suggest:

• 3D CNNs carry locality and translation-equivariance priors, so they remain stronger in low-data regimes.
• Their cost is predictable and their memory profile is friendlier to edge deployment.
• X3D-class models still hold competitive accuracy-per-FLOP, which is the metric that matters on device.

Convolutional and attention-based video models have also converged in practice. Video Swin restricts attention to local 3D windows, which is structurally close to a large separable convolution; MViT pools keys and values, which is a stride. The design space is one space, not two camps.

References#

• Tran et al., Learning Spatiotemporal Features with 3D Convolutional Networks (C3D), ICCV 2015 — arxiv.org/abs/1412.0767
• Carreira and Zisserman, Quo Vadis, Action Recognition? (I3D), CVPR 2017 — arxiv.org/abs/1705.07750
• Tran et al., A Closer Look at Spatiotemporal Convolutions (R(2+1)D), CVPR 2018 — arxiv.org/abs/1711.11248
• Xie et al., Rethinking Spatiotemporal Feature Learning (S3D), ECCV 2018 — arxiv.org/abs/1712.04851
• Feichtenhofer, X3D: Expanding Architectures for Efficient Video Recognition, CVPR 2020 — arxiv.org/abs/2004.04730

What to learn next#

• SlowFast networks — two pathways reading the same video at different speeds.
• Convolutional neural networks — the 2D filter these cubes are built from.
• Transfer learning in PyTorch — the mechanics behind weight inflation.