Action recognition in practice

Setup#

pip install torch torchvision

Run against torch 2.5.1 and torchvision 0.20.1 on CPU, Python 3.10. The torchvision video model API has been stable since 0.13; the current stable release at the time of writing is the 0.28 line, and these calls are unchanged.

What a video classifier actually is#

import torch
import torchvision
from torchvision.models.video import r3d_18, R3D_18_Weights

print("torch", torch.__version__, "| torchvision", torchvision.__version__)

# weights=None keeps this offline: random weights, real architecture, real shapes.
model = r3d_18(weights=None, num_classes=400).eval()
clip = torch.zeros(1, 3, 16, 112, 112)      # (N, C, T, H, W)

with torch.no_grad():
    logits = model(clip)
print("\ninput :", tuple(clip.shape))
print("logits:", tuple(logits.shape))
print("parameters:", f"{sum(p.numel() for p in model.parameters()):,}")

# The preprocessing the pretrained weights expect, read from the weights metadata
# rather than copied from a blog post. No download happens for this.
meta = R3D_18_Weights.KINETICS400_V1
print("\npretrained preprocessing:", meta.transforms())
print("classes:", len(meta.meta["categories"]),
      "| first three:", meta.meta["categories"][:3])

torch 2.5.1+cu121 | torchvision 0.20.1+cu121

input : (1, 3, 16, 112, 112)
logits: (1, 400)
parameters: 33,371,472

pretrained preprocessing: VideoClassification(
    crop_size=[112, 112]
    resize_size=[128, 171]
    mean=[0.43216, 0.394666, 0.37645]
    std=[0.22803, 0.22145, 0.216989]
    interpolation=InterpolationMode.BILINEAR
)
classes: 400 | first three: ['abseiling', 'air drumming', 'answering questions']

Read the preprocessing block carefully, because it is the single most common source of silent accuracy loss.

resize_size=[128, 171] is not square. Kinetics videos are widescreen. The weights expect a resize to 128 by 171 followed by a 112 by 112 crop. Resizing to a square first distorts every frame and costs you accuracy with no error message.

Those mean and standard deviation values are Kinetics statistics, not ImageNet ones. ImageNet's familiar [0.485, 0.456, 0.406] is wrong here. Use meta.transforms() rather than typing numbers from memory.

weights=None gives you the architecture with no download. Use it for shape debugging. For real predictions, pass weights=R3D_18_Weights.DEFAULT, which fetches roughly 130 MB once and caches it.

Available models in torchvision.models.video: r3d_18, mc3_18, r2plus1d_18, s3d, mvit_v1_b, mvit_v2_s, swin3d_t, swin3d_s, swin3d_b. The ResNet-style ones are 112-pixel models; the transformer ones are 224-pixel models with different preprocessing. Always read .transforms() for whichever you pick.

Why one clip is not enough#

The model is fixed. What changes is how many times you ask it.

import numpy as np

CLASSES = ["pouring tea", "washing hands", "chopping onion", "clapping"]
rng = np.random.default_rng(7)

# Pretend the model scored 10 clips cut from the same video. The truth is class 0.
true = 0
logits = rng.normal(0, 1.6, size=(10, len(CLASSES)))
logits[:, true] += 1.0                     # the right class is favoured, but only slightly


def softmax(z):
    e = np.exp(z - z.max(axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)


probs = softmax(logits)
per_clip = probs.argmax(axis=1)
print("per-clip predictions:", [CLASSES[i] for i in per_clip])
print(f"single-clip accuracy: {(per_clip == true).mean():.2f}")

print("\naggregated over all 10 clips")
print(f"  mean of softmax  -> {CLASSES[probs.mean(0).argmax()]}")
print(f"  mean of logits   -> {CLASSES[logits.mean(0).argmax()]}")
print(f"  majority vote    -> {CLASSES[np.bincount(per_clip, minlength=4).argmax()]}")
np.set_printoptions(precision=3, suppress=True)
print("\nmean softmax per class:", probs.mean(0))

per-clip predictions: ['pouring tea', 'clapping', 'chopping onion', 'pouring tea', 'washing hands', 'clapping', 'pouring tea', 'pouring tea', 'chopping onion', 'washing hands']
single-clip accuracy: 0.40

aggregated over all 10 clips
  mean of softmax  -> pouring tea
  mean of logits   -> pouring tea
  majority vote    -> pouring tea

mean softmax per class: [0.343 0.219 0.175 0.263]

Four of ten clips were right on their own. All three aggregation rules got the video right.

The mean probability for the correct class is 0.343 against 0.263 for the runner-up. That is a thin margin, and it is realistic. Aggregation buys you a few points on a benchmark; it does not turn a weak model into a strong one.

Which rule to use, in practice:

• Mean of softmax is the standard and the safest default. Each clip contributes a bounded amount.
• Mean of logits lets one very confident clip dominate. Better when the action occurs in a short window and most clips are uninformative.
• Majority vote discards confidence entirely. It behaves poorly with many classes, because votes scatter.

The full pipeline, in order#

import torch
from torchvision.models.video import r3d_18, R3D_18_Weights

weights = R3D_18_Weights.DEFAULT      # downloads on first use
model = r3d_18(weights=weights).eval()
preprocess = weights.transforms()

# frames: a uint8 tensor (T, C, H, W) decoded from your video, RGB, full resolution.
# Sample several clips of 16 frames each, spread across the video.
def predict(frames, clip_len=16, n_clips=5):
    starts = torch.linspace(0, max(len(frames) - clip_len, 0), n_clips).long()
    batch = torch.stack([preprocess(frames[s:s + clip_len]) for s in starts])
    with torch.no_grad():
        probs = model(batch).softmax(-1)          # (n_clips, 400)
    mean = probs.mean(0)
    top = mean.topk(3)
    return [(weights.meta["categories"][i], float(p)) for p, i in zip(*top)]

No output block, deliberately. The result depends entirely on which video you feed it, and inventing a plausible-looking prediction would be dishonest.

Note preprocess(frames[s:s + clip_len]) operates on a whole clip. The transform applies the same resize and crop to every frame in it, which is what keeps motion intact. Cropping each frame independently destroys the motion the model is looking for.

Fine-tuning on your own classes#

The realistic project. Replace the head, freeze most of the backbone, train briefly.

import torch.nn as nn
from torchvision.models.video import r3d_18, R3D_18_Weights

model = r3d_18(weights=R3D_18_Weights.DEFAULT)
for p in model.parameters():
    p.requires_grad = False               # freeze the pretrained backbone
model.fc = nn.Linear(model.fc.in_features, 5)   # 5 classes of your own, trainable

trainable = [n for n, p in model.named_parameters() if p.requires_grad]
print("trainable parameters:", trainable)

trainable parameters: ['fc.weight', 'fc.bias']

That runs after the checkpoint downloads. Start here, measure, then unfreeze layer4 if you need more. Unfreezing everything at once on a few thousand clips overfits — the same pattern as in transfer learning in PyTorch.

Fine-tuning a 3D model on a few thousand clips is feasible on one mid-range GPU in hours. Training from scratch on Kinetics is not; the published models used many GPUs for days, and that is stated plainly in the papers.

Common mistakes#

Normalising with ImageNet statistics. See the output above. Kinetics models have their own.

Applying random crops per frame during training. Choose the augmentation once per clip. Per-frame randomness is a synthetic camera shake that destroys the motion signal.

Comparing your accuracy against a published number under a different view protocol. A paper's 10 x 3 result is 2 to 4 points above its own 1 x 1 result on Kinetics-400. State your protocol whenever you state a number.

Splitting train and test by clip instead of by source video. Two clips from the same video share background, lighting and person. Splitting by clip leaks, and your reported accuracy is fiction — see train-test split for the general pattern.

Reporting Kinetics-style accuracy as evidence of temporal understanding. Test on Something-Something v2 if that is the claim you want to make.

Downloading Kinetics before checking you need it. It is hundreds of thousands of clips and the download links rot. Fine-tune from a checkpoint on your own data first.

Try it yourself#

Modify the aggregation script so only two of the ten clips actually contain the action, with the other eight scoring near-uniformly. Compare mean-of-softmax against mean-of-logits and against taking the maximum over clips. Then work out which real-world situation each rule suits.

What to learn next#

• Video object segmentation — from naming a clip to outlining a thing in it.
• Model evaluation — the metrics behind these accuracy numbers.
• Transfer learning in PyTorch — the fine-tuning recipe in full.