Data augmentation

Augmentation as an invariance prior#

Let $\mathcal{T}$ be a set of transformations $t: \mathcal{X} \to \mathcal{X}$. Augmentation asserts a label-invariance assumption:

$$ p(y \mid x) \;=\; p\big(y \mid t(x)\big) \qquad \forall\, t \in \mathcal{T} $$

Training on augmented data approximates minimisation of the transformation-averaged risk:

$$ \hat{R}{\mathcal{T}}(f) \;=\; \frac{1}{n}\sum{i=1}^{n} \mathbb{E}_{t \sim \mathcal{T}}\Big[ \ell\big(f(t(x_i)),\, y_i\big) \Big] $$

When the invariance assumption holds, this is a form of regularisation that shrinks the effective hypothesis class to functions approximately invariant under $\mathcal{T}$, reducing variance without adding bias.

When the assumption fails — mirroring a digit — the augmented risk is minimised by a function that cannot be correct on the true distribution. The bias term becomes irreducible, and no amount of data or capacity removes it. The experiment above measures exactly this term.

The variance-reduction view#

Chen, Dobriban and Lee (2020), A Group-Theoretic Framework for Data Augmentation, JMLR, formalise augmentation over a group $G$ acting on the input space. Their main result: for the orbit-averaged estimator, augmentation reduces variance and the reduction is expressible through the group's action, while leaving the bias unchanged provided the target function is genuinely $G$-invariant.

Dao et al. (2019), A Kernel Theory of Modern Data Augmentation, ICML, decompose the effect into a kernel term and a variance-regularisation term, giving a first-order approximation of augmentation as an explicit regulariser. Both papers converge on the same practical statement: augmentation encodes a prior, and a wrong prior is a bias you cannot train away.

Learned augmentation policies#

• AutoAugment (Cubuk et al., 2019, CVPR) searches a discrete policy space of 16 operations with magnitude and probability, using reinforcement learning against validation accuracy. Effective and extremely expensive — thousands of GPU-hours for the original search.
• RandAugment (Cubuk et al., 2020) removes the search entirely, reducing the space to two integers: $N$ operations sampled uniformly, each at a single shared magnitude $M$. It matches or exceeds AutoAugment at a search cost of a small grid over two parameters. This is the sensible default.
• TrivialAugment (Müller and Hutter, 2021, ICCV) removes even that, sampling one operation and one magnitude uniformly per image, with no tuning. Competitive with both, which is a pointed result about how much of the earlier gain came from the search.

Mixing-based augmentation#

mixup (Zhang et al., 2018, ICLR) trains on convex combinations of pairs:

$$ \tilde{x} = \lambda x_i + (1-\lambda) x_j, \qquad \tilde{y} = \lambda y_i + (1-\lambda) y_j, \qquad \lambda \sim \text{Beta}(\alpha, \alpha) $$

Note this breaks the label-invariance framing above: the label changes deliberately. It is better read as enforcing linear behaviour between training examples, and it measurably improves calibration and robustness to label noise.

CutMix (Yun et al., 2019, ICCV) pastes a rectangular patch from one image into another and mixes labels in proportion to patch area. Cutout (DeVries and Taylor, 2017) masks a random square, forcing reliance on distributed evidence rather than one discriminative region.

For tabular data, none of these transfer cleanly. mixup between two customers produces a customer who does not exist, and the interpolated categorical columns are meaningless.

Domain-specific notes#

Audio. SpecAugment (Park et al., 2019, Interspeech) applies time warping, frequency masking and time masking directly to the log-mel spectrogram. It produced substantial word-error-rate improvements on LibriSpeech without any change to the model, and it is now standard. See waveforms and spectrograms.

Text. EDA (Wei and Zou, 2019, EMNLP) — synonym replacement, random insertion, swap and deletion — helps most in low-resource settings and its benefit largely vanishes above a few thousand labelled examples. Back-translation (Sennrich et al., 2016, ACL) is stronger and costs a translation pass. Both risk altering meaning: negation, quantities and named entities are the usual casualties.

Self-supervised learning. In SimCLR (Chen et al., 2020, ICML) augmentation is not a regulariser but the entire learning signal — the model is trained to identify which two views came from the same image. Their ablation shows composition matters enormously: random crop plus colour distortion is essential, and either alone is much weaker. When augmentation defines the task, its design decides what the representation becomes invariant to, for better or worse.

Evaluation discipline#

1. Augment the training split only, after splitting.
2. Report the un-augmented baseline in every table. It is frequently the winner and frequently omitted.
3. Ablate each transform individually. Composed policies hide the one that is hurting.
4. Check test-time augmentation separately: averaging predictions over transformed copies of a test input is a different technique with a different cost profile.
5. Where invariance is claimed, measure it directly — feed transformed inputs and report prediction agreement, not only accuracy.

Reading#

• Shorten and Khoshgoftaar, A survey on Image Data Augmentation for Deep Learning, Journal of Big Data 2019.
• Cubuk et al., RandAugment: Practical automated data augmentation with a reduced search space, 2020 — arxiv.org/abs/1909.13719
• Zhang et al., mixup: Beyond Empirical Risk Minimization, ICLR 2018 — arxiv.org/abs/1710.09412
• Chen, Dobriban and Lee, A Group-Theoretic Framework for Data Augmentation, JMLR 2020 — arxiv.org/abs/1907.10905
• Park et al., SpecAugment, Interspeech 2019 — arxiv.org/abs/1904.08779

What to learn next#

• Synthetic data — generating rows from scratch, and what that cannot give you.
• Overfitting and underfitting — the problem augmentation is really treating.
• Image classification — where these transforms do the most work.