RandAugment and AutoAugment

AutoAugment: the search formulation#

Cubuk et al. (2019), AutoAugment: Learning Augmentation Strategies from Data (CVPR), define a policy as 5 sub-policies. Each sub-policy is a pair of operations, and each operation carries a probability and a magnitude. With 16 operations and discretised probability and magnitude, the space is roughly $10^{32}$ policies. An RNN controller trained by reinforcement learning searched it, using validation accuracy of a child model as reward.

The cost was the problem. Searching required training thousands of child models — thousands of GPU-hours per dataset. Lim et al. (2019), Fast AutoAugment, cut this by density matching rather than child-model training; Ho et al. (2019), Population Based Augmentation, learned a schedule rather than a fixed policy.

RandAugment: removing the search#

Cubuk et al. (2020) (CVPR Workshops; also NeurIPS 2020) reduce the space to two integers:

$$ |\mathcal{S}| = K^{N} $$

• $K$ — the number of available operations (14 in the paper).
• $N$ — operations applied per image.
• $M$ — a single magnitude shared by all operations, held constant.

With $N \le 3$ this is small enough for a naive grid search inside the normal training run. The separate proxy-search phase disappears.

Two findings from that paper deserve emphasis:

The magnitude schedule did not matter. Random, constant, linearly increasing and random-with-increasing-bound magnitudes all worked comparably. They chose constant, for having one hyperparameter.

Optimal strength grows with model and dataset size. This is the paper's core critique of proxy-based search. Policies optimised on small models and reduced datasets are systematically too weak for the large-scale setting. The experiment in the developer section is the same effect running the other way. An ImageNet-calibrated default magnitude is too strong for a small clean dataset.

RandAugment matched AutoAugment and Fast AutoAugment on ResNet-50, and exceeded them on larger models. Their strongest setting reached 85.0% ImageNet accuracy.

The uncomfortable follow-up#

Müller and Hutter (2021), TrivialAugment (ICCV oral), removed the remaining knobs. TrivialAugment applies exactly one operation per image, with the operation and its strength both drawn uniformly at random. It is parameter-free, and they report it matching or beating the tuned methods.

That is a strong negative result for the whole search programme. One uniformly sampled operation is competitive with a policy costing thousands of GPU-hours. Most of what the search found was therefore not transferable structure.

AugMix, for a different objective#

Hendrycks et al. (2020) (ICLR) target corruption robustness rather than clean accuracy. AugMix samples several augmentation chains and mixes their outputs convexly. It then adds a Jensen-Shannon consistency loss, so the clean image and its augmented versions embed similarly:

$$ \mathcal{L}{\text{JS}} = \frac{1}{3}\Big( \mathrm{KL}(p{\text{orig}} \Vert \bar{p}) + \mathrm{KL}(p_{\text{aug1}} \Vert \bar{p}) + \mathrm{KL}(p_{\text{aug2}} \Vert \bar{p}) \Big) $$

• $p_{\text{orig}}, p_{\text{aug1}}, p_{\text{aug2}}$ — output distributions for the clean image and two augmented versions.
• $\bar{p}$ — their mean.
• $\mathrm{KL}$ — Kullback-Leibler divergence.

They report substantially improved robustness and uncertainty on corruption benchmarks. v2.AugMix in torchvision implements the image operation only. The consistency loss is yours to add. Without it you have the weaker half of the method. This is a common and quiet omission.

Papers#

• Cubuk et al., AutoAugment, 2019 — arxiv.org/abs/1805.09501
• Cubuk et al., RandAugment, 2020 — arxiv.org/abs/1909.13719
• Müller and Hutter, TrivialAugment, 2021 — arxiv.org/abs/2103.10158
• Hendrycks et al., AugMix, 2020 — arxiv.org/abs/1912.02781
• Lim et al., Fast AutoAugment, 2019 — arxiv.org/abs/1905.00397

What to learn next#

• Building an Albumentations pipeline — the library to reach for when your labels are masks or boxes.
• Geometric augmentations — the transformations these policies are built from.
• Mixup and CutMix — the batch-level half of a modern augmentation recipe.