What changing batch size actually changes

The gradient as an estimator#

A mini-batch gradient is an unbiased estimator of the full-batch gradient. For a batch $\mathcal{B}$ of size $B$ drawn uniformly with replacement from a training set of size $N$,

$$ g_{\mathcal{B}} = \frac{1}{B}\sum_{i \in \mathcal{B}} \nabla \ell_i(\theta), \qquad \mathbb{E}[g_{\mathcal{B}}] = \nabla \hat{R}(\theta), \qquad \operatorname{Cov}(g_{\mathcal{B}}) = \frac{\Sigma(\theta)}{B} $$

where $\ell_i$ is the loss on example $i$, $\hat{R}$ the empirical risk, and $\Sigma(\theta)$ the per-example gradient covariance. The $1/B$ in the covariance is the whole story: gradient noise falls as $1/B$ in variance and $1/\sqrt{B}$ in standard deviation, while the signal is unchanged. Sampling without replacement (as real epoch shuffling does) attaches a finite-population factor $(N-B)/(N-1)$, negligible when $B \ll N$.

Temperature: what the pair $(\eta, B)$ actually sets#

The continuous-time view treats SGD as a stochastic differential equation whose stationary distribution depends on $\eta$ and $B$ only through the ratio $\eta/B$, often called the noise scale or temperature (Jastrzębski et al., 2017, Three Factors Influencing Minima in SGD; Smith and Le, 2018, A Bayesian Perspective on Generalization and SGD, ICLR). This is the theoretical basis of the linear scaling rule: multiply $B$ by $k$, multiply $\eta$ by $k$, and the trajectory statistics are approximately preserved.

Goyal et al. (2017), Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour, is the practical demonstration — batch 8,192 with linear scaling plus a gradual warmup matches small-batch accuracy, where the warmup exists precisely because the linear rule breaks in the first epochs when curvature is changing fast. You, Gitman and Ginsburg (2017), LARS, and You et al. (2020), LAMB, push further by making the effective rate layer-wise, which is how batch sizes in the tens of thousands became trainable at all. The $\sqrt{k}$ alternative arises naturally for adaptive optimisers, where the preconditioner already absorbs part of the scaling.

Diminishing returns, and where they bite#

Shallue et al. (2019), Measuring the Effects of Data Parallelism on Neural Network Training (JMLR), is the definitive empirical study: across models and datasets, steps-to-target-accuracy falls roughly linearly with batch size up to a workload-dependent knee, after which extra batch buys almost nothing. McCandlish et al. (2018), An Empirical Model of Large-Batch Training, give a predictor for that knee — the gradient noise scale

$$ B_{\text{noise}} = \frac{\operatorname{tr}(\Sigma)}{|\nabla \hat{R}|^2} $$

the ratio of gradient variance to squared gradient norm. Below $B_{\text{noise}}$, doubling the batch roughly halves the steps needed; far above it, steps barely improve and you are paying compute for a smoother chart. The quantity is estimable during training from the variance of gradients across workers or micro-batches, and it grows over the course of training — which is the principled argument for batch-size schedules rather than one fixed value.

The generalisation-gap debate#

Keskar et al. (2017), On Large-Batch Training for Deep Learning (ICLR), reported that large-batch training converges to sharper minima and generalises worse, proposing sharpness as the mechanism. The claim was substantially qualified afterwards: Hoffer, Hubara and Soudry (2017), Train Longer, Generalize Better (NeurIPS), showed much of the gap disappears when large-batch runs are given a matched number of updates rather than epochs — the same axis error this lesson's last two columns demonstrate at toy scale. Dinh et al. (2017) further showed sharpness as usually defined is not reparameterisation-invariant, so a sharp minimum can be made flat without changing the function. The current consensus is narrow but useful: with the learning rate rescaled, an adequate warmup and a matched update budget, large-batch training usually matches small-batch quality up to the noise-scale knee, and the residual differences are regularisation effects rather than a law about minima.

For the curve-reading practitioner, three invariants survive all of the above: plotted jitter scales as $1/\sqrt{B}$; the epoch axis is not comparable across batch sizes; and $\eta$ and $B$ must move together or the experiment measures neither.

What to learn next#

• Finding a learning rate — the knob that must move whenever batch size does.
• Gradient accumulation — keeping a large effective batch on a small GPU.
• How smoothing and log scales mislead you — the other way a curve's shape lies about the model.