Active learning for labelling

The framing#

Pool-based active learning starts with three things. A small labelled set $\mathcal{L}$, a large unlabelled pool $\mathcal{U}$, and a budget $B$. Choose a subset $\mathcal{S} \subset \mathcal{U}$ with $|\mathcal{S}| = B$ to be labelled. The goal is minimising the risk of the model trained on $\mathcal{L} \cup \mathcal{S}$. The problem is combinatorial, and every practical method is a greedy heuristic on a surrogate objective.

Acquisition functions#

Uncertainty family. For predicted distribution $p(y \mid \mathbf{x})$:

$$ \text{least confident:} \quad \arg\max_{\mathbf{x}} \; 1 - \max_y p(y \mid \mathbf{x}) $$ $$ \text{margin:} \quad \arg\min_{\mathbf{x}} \; p(\hat{y}_1 \mid \mathbf{x}) - p(\hat{y}2 \mid \mathbf{x}) $$ $$ \text{entropy:} \quad \arg\max{\mathbf{x}} \; -\sum_y p(y \mid \mathbf{x}) \log p(y \mid \mathbf{x}) $$

$\hat{y}_1, \hat{y}_2$ are the two highest-probability classes. The three coincide for binary problems and diverge as the number of classes grows. Margin targets the decision boundary between the two contending classes. Entropy rewards mass spread over many classes. For large label spaces that selects images the model has no purchase on.

Disagreement family. BALD (Houlsby et al., 2011) maximises the mutual information between the label and the model parameters:

$$ \mathbb{I}[y, \boldsymbol{\theta} \mid \mathbf{x}, \mathcal{D}] = \mathbb{H}\big[\mathbb{E}{p(\boldsymbol{\theta}|\mathcal{D})}[p(y \mid \mathbf{x}, \boldsymbol{\theta})]\big] - \mathbb{E}{p(\boldsymbol{\theta}|\mathcal{D})}\big[\mathbb{H}[p(y \mid \mathbf{x}, \boldsymbol{\theta})]\big] $$

This isolates epistemic uncertainty, reducible by more data, from aleatoric uncertainty, inherent noise that no label will fix. That distinction is the theoretical case for BALD over entropy. A genuinely ambiguous image has high entropy and low mutual information. Labelling it teaches nothing. Gal et al. (2017) implement it with MC dropout; Kirsch et al. (2019), BatchBALD, extend it to batches by accounting for the joint mutual information. That is expensive, and it is the correct treatment of redundancy.

Diversity family. Core-Set (Sener and Savarese, 2018) casts selection as a $k$-centre problem in feature space. The covering radius bounds the generalisation gap. Purely geometric, with no reference to the model's predictions.

Hybrid. BADGE (Ash et al., 2020) computes the gradient of the loss with respect to the final layer. The predicted label acts as a pseudo-label. Its magnitude encodes uncertainty and its direction encodes the class involved. Running k-means++ seeding on those embeddings selects points that are uncertain and mutually diverse. No hyperparameter balances the two.

What the careful comparisons found#

The literature contains a large gap between reported gains and reproducible ones.

Munjal et al., Towards Robust and Reproducible Active Learning Using Neural Networks, use a matched, well-tuned training pipeline. The differences between published acquisition functions then shrink dramatically. Several fail to beat random sampling with statistical significance. Variance across random seeds exceeds the reported differences between methods.

Effective evaluations of deep active learning on image classification report the same pattern, with more nuance. BADGE outperforms entropy sampling without data augmentation. It loses that advantage once augmentation is applied. Feature-diversity methods such as Core-Set perform barely better than random on class-imbalanced problems. Gradient-based methods with k-centre are generally the strongest, with entropy lagging early.

The methodological requirements that follow are not optional:

• Identical training recipe, hyperparameters and augmentation across all strategies.
• Multiple seeds with variance reported, since single-run curves are uninterpretable.
• The same random initial labelled set for every strategy.
• A random baseline on the same plot, always.
• A fixed, randomly drawn test set that is never part of the pool.

Failure modes with named causes#

Cold start. With a very small $\mathcal{L}$ the model is uncalibrated and its uncertainty is uninformative. Visible in the developer table as entropy losing to random for three rounds.

Sampling bias. Actively acquired data is not i.i.d. from the input distribution, so the labelled set becomes systematically unrepresentative. This breaks any downstream use that assumes representativeness, including estimating class priors and calibration.

Batch redundancy. Top-$B$ by any pointwise score selects correlated points. BatchBALD and BADGE address it in principle; clustering the top candidates addresses it cheaply.

Model-strategy coupling. A dataset acquired under model $A$ is optimised for model $A$'s uncertainty landscape. Changing architecture later can lose the advantage entirely, and this is rarely reported.

Outlier attraction. Uncertainty scores are maximised by corrupt, blank or out-of-distribution images. In production pools this is not a corner case; filter the pool first.

When it is worth it#

Active learning helps most when labelling is expensive relative to compute. It also wants a large, highly redundant pool and a cheaply retrained model. It helps least when labels are cheap, the pool is small, or classes are balanced. Strong augmentation and self-supervised pretraining also hurt it. Both reduce the marginal value of an extra label.

Given the reproducibility findings, the defensible engineering position is a small experiment. Run a random-versus-strategy comparison on your own data for a few rounds. Adopt the strategy only if it wins by more than the seed variance.

References#

• Settles, Active Learning Literature Survey, 2009
• Houlsby et al., Bayesian Active Learning for Classification and Preference Learning (BALD), 2011 — arxiv.org/abs/1112.5745
• Gal et al., Deep Bayesian Active Learning with Image Data, 2017 — arxiv.org/abs/1703.02910
• Sener and Savarese, Active Learning for Convolutional Neural Networks: A Core-Set Approach, 2018 — arxiv.org/abs/1708.00489
• Kirsch et al., BatchBALD, 2019 — arxiv.org/abs/1906.08158
• Ash et al., Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds (BADGE), 2020 — arxiv.org/abs/1906.03671
• Munjal et al., Towards Robust and Reproducible Active Learning Using Neural Networks, 2020 — arxiv.org/abs/2002.09564
• Effective Evaluation of Deep Active Learning on Image Classification Tasks, 2021 — arxiv.org/abs/2106.15324

What to learn next#

• Model-assisted labelling — using the model to draw the labels, not only to choose them.
• Finding label errors in an image dataset — the same probabilities, used to audit instead of to select.
• Data labelling — the wider practice this optimises.