Labelling data

Noise models, formally#

Let $Y$ be the true label and $\tilde{Y}$ the observed one. The noise process is characterised by:

$$ \rho_{ij}(x) \;=\; p!\left(\tilde{Y} = j \mid Y = i, X = x\right) $$

Three cases, in increasing difficulty:

Symmetric (uniform) noise. $\rho_{ij}$ is constant in $x$ and equal for all $i \neq j$. Natarajan et al. (2013), Learning with Noisy Labels, NeurIPS, give an unbiased loss correction: for binary labels with rates $\rho_{+1}, \rho_{-1}$ and $\rho_{+1} + \rho_{-1} < 1$,

$$ \tilde{\ell}(t, y) \;=\; \frac{(1 - \rho_{-y})\,\ell(t, y) \;-\; \rho_{y}\,\ell(t, -y)}{1 - \rho_{+1} - \rho_{-1}} $$

Where $\ell$ is the clean loss, $t$ the prediction and $\rho_y$ the flip rate out of class $y$. Minimising $\tilde{\ell}$ on noisy data is equivalent in expectation to minimising $\ell$ on clean data. The Bayes-optimal classifier is unchanged by symmetric noise; only the estimation variance grows.

Asymmetric (class-conditional) noise. $\rho_{ij}$ still constant in $x$ but asymmetric across classes. The optimal decision threshold shifts. Correctable if the noise rates can be estimated, which is the premise of confident learning.

Instance-dependent noise. $\rho_{ij}(x)$ genuinely depends on $x$. This is the strict-rater case: flip probability is high near the decision boundary and near zero away from it. It is not identifiable in general without extra assumptions, and it shifts the Bayes-optimal boundary itself. No reweighting fixes it, because the corrupted conditional is a different function.

The developer experiment is a demonstration of exactly this hierarchy, and of why quoting a single "label noise rate" is close to meaningless.

Confident learning#

Northcutt, Jiang and Chuang (2021), Confident Learning: Estimating Uncertainty in Dataset Labels, JAIR, estimate the joint distribution $p(\tilde{Y}, Y^{*})$ using out-of-sample predicted probabilities and per-class thresholds set to the average self-confidence:

$$ t_j \;=\; \frac{1}{|\mathbf{X}{\tilde{y}=j}|} \sum{x \in \mathbf{X}_{\tilde{y}=j}} \hat{p}(\tilde{y}=j \mid x) $$

Off-diagonal mass in the estimated confident joint identifies candidate mislabelled examples. The method is model-agnostic and requires only calibrated out-of-fold probabilities. It is implemented in cleanlab.

The companion paper, Northcutt, Athalye and Mueller (2021), Pervasive Label Errors in Test Sets, applied it to ten benchmark test sets and found an average 3.4% error rate, with model rankings reversing after correction on several.

Agreement statistics, and their traps#

Cohen's kappa for two raters:

$$ \kappa \;=\; \frac{p_o - p_e}{1 - p_e} $$

Where $p_o$ is observed agreement and $p_e$ is agreement expected by chance under independent marginals. Two known pathologies (Feinstein and Cicchetti, 1990):

1. The prevalence paradox. With a highly skewed class distribution, $p_e \to p_o$ and $\kappa$ collapses towards zero despite high raw agreement.
2. The bias paradox. Two raters with opposite marginal biases can produce a higher $\kappa$ than two raters with matched marginals, because the imbalance lowers $p_e$.

Report $\kappa$ together with $p_o$ and the marginal distributions of each rater. Kappa alone is not interpretable.

Krippendorff's alpha generalises to any number of raters, missing data and ordinal, interval or nominal scales. It is the right default for real annotation pipelines, where not every rater sees every item. Artstein and Poesio (2008), Inter-Coder Agreement for Computational Linguistics, Computational Linguistics, is the definitive practical treatment.

Fleiss' kappa handles multiple raters but assumes a fixed number of ratings per item and different raters per item. It answers a different question from Cohen's and the two are not comparable.

Aggregating multiple raters#

Majority vote discards information about rater reliability. Dawid and Skene (1979), Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, Applied Statistics, model each rater with a confusion matrix and infer true labels and rater reliabilities jointly by expectation-maximisation. It remains a strong baseline four decades on, and typically beats majority vote when rater quality is uneven.

Extensions: Whitehill et al. (2009) add per-item difficulty (GLAD); Welinder et al. (2010) add rater bias and competence in a multidimensional model.

Programmatic labelling#

Ratner et al. (2017), Snorkel: Rapid Training Data Creation with Weak Supervision, VLDB, replace hand labels with labelling functions: noisy, correlated heuristics whose accuracies are estimated without ground truth by fitting a generative model to their agreement structure. Output is a probabilistic label used to train a downstream discriminative model.

The honest limitation: the generative model recovers accuracies only when the labelling functions are not all wrong in the same way. Correlated systematic errors across functions are unidentifiable, which is the programmatic version of the strict-rater failure.

Practical protocol#

1. Write the guideline with 20 or more worked examples, weighted towards boundary cases.
2. Double-label 5 to 10% of every batch, indefinitely, and track $\kappa$ per batch.
3. Maintain a gold set with adjudicated answers; measure each rater against it, and estimate a per-rater confusion matrix.
4. Adjudicate disagreements in a recorded session; every adjudication should either resolve an item or amend the guideline.
5. Run confident learning over the finished set before training, and re-inspect the top-ranked suspects by hand.
6. Estimate the model ceiling as the adjudicated agreement rate, and stop tuning once you are near it.

Reading#

• Natarajan et al., Learning with Noisy Labels, NeurIPS 2013.
• Northcutt, Jiang and Chuang, Confident Learning, JAIR 2021 — arxiv.org/abs/1911.00068
• Dawid and Skene, Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm, Applied Statistics 1979.
• Artstein and Poesio, Inter-Coder Agreement for Computational Linguistics, Computational Linguistics 2008.
• Ratner et al., Snorkel, VLDB 2017 — arxiv.org/abs/1711.10160

What to learn next#

• Cleaning data — the mechanical errors that sit beneath the judgement calls.
• Model evaluation — why accuracy alone hid the strict rater's damage.
• Fairness metrics — measuring whether a rater's bias reached particular groups.