Imbalanced data

Imbalance is a threshold problem before it is a learning problem#

For a probabilistic classifier, the Bayes-optimal decision under asymmetric costs is a threshold on the posterior. With cost $c_{01}$ for a false negative and $c_{10}$ for a false positive, predict positive when:

$$ p(y=1 \mid x) \;>\; \tau^{*} \;=\; \frac{c_{10}}{c_{10} + c_{01}} $$

The class prior enters through $p(y=1 \mid x)$, not through $\tau^{*}$. Imbalance therefore does not require a change of algorithm. It requires that you stop using $\tau = 0.5$, which silently asserts equal costs.

Elkan (2001), The Foundations of Cost-Sensitive Learning, IJCAI, gives the correction for training on a resampled prior. If the base rate is changed from $\pi$ to $\pi'$, calibrated probabilities are recovered by:

$$ p(y=1 \mid x) \;=\; \frac{\pi \, p'(y=1\mid x)}{\pi\,p'(y=1\mid x) + (1-\pi)\big(1 - p'(y=1\mid x)\big)} \cdot \frac{1}{Z} $$

with the appropriate normalisation $Z$ from the sampling ratio. For logistic regression, undersampling the majority class by factor $\beta$ shifts only the intercept, by $\log \beta$ — every other coefficient is unchanged in expectation. This is the analytic reason class_weight="balanced" left ROC-AUC and PR-AUC untouched in the developer experiment: for a linear model it is close to an intercept shift.

Where imbalance genuinely becomes a learning problem#

Three regimes where reweighting does more than move a threshold:

1. Absolute scarcity. Not the ratio but the count. Fifty positives is a small-sample estimation problem, and no reweighting adds examples.
2. Non-convex losses and deep networks. With finite optimisation budgets, gradients dominated by the majority class can stall minority-class learning before convergence. Cui et al. (2019), Class-Balanced Loss Based on Effective Number of Samples, CVPR, reweight by $(1-\beta)/(1-\beta^{n_c})$ rather than by $1/n_c$, arguing for diminishing returns from additional samples of the same class.
3. Separable-but-imbalanced margins. Cao et al. (2019), Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS, derive a class-dependent margin $\Delta_j \propto n_j^{-1/4}$ from a generalisation bound, and show gains that threshold tuning cannot reproduce.

Focal loss (Lin et al., 2017, Focal Loss for Dense Object Detection) down-weights easy examples by $(1-p_t)^{\gamma}$. It was designed for extreme foreground-background imbalance in dense detection, roughly 1000:1, where almost every anchor is an easy negative. Applying it to mildly imbalanced tabular data is cargo cult.

Rare-events bias in logistic regression#

King and Zeng (2001), Logistic Regression in Rare Events Data, Political Analysis, show that maximum likelihood underestimates event probabilities when events are rare, with a bias of order $O(1/n)$ that is not negligible at small event counts. The bias-corrected estimator is:

$$ \tilde{\beta} = \hat{\beta} - \text{bias}(\hat{\beta}), \qquad \text{bias}(\hat{\beta}) = (X^\top W X)^{-1} X^\top W \xi $$

with $\xi_i = 0.5 \, Q_{ii}\big[(1+w_1)\hat{\pi}_i - w_1\big]$, $Q$ the hat matrix, $W$ the weight matrix and $w_1$ the sampling weight for events. This matters when the number of positives is in the tens or low hundreds, which is common in fraud and rare-disease work.

Resampling methods, assessed honestly#

• Random oversampling duplicates minority rows. Equivalent to instance reweighting for most losses. Increases overfitting risk on the duplicated points.
• Random undersampling discards majority rows. Fast, and throws away real information; competitive when the majority is enormous and redundant.
• SMOTE (Chawla et al., 2002, JAIR) interpolates linearly between a minority point and one of its $k$ minority neighbours. The synthetic points lie on segments inside the minority convex hull, so it cannot extrapolate to unseen regions. In high dimensions, distances concentrate and neighbour choice becomes near-arbitrary. For categorical or one-hot features, interpolation produces fractional values with no meaning.
• ADASYN, Borderline-SMOTE focus synthesis near the boundary, which also concentrates synthesis near the noisiest points.

Elor and Averbuch-Elor (2022), To SMOTE, or not to SMOTE?, evaluate across many datasets and find that for strong classifiers with properly tuned thresholds, balancing gives no consistent gain in AUC or in cost-sensitive metrics. The gains reported in older papers largely disappear once the threshold is tuned rather than fixed at 0.5.

That is the finding the developer experiment reproduces in miniature.

Metrics#

• Average precision $= \sum_n (R_n - R_{n-1}) P_n$, the step-wise integral of the precision-recall curve. Its baseline is the positive class prevalence, so 0.285 against a prevalence of 0.0122 is roughly a 23-fold lift.
• Precision@k when the review capacity is fixed at $k$ alerts per day. This is usually the honest operational metric and it is rarely reported.
• Expected cost $= c_{10}\,\text{FP} + c_{01}\,\text{FN}$, evaluated over the full threshold sweep. If you can name the two costs, this dominates every other choice.
• Calibration. Resampling and reweighting break calibration by construction. If a downstream system consumes the probability rather than the decision, recalibrate on the original prior with Platt scaling or isotonic regression.

Reading#

• Elkan, The Foundations of Cost-Sensitive Learning, IJCAI 2001.
• King and Zeng, Logistic Regression in Rare Events Data, Political Analysis 2001.
• Chawla et al., SMOTE: Synthetic Minority Over-sampling Technique, JAIR 2002 — arxiv.org/abs/1106.1813
• Davis and Goadrich, The Relationship Between Precision-Recall and ROC Curves, ICML 2006.
• Saito and Rehmsmeier, The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets, PLOS ONE 2015.
• Cao et al., Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019 — arxiv.org/abs/1906.07413

What to learn next#

• Data augmentation — making more examples honestly, when you have too few.
• Model evaluation — precision, recall and the curves between them.
• Anomaly detection — what to do when the rare class has almost no labels at all.