Learn AI Create the Future — with Pranay
Main site

Imbalanced, Multi-class and Multi-label

Class weights

One argument makes mistakes on the rare class cost more during training — often matching SMOTE without inventing a single row of data.

Read these first

On this page 5
  1. Why it exists
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Class weights tell the model that a mistake on the rare class hurts more than a mistake on the common one.

Think of a cricket coach watching a batsman in the nets. Most balls are ordinary, and one in twenty is a yorker the batsman keeps missing. A good coach does not treat every miss equally. The rare, dangerous mistake gets the loud correction.

Why it exists

SMOTE and undersampling both change the data — inventing rows or deleting them. Both feel drastic. What you actually want is to change the model's attention.

Class weights do exactly that. Training is a long loop of "make a guess, get told the mistake, adjust". A class weight is a penalty multiplier attached to each class. Give the rare class a weight of twenty, and every mistake on it stings twenty times harder.

The data stays untouched. Every row is real, and every row is kept. In most libraries this is one argument, not a pipeline step.

How it works

mistake on a common example  →  small nudge to the model
mistake on a rare example    →  BIG nudge to the model
                                (same mistake, bigger correction)

The usual recipe, called balanced weights, sets each class's weight from how rare it is. A class with 5% of the rows gets roughly twenty times the weight of one with 95%. The model then behaves as if it had seen a balanced dataset, without the dataset changing.

A real example you have seen

Your bank's fraud model would rather annoy you with a false "was this you?" SMS than miss a real theft. That preference is a cost decision: a missed fraud costs far more than a wasted SMS. Class weights are how that business preference is pushed into the mathematics of training.

Remember this

  • Class weights make rare-class mistakes cost more during training.
  • The data is untouched — no invented rows, no deleted rows.
  • Attempt this before SMOTE or undersampling. It is one argument.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn

Outputs verified with scikit-learn 1.7.2, CPU only.

One argument, same effect as SMOTE

class_weights.py
import numpy as np
from sklearn.datasets import make_classification
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score, recall_score
from sklearn.model_selection import train_test_split
from sklearn.utils.class_weight import compute_class_weight

X, y = make_classification(n_samples=1000, n_features=6, weights=[0.95, 0.05],
                           class_sep=0.8, random_state=42)
X_tr, X_te, y_tr, y_te = train_test_split(X, y, stratify=y, random_state=42)

# what 'balanced' works out to for this data
w = compute_class_weight("balanced", classes=np.array([0, 1]), y=y_tr)
print("weight per class:", dict(zip([0, 1], w.round(2))))

for name, cw in [("unweighted", None), ("balanced  ", "balanced")]:
    model = LogisticRegression(max_iter=1000, class_weight=cw).fit(X_tr, y_tr)
    pred = model.predict(X_te)
    print(f"{name}: recall={recall_score(y_te, pred):.2f}"
          f"  precision={precision_score(y_te, pred):.2f}")
Output
weight per class: {0: 0.53, 1: 9.62}
unweighted: recall=0.08  precision=1.00
balanced  : recall=0.85  precision=0.20

The walkthrough

The weights are computed, not guessed. balanced uses the rule total ÷ (classes × class count). The rare class (39 of 750 training rows) gets 9.62; the common one gets 0.53.

Compare with the SMOTE lesson: identical scores. Recall 0.85, precision 0.20, on the same data — with no synthetic rows, no extra memory, no pipeline stage. This is why weighting is the recommended first move.

Where it plugs in elsewhere. Most classifiers in scikit-learn accept class_weight, including trees, forests and SVMs. XGBoost calls it scale_pos_weight. PyTorch's CrossEntropyLoss takes a weight tensor. Keras fit takes a class_weight dict. Same idea everywhere.

A dict gives you manual control. class_weight={0: 1, 1: 50} encodes "a missed fraud costs fifty times a false alarm" directly from your business, not from class counts.

Common mistakes

Tuning weights on the test set. The weight is a hyperparameter. Choose it with cross-validation, like any other. Nudging it while staring at test recall is quiet overfitting.

Expecting calibrated probabilities afterwards. A weighted model's predict_proba is deliberately skewed toward the rare class. If you need true probabilities, recalibrate, or keep the plain model and move the decision threshold instead.

Stacking weights on top of SMOTE. Doing both double-corrects, and the model overshoots into false-alarm territory. Pick one lever.

Forgetting the alternative: move the threshold. predict cuts at a score of 0.5 by default. Lowering that cut on an unweighted model often achieves the same recall trade without retraining anything. Weighting changes what the model learns; thresholding changes only the final cut.

Try it yourself

Replace "balanced" with dicts: {0: 1, 1: 3}, {0: 1, 1: 30}, {0: 1, 1: 300}. Watch recall and precision trade places as the weight climbs. Find the weight where precision drops below 0.5.

What to learn next

Researcher — Mathematics and papers.

Weighted loss

For binary labels y_i ∈ {0, 1}, per-class weights w₀ and w₁, and predicted probability p_i, weighted log-loss is:

L = −(1/n) Σ_i [ w₁ y_i log p_i + w₀ (1 − y_i) log(1 − p_i) ]

Where n is the sample count. The gradient contribution of each sample scales linearly with its class weight, so w₁/w₀ = 20 is equivalent in expectation to duplicating each minority sample 20 times — without the memory, and with smoother optimisation (fractional weights are allowed; duplicates are not).

The balanced heuristic sets w_c = n / (K · n_c) for K classes with class counts n_c (King and Zeng, 2001, popularised via scikit-learn). It equalises the total loss mass each class contributes.

The decision-theory view

Cost-sensitive learning theory (Elkan, 2001, The foundations of cost-sensitive learning) shows the optimal decision threshold under costs c_FP and c_FN is p* = c_FP / (c_FP + c_FN). Two consequences:

  1. Reweighting during training and threshold-shifting after training are two implementations of the same target decision rule. For well-specified models they coincide; for misspecified models reweighting also moves the fitted boundary, which can genuinely help.
  2. If you can state costs, you do not need to guess weights: set w₁/w₀ = c_FN/c_FP.

Calibration cost

Weighting distorts posterior estimates exactly as resampling does: the model fits a tilted prior. Recovering calibrated probabilities needs either Elkan's prior-correction formula or post-hoc recalibration (Platt scaling / isotonic regression) on unweighted validation data. Van den Goorbergh et al. (2022) show that for clinical risk models this distortion is often worse than the imbalance being treated — when the deliverable is a probability, prefer an unweighted model plus a moved threshold.

Focal loss — the deep-learning descendant

Lin et al. (2017), Focal loss for dense object detection, generalise weighting from classes to samples: FL(p_t) = −α_t (1 − p_t)^γ log p_t, where p_t is the probability of the true class, α_t a class weight, and γ ≥ 0 down-weights already-easy examples. With γ = 0 it reduces to standard weighted cross-entropy. It became standard in object detection, where background boxes outnumber objects thousands to one.

What to learn next