Linear discriminant analysis

Setup#

pip install scikit-learn

Outputs verified with scikit-learn 1.7.2 on CPU.

Three flowers, four measurements, two directions#

from sklearn.datasets import load_iris
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.model_selection import cross_val_score

X, y = load_iris(return_X_y=True)    # ships inside scikit-learn, no download

lda = LinearDiscriminantAnalysis()
acc = cross_val_score(lda, X, y, cv=5).mean()
print(f"5-fold accuracy: {acc:.3f}")

proj = lda.fit(X, y).transform(X)    # 4 measurements squeezed to 2 axes
print("projected shape:", proj.shape)
print("class separation carried by each axis:", lda.explained_variance_ratio_.round(3))
for cls in (0, 1, 2):
    print(f"class {cls} mean on axis 1: {proj[y == cls, 0].mean():6.2f}")

5-fold accuracy: 0.980
projected shape: (150, 2)
class separation carried by each axis: [0.991 0.009]
class 0 mean on axis 1:   7.61
class 1 mean on axis 1:  -1.83
class 2 mean on axis 1:  -5.78

The walkthrough#

98% accuracy from a model with no knobs. No learning rate, no tree depth, no C. LDA fits class averages and spreads in closed form — one shot, no iterations. As a fast, honest baseline before anything fancier, it is hard to beat.

Why 2 axes from 4 measurements? With 3 classes, all the between-average geometry lives in a flat sheet through 3 points. LDA can produce at most classes - 1 directions — here 2, whatever the number of input features. This cap is structural, not a setting.

explained_variance_ratio_ reads [0.991, 0.009]. The first direction carries 99% of the separation; the second is almost decorative. For plotting, one axis nearly suffices — and the class means on it confirm the story: 7.61, -1.83, -5.78. Setosa sits far from the other two, versicolor and virginica are neighbours. Anyone who has plotted iris knows that picture; LDA found it from the ratio alone.

transform vs predict. transform gives coordinates in the discriminant directions — LDA as compressor, a supervised cousin of PCA. predict classifies. Same fitted object, two products.

Common mistakes#

Using PCA when labels exist and separation is the goal. PCA finds directions of biggest overall spread, labels unseen; the most spread-out direction can be exactly the one where classes overlap. If you have labels and want separation, LDA is the tool built for the question.

Expecting more than classes - 1 components. LinearDiscriminantAnalysis(n_components=3) with 3 classes raises ValueError. The ceiling is baked into the geometry.

Ignoring the shared-spread assumption. LDA assumes every class scatters in roughly the same shape and size. When one class is much more spread out, its boundary gets misplaced — QuadraticDiscriminantAnalysis drops the assumption at the cost of many more parameters.

Many features, few samples. With more features than samples, the within-class spread estimate becomes singular and everything gets shaky. Use LinearDiscriminantAnalysis(solver="lsqr", shrinkage="auto") — regularised spread estimation designed for exactly this regime.

Try it yourself#

Run PCA(n_components=2) on the same iris data and print each class's mean on PCA's first axis. Compare with LDA's means above — which axis separates versicolor from virginica better? Then try shrinkage="auto" with solver="lsqr" and check whether accuracy moves.

What to learn next#

• Statistics — means, spreads and the language LDA is written in.
• Naive Bayes — a different generative classifier, built on counting.
• Logistic regression — the discriminative model sharing LDA's linear boundary.