Overfitting and underfitting

Setup#

pip install scikit-learn numpy

Watching a model memorise#

We generate points along a wave and add noise. Then we fit curves of increasing bendiness. Each one is scored twice — on data it trained on, and on data it never saw.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

rng = np.random.default_rng(0)
x = np.linspace(0.0, 1.0, 20)
y = np.sin(2 * np.pi * x) + rng.normal(0, 0.25, size=20)

# Odd-numbered points are held back so the model never sees them while learning
x_train, y_train = x[0::2].reshape(-1, 1), y[0::2]
x_test, y_test = x[1::2].reshape(-1, 1), y[1::2]

for degree in (1, 3, 12):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(x_train, y_train)
    train_err = mean_squared_error(y_train, model.predict(x_train)) ** 0.5
    test_err = mean_squared_error(y_test, model.predict(x_test)) ** 0.5
    print(f"degree {degree:2d} | train error {train_err:6.3f} | test error {test_err:8.3f}")

degree  1 | train error  0.631 | test error    0.635
degree  3 | train error  0.194 | test error    0.230
degree 12 | train error  0.000 | test error   20.581

Reading the three rows#

Degree 1 — underfitting. A straight line cannot follow a wave. Training error and test error are both high, and they are close to each other. That closeness is the signature of underfitting. The model is failing equally everywhere, which means the problem is the model, not the data split.

Degree 3 — about right. Both errors dropped. Test error (0.230) sits a little above training error (0.194), which is normal and healthy. A small honest gap is what a good model looks like.

Degree 12 — overfitting, spectacularly. Training error is 0.000. The model passes through all ten training points exactly. Test error is 20.581, which is roughly ninety times worse than the degree-3 model.

Sit with that contrast. Judged on training error alone, degree 12 is flawless and looks like the best model. Judged on data it has never seen, it is worse than useless. Both statements are about the same model, at the same moment.

Why degree 12 could hit zero#

Ten training points and thirteen coefficients. There are more knobs than constraints, so an exact fit through every point is available. Between those points the curve is free to swing wildly, and it does. That is why predictions on the in-between test points explode.

This is the mechanical heart of overfitting. Enough parameters relative to data means memorisation becomes possible, and optimisers will happily find it.

Note on reproducibility#

np.random.default_rng(0) fixes the noise, so these numbers reproduce exactly on any machine with a current NumPy. Swap in np.random.seed() with the legacy np.random.normal and you will get different values. If you change the seed, expect the degree-12 test error to move a great deal. It is a wild curve, so its error is unstable by nature. The pattern across the three rows will hold regardless.

Catching it properly with cross-validation#

A single train/test split is noisy when data is small. Cross-validation splits the data several ways and averages the results.

import numpy as np
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import cross_val_score, KFold

rng = np.random.default_rng(0)
x = np.linspace(0.0, 1.0, 20).reshape(-1, 1)
y = np.sin(2 * np.pi * x.ravel()) + rng.normal(0, 0.25, size=20)

# shuffle=True matters here: x is sorted, and unshuffled folds would
# ask the model to extrapolate into a stretch of x it never saw
folds = KFold(n_splits=5, shuffle=True, random_state=0)

for degree in (1, 3, 5, 12):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    # negative RMSE, because scikit-learn scorers are "higher is better"
    scores = -cross_val_score(model, x, y, cv=folds,
                              scoring="neg_root_mean_squared_error")
    print(f"degree {degree:2d} | mean CV error {scores.mean():8.3f}")

degree  1 | mean CV error    0.745
degree  3 | mean CV error    0.228
degree  5 | mean CV error    0.227
degree 12 | mean CV error   28.454

Three things worth noticing.

Degree 1 is far too simple, and degree 12 is catastrophic. That much matches the single split.

Degrees 3 and 5 come out at 0.228 and 0.227. Treat those as tied. A gap in the third decimal place, across five folds of twenty points, is noise rather than evidence. Reporting "degree 5 is the best model" would be false precision. When two models are this close, pick the simpler one.

Most of these numbers are also higher than the single-split errors above. That is odd at first glance, because each fold trains on sixteen points rather than ten. More training data would normally mean lower error.

The reason is that the earlier split was unusually kind. Taking every other point puts each test point neatly between two training points, so every prediction is a short interpolation. Random folds sometimes remove two neighbouring points at once, leaving a real gap to guess across. Cross-validation is the harsher and more realistic estimate here. Compare models within one scheme, never across schemes.

The shuffle=True argument is doing real work. Passing cv=5 alone uses KFold without shuffling. Since x runs in sorted order, each fold would then hold out one contiguous stretch of the wave. The model would be scored on extrapolation rather than interpolation, and every error would inflate. On this data that pushes the degree-12 mean error into the thousands.

Common mistakes#

Choosing a model by training score. Degree 12 has a perfect training score. This is the mistake the whole lesson exists to prevent.

Tuning on the test set. Try twenty models, keep the one with the best test score, and that score is now optimistic. You have leaked the test set into your decision. Use a three-way split, or cross-validate on training data and touch the test set once at the end.

Scaling before splitting. Calling StandardScaler().fit_transform(X) on all data before splitting leaks test statistics into training. Put the scaler inside a Pipeline, which scikit-learn refits correctly within each fold.

Assuming more data always fixes it. More data helps overfitting. It does nothing for underfitting — a straight line fitted to a million wave points is still a straight line.

Shuffling time-series data. train_test_split shuffles by default. On time-ordered data this lets the model train on the future and test on the past. Use TimeSeriesSplit.

Try it yourself#

In overfit_demo.py, change the sample count from 20 to 200. It appears twice: in the linspace call, and in the size argument of the noise. Change nothing else — same model, same degrees.

degree  1 | train error  0.528 | test error    0.520
degree  3 | train error  0.262 | test error    0.240
degree 12 | train error  0.224 | test error    0.268

Degree 12 went from a test error of 20.581 to 0.268. It is now a reasonable model.

Nothing about the model changed. One hundred training points against thirteen coefficients leaves no room to memorise. The same flexible curve is now forced to find the actual pattern.

You have watched more data cure overfitting without touching the algorithm. This is why data collection is so often the highest-value work available to you.

Notice also that degree 1 barely improved (0.635 to 0.520). More data does nothing for underfitting, exactly as the common-mistakes section warned.

What to learn next#

• Train, test and validation splits — splitting strategies and their failure modes.
• Model evaluation — choosing metrics that expose these problems.
• Random forest — averaging many overfit trees into one that generalises.