Learn AI Create the Future — with Pranay
Main site

AI glossary

Overfitting

In one sentence Overfitting is when a model memorises its training examples instead of learning the pattern, so it scores well in practice runs and badly on new data.

By Updated

Overfitting is when a model memorises the training examples instead of learning the pattern behind them — brilliant on data it has seen, poor on anything new.

Two students prepare for the same exam. One works through last year's paper until they can recite every answer. The other learns the topics. On last year's paper the first student scores higher. On this year's paper, with different numbers in the questions, they collapse. Your model is capable of being the first student, and it will be, given the chance.

The signal is unmistakable once you look for it, and it is why you keep a validation set the model never trains on. While the model is genuinely learning, both training loss and validation loss fall together. When memorising starts, they separate.

epoch    :   1     3     5     7     9    11    13
train acc:  61%   74%   83%   90%   95%   98%   99%
valid acc:  60%   72%   80%   84%   83%   79%   74%
                                     ▲
                          learning stopped here, memorising began

What actually causes it

Too much model capacity for too little data is the usual pairing — a network with millions of parameters and 500 training rows has enough room to store every row. Training for too many epochs does it too. So does data leakage, where something in your features quietly encodes the answer, which produces suspiciously excellent validation scores and terrible real-world ones.

The fixes, in the order worth trying: get more data, or generate more with augmentation (flips, crops, noise). Stop early, keeping the weights from the best validation epoch. Add dropout, which switches off random neurons during training so no single path can memorise. Add weight decay to penalise large weights. Reduce model size. For tree models, limit depth and set a minimum number of samples per leaf.

The opposite failure is underfitting — the model is too simple and does badly on both training and validation data. Check which one you have before reaching for a cure, because the treatments point in opposite directions.

Where to go next