Ensemble

An ensemble is several models whose predictions are combined — by voting or averaging — into a single, usually better, prediction.

A hospital gets a second opinion before major surgery, and a third if the first two disagree. Each doctor has blind spots, but different blind spots, so their agreement is far more trustworthy than any one voice. That is the entire theory of ensembling: combine models whose errors are different, and the errors partially cancel while the signal reinforces.

The one requirement is genuine disagreement. Averaging five identical models achieves nothing. The classic strategies manufacture diversity in different ways:

bagging   train the same model on different random data slices   → random forest
boosting  train models in sequence, each on the last one's errors → XGBoost
stacking  train a small model to combine other models' outputs

In bias-variance-tradeoff terms, bagging attacks variance and boosting attacks bias — which is why each pairs naturally with a different kind of weak learner.

Ensembles of trees (random-forest, gradient-boosting) dominate tabular ML. Averaging a few independently trained networks reliably adds a point or two of accuracy, at the price of running all of them at inference — which is exactly the cost distillation was invented to remove: compress the committee back into one model.

Where to go next#

• Full lesson: Random forest
• Related terms: random-forest, gradient-boosting, decision-tree, bias-variance-tradeoff