Why ensembles work

Condorcet's jury theorem#

The oldest result here predates machine learning by two centuries. For $M$ independent voters, each correct with probability $p$, the majority is correct with probability

$$ P_{maj} = \sum_{k=\lceil M/2 \rceil}^{M} \binom{M}{k} p^k (1-p)^{M-k} $$

Where:

• $M$ — the number of voters (odd, to avoid ties).
• $p$ — each voter's probability of being correct, assumed identical.
• $k$ — the number of voters who happen to be correct.

If $p > 0.5$, then $P_{maj} \to 1$ as $M \to \infty$; if $p < 0.5$, it tends to 0 (Condorcet, 1785, Essai sur l'application de l'analyse). With $p = 0.65$, $M = 11$: $P_{maj} \approx 0.847$ — matching the simulation above to within sampling noise.

The theorem's independence assumption is exactly what real ensembles violate, which motivates the next decomposition.

Variance of a correlated average#

For $M$ regression estimators, each with variance $\sigma^2$ and average pairwise correlation $\rho$, the variance of their mean is

$$ \operatorname{Var}\left(\bar{f}\right) = \rho \sigma^2 + \frac{1-\rho}{M} \sigma^2 $$

Where:

• $\sigma^2$ — the variance of a single member's prediction across training samples.
• $\rho$ — the average correlation between two members' errors.
• $M$ — the ensemble size.

The second term dies as $M$ grows. The first term does not. Ensemble size attacks only the uncorrelated part; correlation sets the floor. Every practical ensemble method — bagging, random feature selection, different algorithms — is a device for pushing $\rho$ down.

The ambiguity decomposition#

Krogh and Vedelsby (1995), Neural network ensembles, cross validation, and active learning, show for a weighted ensemble average under squared error:

$$ E = \bar{E} - \bar{A} $$

Where:

• $E$ — the ensemble's squared error at a point.
• $\bar{E}$ — the weighted average of the members' individual squared errors.
• $\bar{A}$ — the ambiguity: the weighted average squared deviation of each member from the ensemble output.

Since $\bar{A} \geq 0$ always, the ensemble is never worse than the average member, and the gain equals the disagreement. This is an identity, not a bound — disagreement is not a proxy for the benefit, it is the benefit. The generalisation to bias-variance-covariance is Ueda and Nakano (1996).

Why single models fail: Dietterich's three reasons#

Dietterich (2000), Ensemble methods in machine learning, gives three orthogonal reasons ensembles help:

1. Statistical — with limited data, many hypotheses fit equally well; averaging them hedges the risk of picking a wrong one.
2. Computational — training is a local search that gets stuck; different starting points reach different local optima, and averaging approximates the search done right.
3. Representational — the true function may sit outside any single member's hypothesis class, but inside the span of several.

Current state#

On tabular data, ensembles of trees remain the strongest general-purpose models — Grinsztajn, Oyallon and Varoquaux (2022), Why do tree-based models still outperform deep learning on typical tabular data? (NeurIPS). In deep learning, deep ensembles — several networks trained from different initialisations — are a strong baseline for uncertainty estimation (Lakshminarayanan et al., 2017, Simple and scalable predictive uncertainty estimation using deep ensembles).

The two dominant construction strategies get their own lessons: independent members averaged (bagging, Breiman 1996) and sequential members added (boosting, Freund and Schapire 1997).

What to learn next#

• Bagging — the standard recipe for manufacturing diverse models from one dataset.
• Random forest — bagging plus feature randomness, the workhorse ensemble.
• XGBoost — the other family: models built in sequence, each fixing the last.