Linear Models and Regularisation
Bayesian linear regression
Instead of one best-fit line, Bayesian regression keeps every line the data has not ruled out — so each prediction arrives with an honest error bar.
- 7 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
Bayesian linear regression keeps every line the data has not ruled out, and answers questions with the whole surviving bundle.
A detective with twelve clues does not announce one certain culprit. She keeps a shortlist — strongly suspecting one name, unable to rule out two others — and acts with that spread in mind. More clues arrive, the shortlist narrows.
Ordinary regression names one culprit: the single best-fit line. Bayesian regression keeps the shortlist.
Why it exists
With twelve data points, many different lines fit almost equally well. Picking one and discarding the rest throws away the most decision-relevant fact you own: how much the data still leaves open.
The Bayesian recipe runs in three movements. Start with a prior — what is plausible before seeing data ("the effect is probably modest"). Let the data eliminate: lines that miss badly lose credibility, lines that fit keep it. What remains is the posterior — the shortlist, with a credibility score per line.
To predict, ask every surviving line. Where the lines agree, the answer comes with a tight range. Where they fan out — far from the data, or where data is thin — the range widens by itself. Twelve students' study hours cannot pin down what happens at hour 12; a Bayesian model says so instead of bluffing.
How it works
marks │ ⟋⟋⟋ ← the bundle of surviving lines
│ ⟋⟋⟋
│ ⟋⟋⟋∙ ∙ tight where data lives
│ ⟋⟋∙ ∙ ∙
│ ∙∙ fans WIDE past the last point —
└──────────────── hours uncertainty, reported honestlyA real example you have seen
Election forecasts saying "candidate A: 60–80% likely" are shortlist answers — many futures, weighted. Medicine dosage curves fitted on small trials work the same way. With ten patients, claiming one exact response curve would be malpractice. A bundle of curves with a range is the honest deliverable. Small data is precisely where the Bayesian habit pays.
Remember this
- One best line hides how much the data left open; the bundle shows it.
- Prior = beliefs before data. Posterior = the shortlist after data eliminates.
- Predictions get honest ranges that widen where data is thin — automatically.
What to learn next
- Gaussian process regression — the same bundle-of-fits idea, freed from straight lines.
- Ridge regression — the point-estimate shadow of this lesson.
- Probability — priors, posteriors and the algebra connecting them.
Developer — Code and libraries.
Setup
pip install scikit-learnOutputs verified with scikit-learn 1.7.2 on CPU.
Twelve students, one honest model
import numpy as np
from sklearn.linear_model import BayesianRidge
rng = np.random.default_rng(4)
hours = rng.uniform(1, 8, 12) # study hours, only 12 students
marks = 30 + 5 * hours + rng.normal(0, 5, 12)
model = BayesianRidge().fit(hours.reshape(-1, 1), marks)
for h in (2.0, 5.0, 12.0): # 12 hours is far outside the data
mean, std = model.predict([[h]], return_std=True)
print(f"{h:4.0f} hours -> {mean[0]:5.1f} marks, 95% range +/- {1.96 * std[0]:.1f}")
print(f"learned noise level: +/- {np.sqrt(1 / model.alpha_):.1f} marks")2 hours -> 43.6 marks, 95% range +/- 10.5 5 hours -> 54.7 marks, 95% range +/- 12.3 12 hours -> 80.5 marks, 95% range +/- 19.8 learned noise level: +/- 5.1 marks
The walkthrough
Follow the ranges, not the means. Inside the data (2 and 5 hours), the 95% range is ±10–12 marks. At 12 hours — four hours past the hardest-working student observed — it swells to ±19.8. The surviving lines agree where data constrained them and fan apart where nothing did. No ordinary regression output contains that third column.
The model audited its own noise. We generated marks with ±5 of randomness; the model reports +/- 5.1, learned from twelve points. BayesianRidge estimates the noise level (alpha_) and the coefficient prior's tightness (lambda_) from the data — the knob that ridge regression makes you cross-validate is inferred here.
Two flavours of doubt live in that range. Noise doubt: students wobble ±5 whatever the model knows — irreducible. Line doubt: which line is true — shrinks as data grows, and explodes on extrapolation. The ±19.8 at hour 12 is mostly line doubt. With ten times the students, watch it approach the noise floor while ±10.5 barely moves.
Bayesian ridge is ridge, upgraded. The posterior's single most-credible line is a ridge solution — same shrinkage, same stability. What the Bayesian machinery adds: the spread around it, and self-tuned strength.
Common mistakes
Reading ±19.8 as a guarantee. The range is honest within the model's assumptions — straight line, bell-curve noise. If reality bends past 8 hours (fatigue, ceiling effects), the true error at hour 12 exceeds what any straight-line bundle can express. Uncertainty from a wrong model family is still wrong.
Extrapolating because the error bar came along. The wide range at 12 hours is the model warning you, not licensing the trip. 80.5 ± 19.8 includes marks above 100 — the model does not know marks have a ceiling.
Forgetting return_std=True. Without it, predict returns only means and the entire point of going Bayesian is discarded. The std is there; ask for it.
Assuming this scales like OLS. BayesianRidge is closed-form and cheap at modest width. Full Bayesian inference with hand-chosen priors and non-Gaussian noise needs sampling (PyMC, NumPyro) — powerful, and paid for in compute and diagnostics.
Try it yourself
Refit with 120 students instead of 12 (change one number). Predict at 2, 5 and 12 hours again: which ranges shrank, which barely moved, and why? Then print model.coef_ and compare with Ridge(alpha=1).fit(...) on the same data.
What to learn next
- Gaussian process regression — the same bundle-of-fits idea, freed from straight lines.
- Ridge regression — the point-estimate shadow of this lesson.
- Probability — priors, posteriors and the algebra connecting them.
Researcher — Mathematics and papers.
Conjugate model and posterior
Model: $y = X\beta + \varepsilon$, $\varepsilon \sim \mathcal{N}(0, \sigma^2 I)$, prior $\beta \sim \mathcal{N}(0, \tau^2 I)$. With Gaussian likelihood and prior, the posterior is Gaussian in closed form:
$$ \beta \mid X, y \sim \mathcal{N}(\mu_n, \Sigma_n), \qquad \Sigma_n = \left( \frac{X^\top X}{\sigma^2} + \frac{I}{\tau^2} \right)^{-1}, \quad \mu_n = \frac{\Sigma_n X^\top y}{\sigma^2} $$
Where:
- $\mu_n$ — posterior mean; identical to the ridge estimate with $\lambda = \sigma^2 / \tau^2$.
- $\Sigma_n$ — posterior covariance: the "shortlist spread" over coefficients.
- $\sigma^2, \tau^2$ — noise and prior variances (scikit-learn works with precisions $\alpha = 1/\sigma^2$, $\lambda = 1/\tau^2$).
The predictive distribution at $x_*$ integrates over the posterior:
$$ p(y_* \mid x_, X, y) = \mathcal{N}!\left( x_^\top \mu_n, \;\; \sigma^2 + x_^\top \Sigma_n x_ \right) $$
The variance decomposes exactly into the lesson's "two doubts": irreducible noise $\sigma^2$ plus parameter uncertainty $x_^\top \Sigma_n x_$, which grows quadratically along directions the data left unconstrained — the fan.
Hyperparameters by evidence maximisation
BayesianRidge sets $\alpha, \lambda$ by maximising the marginal likelihood (evidence) $p(y \mid X, \alpha, \lambda)$ — integrating $\beta$ out — via the EM-like updates of MacKay (1992), Bayesian interpolation, and Tipping (2001). This is empirical Bayes: an automatic Occam's razor identical in spirit to Gaussian process kernel tuning. Indeed Bayesian linear regression is GP regression with the linear kernel $k(x, x') = \tau^2 x^\top x'$ — the weight-space and function-space views of one model (Rasmussen and Williams, 2006, ch. 2).
Per-coefficient precisions $\lambda_j$ instead of one shared $\lambda$ give ARD — automatic relevance determination (sklearn.linear_model.ARDRegression): irrelevant features' precisions diverge, pruning them — a Bayesian route to the sparsity lasso reaches by penalty. The lasso itself is the MAP estimate under a Laplace prior (Park and Casella, 2008).
Computation and modern context
Closed-form cost: $O(nd^2 + d^3)$ — same as ridge; the evidence iterations add a small constant factor. Beyond conjugacy (non-Gaussian likelihoods, hierarchies): MCMC via NUTS (Hoffman and Gelman, 2014) in Stan/PyMC/NumPyro, or variational inference for scale.
Honest inference requires model checking — posterior predictive checks and calibration curves (Gelman et al., Bayesian Data Analysis, 3rd ed., 2013) — since credible intervals inherit every modelling assumption. The framework's reach today: Bayesian optimisation's surrogate models, Thompson sampling in bandits (linear posterior sampling; Agrawal and Goyal, 2013), and the last-layer Laplace approximations bringing cheap uncertainty to deep networks (Daxberger et al., 2021) — twelve-point honesty, ported to millions of parameters.
What to learn next
- Gaussian process regression — the same bundle-of-fits idea, freed from straight lines.
- Ridge regression — the point-estimate shadow of this lesson.
- Probability — priors, posteriors and the algebra connecting them.