Learn AI Create the Future — with Pranay
Main site

Validation and Hyperparameter Search

Bayesian optimisation

Bayesian optimisation keeps a running guess of how good every untried setting is and spends each expensive trial where the guess says it will pay most.

Read these first

On this page 5
  1. Why this had to be invented
  2. How it works
  3. A real example you have seen
  4. Remember this
  5. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Bayesian optimisation remembers every trial it has run.

It keeps a running guess about the settings it has not tried. Each next trial goes where that guess looks most promising.

Think of drilling borewells on a large farm. Each borewell costs serious money, so nobody drills in a blind pattern. After every drill you update your mental map: water here at 40 feet, dry rock there. The next spot is chosen by that map — near a good strike, or in an untouched corner the map knows nothing about.

That balance — exploit what worked, explore what is unknown — is the whole art, and Bayesian optimisation does it with arithmetic instead of instinct.

Why this had to be invented

Random search has amnesia. Trial forty is chosen exactly as blindly as trial one, ignoring the thirty-nine results already paid for. When one trial costs seconds, amnesia is fine — buy more trials.

When one trial means training a big model for six hours, amnesia is a luxury nobody can afford. Every scrap of information from finished trials should shape the next choice. "Bayesian" is the statistician's word for exactly this habit: start with a belief, update it with each new piece of evidence.

How it works

Two parts run in a loop.

The surrogate is the mental map: a cheap stand-in model that predicts, for every untried setting, what score to expect and how unsure it is. Near tried points it is confident; far away it admits ignorance.

The acquisition rule is the drilling decision. It scores every candidate by combining "predicted to be good" with "still uncertain". The next trial goes where that combination peaks.

settings axis  →   x=0.0        x=0.5        x=1.0
tried:             poor         poor         good
map's guess:    ~~ low ~~~~~~ dips ~~~~~~ rising ~~
map's doubt:      small        small     ← large near
                                           the edges
                                   ↑
                    next trial: promising AND uncertain

Run the trial, feed the result back into the map, repeat. Ten well-chosen trials can beat a hundred blind ones.

A real example you have seen

Google's tuning service Vizier runs this loop across the company. Its team once used it to improve a chocolate-chip cookie recipe over weeks of real baking. Each batch was a costly "trial". Anyone who has adjusted a new pressure cooker over several attempts has run the loop by hand. More water, less time, updating after each result.

Remember this

  • BO = a memory (surrogate map) plus a decision rule (explore vs exploit).
  • It exists for expensive trials — hours per training run, not seconds.
  • Each result updates the map, so late trials are far better aimed than early ones.

What to learn next

Developer — Code and libraries.

Setup

bash
pip install scikit-learn scipy

Outputs verified with scikit-learn 1.7.2, scipy 1.14.1 and numpy 1.26.4 on CPU. In real work you would use Optuna and never write this loop — the code below exists to remove the mystery, in about twenty-five lines.

The loop, from scratch

A pretend "validation error versus learning-rate setting" curve stands in for an expensive training run. The Gaussian process is the map; expected improvement is the drilling rule.

bo_from_scratch.py
import numpy as np
from scipy.stats import norm
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern

def validation_error(x):        # stands in for "train the model, measure the error"
    return 0.35 + 0.28 * np.sin(5 * x) + 0.2 * (x - 0.6) ** 2

grid = np.linspace(0, 1, 201).reshape(-1, 1)   # every setting we could try
tried = [[0.0], [0.5], [1.0]]                  # three starting probes
seen = [validation_error(x[0]) for x in tried]

# fixed kernel width keeps this demo stable with very few points
gp = GaussianProcessRegressor(Matern(length_scale=0.2, nu=2.5),
                              optimizer=None, normalize_y=True)
for step in range(1, 8):
    gp.fit(np.array(tried), np.array(seen))
    mu, sigma = gp.predict(grid, return_std=True)
    sigma = np.maximum(sigma, 1e-9)            # no dividing by zero at points already tried
    gain = np.min(seen) - mu                   # how much better than our best could this be?
    ei = gain * norm.cdf(gain / sigma) + sigma * norm.pdf(gain / sigma)
    x_next = grid[np.argmax(ei)]
    tried.append(list(x_next))
    seen.append(validation_error(x_next[0]))
    print(f"step {step}: tried x={x_next[0]:.3f}  error={seen[-1]:.4f}  "
          f"best so far={min(seen):.4f}")

truth = grid.ravel()[np.argmin(validation_error(grid.ravel()))]
print(f"true best setting: x={truth:.3f}  error={validation_error(truth):.4f}")
Output
step 1: tried x=0.905  error=0.0935  best so far=0.0935
step 2: tried x=0.810  error=0.1380  best so far=0.0935
step 3: tried x=0.940  error=0.0931  best so far=0.0931
step 4: tried x=0.925  error=0.0922  best so far=0.0922
step 5: tried x=0.230  error=0.6330  best so far=0.0922
step 6: tried x=0.700  error=0.2538  best so far=0.0922
step 7: tried x=0.870  error=0.1028  best so far=0.0922
true best setting: x=0.925  error=0.0922

Ten evaluations in total — three probes plus seven chosen steps — and it found the exact best point on a 201-point grid: x=0.925. A grid search of the same resolution would have paid for 201 evaluations.

The walkthrough

Step 1 already lands near the optimum. With three probes, the map noticed the trend "error falls toward x=1" and pounced. This is the memory advantage in action: information from probe results aimed the very first decision.

Step 5 is the explore move. After milking the good region for three steps, the acquisition rule's uncertainty term won out and it paid one trial to check the unexplored middle-left. The result was bad — and valuable: the map now knows that region is bad, and never returns. A pure exploiter could circle a local dip forever; that one "wasted" trial is the insurance against it.

gain * norm.cdf(...) + sigma * norm.pdf(...) is expected improvement: the average amount by which a candidate would beat the current best, under the map's uncertainty. Its two terms are literally exploit (high predicted gain) plus explore (high sigma).

optimizer=None with a fixed kernel width is a teaching simplification. Real implementations re-estimate the kernel's smoothness from the data each step — with three points that estimation is unstable, so we pin it.

Common mistakes

Using BO for cheap trials. If a trial takes twenty seconds, 200 random trials cost an hour and need no cleverness. BO's bookkeeping and serial nature pay off when single trials cost many minutes to hours.

Too many knobs. The map-building machinery degrades past roughly twenty dimensions — uncertainty becomes vague everywhere and the method drifts toward expensive random search. Prune the space first using knowledge like the gradient-boosting priority list.

Ignoring noise. Rerun the same setting and the score wobbles (different seeds, data order). Tell the surrogate scores are noisy (an alpha/noise term) or it will chase wiggles that are not real.

Skipping the random startup. Fitting the map to one or two points produces confident nonsense. Every serious implementation begins with a handful of random or space-filling trials — mirroring what random search would do — before trusting the map.

Try it yourself

Add noise to the objective — + rng.normal(0, 0.03) — and watch the loop's behaviour with and without telling the GP (alpha=0.03**2 in the constructor). Then change the acquisition to pure exploitation by setting ei = -mu and count how many steps it spends stuck circling one region.

What to learn next

Researcher — Mathematics and papers.

Gaussian-process surrogate

Model the objective $f$ with a GP prior: $f \sim \mathcal{GP}(0, k)$ with kernel $k$ (Matérn 5/2 is the standard choice for hyperparameter response surfaces — rougher than RBF's infinite smoothness). Given observations $\mathbf{y}$ at points $X$, the posterior at $x$ is Gaussian with

$$ \mu(x) = k(x, X)\,[K + \sigma_n^2 I]^{-1}\,\mathbf{y}, \qquad \sigma^2(x) = k(x, x) - k(x, X)\,[K + \sigma_n^2 I]^{-1}\,k(X, x) $$

Where $K = k(X, X)$ is the kernel matrix and $\sigma_n^2$ the observation-noise variance. Fitting costs $O(n^3)$ in the number of trials $n$ — irrelevant at $n \sim 100$, prohibitive at $n \sim 10^5$, which cleanly delimits BO's regime.

Expected improvement

With current best $f^* = \min_i y_i$ and $z = (f^* - \mu(x))/\sigma(x)$:

$$ \mathrm{EI}(x) = (f^* - \mu(x))\,\Phi(z) + \sigma(x)\,\phi(z) $$

Where $\Phi, \phi$ are the standard normal CDF and PDF. The first term rewards predicted improvement, the second rewards uncertainty — the closed-form explore–exploit trade. EI dates to Mockus (1978); its modern form and the EGO framework are Jones, Schonlau and Welch (1998), Efficient global optimization of expensive black-box functions. Alternatives: upper/lower confidence bound $\mu - \beta\sigma$ (GP-UCB, Srinivas et al. 2010, with regret bounds), knowledge gradient, and entropy search (predictive-information maximisation).

The paper that brought BO to ML

Snoek, Larochelle and Adams (2012), Practical Bayesian optimization of machine learning algorithms (NeurIPS): Matérn 5/2, MCMC marginalisation of kernel hyperparameters, an expected-improvement-per-second variant for cost-aware search, and parallelism via fantasised outcomes. Its CNN tuning results beat human experts and made "throw a GP at it" the default for expensive tuning.

Tree-structured Parzen estimator

TPE (Bergstra, Bardenet, Bengio and Kégl, 2011, Algorithms for hyper-parameter optimization, NeurIPS) flips the modelling: instead of $p(y \mid x)$, model two densities — $\ell(x)$ over configurations in the best $\gamma$ quantile of results and $g(x)$ over the rest — and maximise $\ell(x)/g(x)$, which is monotone in EI under the quantile formulation. It handles the conditional, tree-structured spaces where GPs struggle, scales linearly in trials, and is the default sampler in Optuna. SMAC (Hutter, Hoos and Leyton-Brown, 2011) instead uses random-forest surrogates — robust for categorical-heavy spaces.

Frontiers and limits

  • High dimensions: trust-region methods (TuRBO, Eriksson et al. 2019), additive/low-dimensional-embedding kernels (REMBO, Wang et al. 2016).
  • Multi-fidelity: combine with cheap approximations — the bridge to Hyperband; BOHB (Falkner et al., 2018) is the standard hybrid.
  • Parallel/async: batch EI, Thompson sampling; industrial systems in Vizier (Golovin et al., 2017, KDD).
  • Benchmark honesty: on cheap objectives with large budgets, well-tuned random search remains embarrassingly competitive (Li and Talwalkar, 2020); BO's wins concentrate exactly where theory says — few, expensive, sequential trials on smooth-ish low-dimensional objectives.

What to learn next