Monitoring Models in Production
Measuring data drift
Data drift is when the inputs a live model sees stop looking like the inputs it was trained on, and PSI is the simplest number that tells you how much.
- 8 min read
- 3 reading levels
- Published
Read these first
On this page 8
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The short answer
Data drift is when the real-world inputs reaching your model slowly stop matching the inputs it learned from.
The analogy you have already lived
Think of a mother who has cooked the same dal every week for ten years. She never changed the recipe.
But the flour brand in the shop changed twice. The salt got a little less salty last year. The pressure cooker was replaced.
Nobody decided to change the dish. The ingredients changed under her, slowly enough that no single week felt different. A model has the same problem, except its ingredients are the numbers a user sends in.
Why it exists
A model learns from a snapshot of data taken at one point in time. Incomes, ages, click patterns, whatever the columns are.
The world does not hold still after that snapshot is taken. Prices rise. A new city gets added to your app. A festival season changes what people buy.
The model itself never updates on its own. It keeps applying rules learned from an older world to today's inputs, quietly, without complaining. Drift is the name for that gap growing.
How it works
training data (last year) live data (this week)
income: mostly 30k-60k income: mostly 45k-90k
| |
v v
[ compare the two shapes of the same column ]
|
v
"these look different now" -> alertYou are not checking whether any single prediction is right. You are checking whether the shape of the incoming data still looks like the shape the model was trained on.
A real example you have seen
A food delivery app's model was trained mostly on orders from before a long lockdown changed how people cooked at home. Order sizes, timings and cuisines afterward looked different enough that its "usual order" predictions stopped being useful. Nobody measured that for weeks.
Data drift is the tool that would have caught that early, instead of someone noticing complaints piling up.
The honest part
A drift alert does not tell you the model got worse. It tells you the inputs changed shape. Those are related but not the same thing, and the difference matters.
Sometimes drifted data still produces a perfectly good score. Read the alert as "go check", not as "the model is broken".
Remember this
- Data drift is the input data changing shape after training, not the model changing.
- It is measured by comparing distributions, the training snapshot against a recent window.
- A drift alert means "investigate", not automatically "the model is wrong".
What to learn next
- Concept drift — when the relationship between inputs and the right answer shifts, not the inputs themselves.
- Monitoring and model drift — the wider picture this lesson zooms into.
- Logging every prediction — you need this data captured before you can measure drift at all.
Developer — Code and libraries.
Setup
pip install numpy pandasThe idea in one function
The Population Stability Index (PSI) buckets a column into ten groups using the training data, then checks how much the live data's group sizes moved. A plain-English name for it: a distribution-shape difference score.
import numpy as np
import pandas as pd
rng = np.random.RandomState(0)
# "training" snapshot: income in thousands per month, the same feature
# the loan-scoring model in the MLOps lessons was trained on.
train_income = rng.normal(loc=45, scale=12, size=2000)
# "this week" snapshot, six months later. The average has quietly
# climbed and the spread has widened -- a slow, believable drift.
live_income = rng.normal(loc=53, scale=16, size=2000)
def psi(expected, actual, buckets=10):
"""Population Stability Index between two 1-D samples."""
breakpoints = np.quantile(expected, np.linspace(0, 1, buckets + 1))
breakpoints[0], breakpoints[-1] = -np.inf, np.inf
e_counts = np.histogram(expected, bins=breakpoints)[0] / len(expected)
a_counts = np.histogram(actual, bins=breakpoints)[0] / len(actual)
e_counts = np.clip(e_counts, 1e-4, None)
a_counts = np.clip(a_counts, 1e-4, None)
return float(np.sum((a_counts - e_counts) * np.log(a_counts / e_counts)))
score = psi(train_income, live_income)
print(f"PSI(income): {score:.3f}")
if score < 0.1:
print("verdict: stable, no action")
elif score < 0.25:
print("verdict: moderate shift, watch it")
else:
print("verdict: significant drift, investigate")
# A second feature that did NOT drift, scored the same way, as a contrast.
train_age = rng.normal(loc=38, scale=9, size=2000)
live_age = rng.normal(loc=38, scale=9, size=2000)
print(f"PSI(age): {psi(train_age, live_age):.3f} (same generator, both snapshots)")PSI(income): 0.361 verdict: significant drift, investigate PSI(age): 0.008 (same generator, both snapshots)
These exact numbers are real output from this script with a fixed random seed (RandomState(0)), so running it yourself reproduces them precisely. Your production PSI values will differ, because your data differs.
Line-by-line walkthrough
np.quantile(expected, ...) cuts the training column into ten equal-sized buckets. The buckets themselves come only from training data, and stay fixed.
np.histogram(actual, bins=breakpoints) counts how many live rows fall into each of those same buckets. If live data shifted upward, the low buckets thin out and the high buckets fill up.
np.clip(..., 1e-4, None) stops any bucket from being exactly zero, because the next line divides by it. A zero there would crash the score, not only distort it.
The thresholds (0.1, 0.25) are an industry rule of thumb, not a law of nature. Treat them as a starting point to tune against your own alert fatigue.
Common mistakes
Running PSI on a categorical column with hundreds of values. Bucket it into the top N categories plus "other" first, or the score becomes noise.
Comparing today against yesterday instead of against training. That measures day-to-day wobble, not drift from what the model actually learned. Always anchor to the training snapshot.
Treating a high PSI as proof the model got worse. It only proves the inputs changed shape. Confirm actual harm with the labelled checks in working with delayed labels.
Recomputing bucket edges on every run. The breakpoints must come from training data once, and get reused every time after. Recomputing them from live data each time hides the drift instead of showing it.
Try it yourself
Change live_income's scale from 16 back to 12, keeping the mean shift. Rerun, and watch the PSI score. It should drop, because now only the centre moved, not the spread.
What to learn next
- Concept drift — when the relationship between inputs and the right answer shifts, not the inputs themselves.
- Monitoring and model drift — the wider picture this lesson zooms into.
- Logging every prediction — you need this data captured before you can measure drift at all.
Researcher — Mathematics and papers.
The formula
$$\text{PSI} = \sum_{i=1}^{k} (a_i - e_i) \cdot \ln\left(\frac{a_i}{e_i}\right)$$
Where $k$ is the number of buckets, $e_i$ is the fraction of the expected (training) sample in bucket $i$, and $a_i$ is the fraction of the actual (live) sample in the same bucket. Each bucket's contribution is zero only when $a_i = e_i$, and grows for both increases and decreases, since it is not signed.
PSI is a discretised, symmetrised relative-entropy measure: each term resembles a two-way Kullback-Leibler contribution. Conventional bands, widely cited but not derived from any distributional theorem: below 0.1 stable, 0.1–0.25 moderate shift, above 0.25 material shift requiring action.
Alternatives and when they are better
| Method | Handles | Weakness |
|---|---|---|
| PSI | Continuous, binned | Bucket-count sensitive, no p-value |
Kolmogorov–Smirnov (scipy.stats.ks_2samp) | Continuous, unbinned | Univariate only, sensitive to sample size |
| Jensen–Shannon divergence | Continuous or categorical | Needs density estimation for continuous data |
| Maximum Mean Discrepancy (MMD) | Multivariate, high-dimensional | Kernel choice matters, costlier to compute |
| Domain classifier ("can a model tell the two datasets apart?") | Multivariate | A trained-model result standing in for a hypothesis test |
The domain-classifier approach — train a binary classifier to separate "old" rows from "new" rows, and read its held-out AUC as the drift signal — captures joint, multivariate shift that per-column PSI misses entirely, at the cost of another model to maintain.
Complexity and cost
PSI on one column is $O(n)$ per snapshot for the histogram pass, after an $O(n \log n)$ sort for the training quantiles, computed once and cached. Running it across $p$ columns on every batch is $O(p \cdot n)$, cheap enough to run on every prediction window for typical production feature counts.
Papers and further reading
- Chen and Guestrin discuss distribution shift diagnostics in the broader context of model monitoring surveys; PSI itself originates from credit-risk scorecard monitoring practice rather than a single paper.
- Rabanser, Günnemann and Lipton, Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift, NeurIPS 2019 — arxiv.org/abs/1810.11953, a systematic comparison of statistical-test and classifier-based shift detectors.
- Gretton et al., A Kernel Two-Sample Test (MMD), JMLR 2012 — jmlr.org/papers/v13/gretton12a.html
What to learn next
- Concept drift — when the relationship between inputs and the right answer shifts, not the inputs themselves.
- Monitoring and model drift — the wider picture this lesson zooms into.
- Logging every prediction — you need this data captured before you can measure drift at all.