Training and serving environment parity

Setup#

pip install scikit-learn pandas joblib

A real skew bug, and the exact numbers it produces#

This trains a small model, then scores the same applicant two ways. First through the correctly saved pipeline. Then through a common, real mistake — serving code that recomputes normalisation statistics from whatever batch happens to be nearby, instead of reusing the frozen statistics from training.

import joblib
import numpy as np
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

rng = np.random.RandomState(0)
n = 400
X = pd.DataFrame({
    "income": rng.uniform(5, 80, n).round(1),
    "years": rng.uniform(0, 10, n).round(1),
})
y = (0.05 * X["income"] + 0.4 * X["years"] - 2 + rng.normal(0, 0.5, n) > 0).astype(int)
model = make_pipeline(StandardScaler(), LogisticRegression()).fit(X, y)
joblib.dump(model, "parity_model.joblib")

applicant = pd.DataFrame([{"income": 40.0, "years": 3.0}])
correct_score = model.predict_proba(applicant)[0, 1]
print(f"the applicant's true score, from the saved pipeline: {correct_score:.4f}\n")


def buggy_serving_score(applicant_row, batch):
    """A real mistake: re-fitting 'standardisation' on whatever else is in
    the current batch, instead of reusing the training-time mean and std."""
    combined = pd.concat([batch, applicant_row], ignore_index=True)
    scaled = (combined - combined.mean()) / combined.std()
    lr = model.named_steps["logisticregression"]
    z = applicant_row.copy()
    z.iloc[0] = scaled.iloc[-1]
    return float(1 / (1 + np.exp(-(lr.intercept_[0] + z.values[0] @ lr.coef_[0]))))


def correct_serving_score(applicant_row):
    """Reuses the exact StandardScaler fitted during training. No refitting."""
    return float(model.predict_proba(applicant_row)[0, 1])


quiet_batch = X.sample(5, random_state=1)
busy_batch = X.sample(5, random_state=2) * 1.8  # a batch of unusually high earners

print("buggy serving code, same applicant, two different surrounding batches:")
print(f"  next to a quiet batch: {buggy_serving_score(applicant, quiet_batch):.4f}")
print(f"  next to a busy batch:  {buggy_serving_score(applicant, busy_batch):.4f}")

print("\ncorrect serving code, same two situations:")
print(f"  next to a quiet batch: {correct_serving_score(applicant):.4f}")
print(f"  next to a busy batch:  {correct_serving_score(applicant):.4f}")

the applicant's true score, from the saved pipeline: 0.9589

buggy serving code, same applicant, two different surrounding batches:
  next to a quiet batch: 0.9989
  next to a busy batch:  0.5027

correct serving code, same two situations:
  next to a quiet batch: 0.9589
  next to a busy batch:  0.9589

Deterministic with the fixed seeds throughout — this reproduces exactly on any machine, every run.

Walking through it#

The exact same applicant scores 0.9989 in one batch and 0.5027 in another, with the buggy code. Nothing about the applicant changed between those two lines — only which other rows happened to be nearby when the "standardisation" was computed. A score that depends on unrelated people in the same batch is not a model behaving correctly.

The correct code gives the identical answer, 0.9589, regardless of the batch. It reuses the exact StandardScaler that was fitted once, during training, and saved inside the pipeline. That is the same discipline the training script in model serving already follows, keeping the scaler inside the saved pipeline rather than as a separate step.

The bug is subtle because both versions "compute a z-score." Read in isolation, buggy_serving_score looks like a reasonable, even faster, reimplementation. The defect only shows up by comparing its output against the correct version, side by side. That is exactly why this class of bug survives code review so often.

Common mistakes#

Reimplementing preprocessing by hand "for speed" in the serving path. The pipeline object already knows exactly how to transform new data correctly — calling model.predict_proba on it, as correct_serving_score does, is both simpler and correct. A hand-written reimplementation is extra code that can drift, for a speed gain that is rarely worth the risk.

Computing any statistic — mean, std, min, max — from the incoming request or batch, instead of from training. Anything computed "live" from whatever data happens to be nearby depends on that data, by definition. Only statistics frozen at training time, and saved alongside the model as in packaging a model artifact, give a stable, reproducible answer.

Trusting that two pipelines are equivalent because their code "looks similar." As shown above, similar-looking code can compute genuinely different numbers. Test parity directly — score identical known inputs through both paths, and assert the outputs match — rather than reviewing the code by eye alone.

Assuming this only happens with hand-written features. Even automated feature pipelines can skew, if the batch job and the real-time job pull from different underlying tables, or apply a transformation library at two different versions. It is the version-pinning problem from pinning ML dependencies, applied specifically to feature computation.

Try it yourself#

Change busy_batch to X.sample(5, random_state=2) * 5 — an even more extreme batch. Watch the buggy score move even further from the correct 0.9589, while the correct version, unaffected by any batch at all, stays exactly the same.

What to learn next#

• Model serving — the pipeline discipline (scaling inside the saved object) that prevents this bug in the first place.
• Pinning ML dependencies — closing the library-version half of the parity gap this lesson focuses on for feature logic.
• CI/CD for machine learning — where a parity check like this one belongs, so it runs on every change instead of being discovered in production.