Reproducing a failure you only see in production

Setup#

pip install scikit-learn pandas

Verified with scikit-learn 1.7.2, pandas 2.2.3, numpy 1.26.4, Python 3.10, CPU. Runs in about three seconds. Seeded, so your numbers should match.

The bug, and the test that finds it#

A fraud model with one numeric column and one category column. The serving code was written by hand, months later, and it lists the cities in a sensible-looking order that is not the training order.

import numpy as np, pandas as pd
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n = 500
df = pd.DataFrame({
    "amount": rng.gamma(2.0, 50.0, n).round(2),
    "city": rng.choice(["mumbai", "pune", "nagpur"], n, p=[.6, .3, .1]),
})
df["fraud"] = ((df.amount > 150) ^ (df.city == "nagpur")).astype(int)

# ---------- offline training ----------
Xtr = pd.get_dummies(df[["amount", "city"]], columns=["city"])
TRAIN_COLUMNS = list(Xtr.columns)
model = LogisticRegression(max_iter=1000).fit(Xtr.to_numpy(dtype=float), df.fraud)
print("training column order:", TRAIN_COLUMNS)

request = {"amount": 220.0, "city": "pune"}

# ---------- how the serving code was written ----------
def featurise_serving(req):                      # hand-written, "obvious" order
    return np.array([[req["amount"],
                      req["city"] == "mumbai",
                      req["city"] == "pune",
                      req["city"] == "nagpur"]], dtype=float)

# ---------- how it should be built ----------
def featurise_correct(req):
    row = pd.get_dummies(pd.DataFrame([req]), columns=["city"])
    return row.reindex(columns=TRAIN_COLUMNS, fill_value=0).to_numpy(dtype=float)

print("serving  column order: ['amount', 'city_mumbai', 'city_pune', 'city_nagpur']")
print()
print(f"probability from the serving code:  {model.predict_proba(featurise_serving(request))[0,1]:.4f}")
print(f"probability it should have been:    {model.predict_proba(featurise_correct(request))[0,1]:.4f}")

# ---------- the test that would have caught it ----------
def skew_report(featurise, k=200):
    bad = 0
    for i in range(k):
        req = {"amount": float(df.amount[i]), "city": df.city[i]}
        if not np.allclose(featurise(req), Xtr.to_numpy(dtype=float)[i]):
            bad += 1
    return bad

print()
print(f"rows where the serving featuriser disagrees with training: {skew_report(featurise_serving)} of 200")
print(f"rows where the correct featuriser disagrees with training: {skew_report(featurise_correct)} of 200")

training column order: ['amount', 'city_mumbai', 'city_nagpur', 'city_pune']
serving  column order: ['amount', 'city_mumbai', 'city_pune', 'city_nagpur']

probability from the serving code:  0.9986
probability it should have been:    0.8319

rows where the serving featuriser disagrees with training: 81 of 200
rows where the correct featuriser disagrees with training: 0 of 200

The walkthrough#

Two probabilities, 0.9986 and 0.8319, from one model and one request. If the alerting threshold is 0.9, this bug flips the decision. Nothing in the serving code is wrong as code — it reads well, it has no bug a reviewer would circle, and it produces a four-element float array exactly as intended.

pd.get_dummies sorts categories alphabetically: mumbai, nagpur, pune. The human wrote them in frequency order. Both orders are reasonable. Only one matches the model.

Only 81 of 200 rows disagree. The mumbai rows — the majority class — are identical under both orders, because swapping two zeros changes nothing. So the bug is invisible on most traffic and fires on a minority segment. That is why it survived launch and arrives as a strange complaint from one region. Bugs that affect everything get caught in staging.

skew_report is the whole reproduction technique. It needs no production access, no traffic replay and no logs. Take rows you already have, push them through both paths, and compare arrays. Nought disagreements or a bug; there is no third answer.

The fix is reindex(columns=TRAIN_COLUMNS, fill_value=0). Saving the column list with the model turns a positional contract into a named one. Positional contracts between two programs written months apart always break eventually.

Where skew comes from, in rough order of frequency#

1. Two implementations of one featuriser. The case above. The structural fix is to ship one function, imported by both sides, rather than two that agree by inspection.
2. Statistics recomputed at serve time. A scaler fitted on the request batch instead of loaded from training. Live means and standard deviations drift by the hour.
3. A category the model never saw. A new city arrives, get_dummies produces a column that does not exist, and naive code either crashes or silently drops it.
4. Type and precision differences. A float64 pipeline served as float32, or an integer column arriving as a string of digits. See dtype mismatch errors.
5. Time-dependent features computed at the wrong moment. "Transactions in the last 30 days" measured from the request time offline and from midnight online.
6. Missing values handled differently. Training filled with the median; serving sends the literal string "null" or a zero.

The debugging order that works#

1. Log the exact feature vector the model received in production, for a request known to be wrong. One row is enough. Without this, everything else is guesswork.
2. Rebuild that same request offline with the training featuriser and diff the two vectors. The index that differs names the bug.
3. If the vectors match, the model is genuinely the problem — and it is now a data-distribution question, not a code question. That is monitoring and drift, a different investigation.
4. Write the skew check as a test so this specific bug cannot come back. Then add it to CI, since the featuriser will be edited again.

Two things make step 1 possible, and both must be built before the incident: log the feature vector alongside the prediction, and attach a version identifier for both the model and the featuriser. Reconstructing a feature vector after the fact turns a one-hour investigation into a one-week one — raw request logs, and a featuriser that has changed twice since.

Common mistakes#

Retraining the model as the first response. It changes the one component that was working and destroys the evidence. It often appears to help briefly, because the new model is less sensitive to the corrupted column.

Testing the serving path only with fresh requests. Fresh requests have no ground truth, so you cannot tell a wrong answer from a surprising one. Replay rows whose correct feature vector you already have.

Comparing predictions instead of features. Two nearby probabilities can come from very different inputs, and a matching probability does not prove matching inputs. Compare the vectors — they are exact.

Fixing serving to match training without a test. The next edit to either side reopens the bug. The test is the deliverable; the fix is a consequence of it.

Assuming a crash is the worst case. The ValueError raised when scikit-learn checks feature names is the lucky outcome. Converting to a bare NumPy array to make that error go away is how a loud failure gets turned into a silent one.

Try it yourself#

Add a fourth city, "thane", to the serving requests only, and run both featurisers on it. Watch what each one does with a category the model has never seen — one produces a wrong-width array, the other silently sends all-zero city flags. Then decide which behaviour you want in production, and write the assertion that enforces it.

What to learn next#

• Monitoring and drift — what to do when the features match and the world has moved.
• Model deployment — the serving path where the second featuriser lives.
• Building a demo that does not break — the smallest possible production system, with the same failure modes.