Guardrail metrics

Setup#

pip install numpy scipy

Verified with numpy 1.26.4 and scipy 1.14.1.

A win with a broken guardrail#

A new product page converts better — and, being heavier, loads slower. Both facts are in the data, waiting to be checked.

import numpy as np
from scipy import stats

rng = np.random.default_rng(21)
n = 20_000
buy_c = rng.random(n) < 0.100          # control conversion: 10.0%
buy_t = rng.random(n) < 0.112          # the new page converts better...
ms_c = rng.gamma(4, 90, n)             # ...but it is heavier, so it loads slower
ms_t = rng.gamma(4, 99, n)

for name, c, t in [("purchase rate", buy_c, buy_t),
                   ("load time, ms", ms_c, ms_t)]:
    p = stats.ttest_ind(t.astype(float), c.astype(float)).pvalue
    print(f"{name}: control {c.mean():7.3f}  treatment {t.mean():7.3f}  p={p:.4f}")

purchase rate: control   0.097  treatment   0.111  p=0.0000
load time, ms: control 357.284  treatment 397.305  p=0.0000

The walkthrough#

Both rows are significant — and they disagree about shipping. Conversion rose from 9.7% to 11.1%. Load time rose 40 ms. A dashboard showing only the first row says ship; the second row says the win was partly bought with speed, and speed debts compound across teams.

The decision is a policy, not a p-value. A sane policy, agreed before launch: ship if the goal metric wins and no guardrail shows damage beyond its margin — say, 20 ms for load time. Here the measured 40 ms breaches the margin, so the feature goes back for a diet, even though its own metric won.

Choose few, sensitive guardrails. Standard set: latency percentiles (p50/p95), crash and error rates, revenue per user, retention or return rate, unsubscribe and complaint rates, plus the sample ratio check, which is a guardrail on the experiment itself.

Watch the multiple-testing arithmetic. Ten guardrails, each read at 0.05, fire falsely about 40% of the time somewhere. Either apply a correction, or treat single marginal guardrail alarms as triggers for a closer look rather than automatic vetoes. What you must not do is scan ten dials and quote the worst one as if it were the only test run.

Common mistakes#

Reading "no significant change" as "safe". An underpowered experiment cannot distinguish "no damage" from "damage too small for this sample". For guardrails that matter, state a margin and require the confidence interval to exclude damage beyond it — a non-inferiority test, not a significance test.

Adding twenty guardrails to look rigorous. Each extra dial adds false alarms and dilutes attention. Five well-chosen guardrails that always trigger investigation beat twenty that everyone learns to ignore.

Letting the goal team define its own guardrails. The team's incentive is a green dashboard. Guardrail definitions and margins belong to a central experimentation function, agreed before the experiment starts.

Averaging away tail damage. A mean load time can hold steady while the p95 doubles — the slowest users absorbing all the pain. Guardrail latency at percentiles, not means.

Try it yourself#

Implement the non-inferiority version: with a 20 ms margin, compute the 95% confidence interval for the load-time difference (stats.ttest_ind(...).confidence_interval()) and check whether it lies entirely below +20 ms. Then shrink n to 2,000 and watch the interval widen until safety can no longer be demonstrated — the honest cost of small experiments.

What to learn next#

• Bayesian A/B testing — expressing shipping decisions as expected loss rather than thresholds.
• Sample ratio mismatch — the guardrail on the experiment itself.
• Novelty and primacy effects — why guardrails need watching over time, not once.