What is MLOps?

The framing paper#

Sculley et al. (2015), Hidden Technical Debt in Machine Learning Systems, is the origin document for this field and still the sharpest thing written about it. Its central observation is a figure: the box labelled "ML code" occupies a few percent of a real system's area. Configuration, data collection, feature extraction, serving infrastructure, process management and monitoring occupy the rest.

Its most-cited contribution is the CACE principle — Changing Anything Changes Everything. In a system with $n$ input features, no feature has an isolated effect. Removing feature $x_i$ changes the learned weights on all remaining features, because the optimiser re-solves a coupled problem. Consequently:

• Model components cannot be unit-tested in the way software components can.
• There is no meaningful notion of an isolated "interface" between a feature and a model.
• Ablation cost is $O(n)$ retrains, not $O(1)$ inspections.

The paper's taxonomy of ML-specific debt is worth memorising: entanglement (CACE), correction cascades, undeclared consumers, unstable data dependencies, underutilised data dependencies, feedback loops (direct and hidden), pipeline jungles, dead experimental codepaths, and configuration debt.

Training-serving skew, formally#

Let $p_{\text{tr}}(x, y)$ be the joint distribution the model was fitted on and $p_{\text{sv}}(x, y)$ the distribution encountered at serving time. Skew is any $p_{\text{tr}} \neq p_{\text{sv}}$, but the causes decompose usefully:

• Implementation skew — the feature transformation code differs between the training path and the serving path. This is a software defect, not a distributional one, and it is by far the most common. The demonstration in the Developer tab is this case.
• Covariate shift — $p_{\text{tr}}(x) \neq p_{\text{sv}}(x)$ while $p(y \mid x)$ is stable (Shimodaira, 2000).
• Label shift — $p_{\text{tr}}(y) \neq p_{\text{sv}}(y)$ while $p(x \mid y)$ is stable.
• Concept drift — $p(y \mid x)$ itself changes over time.

Only the last three are statistical problems. The first is a plumbing problem, and it dominates real incident counts. Feature stores exist mainly to make implementation skew structurally impossible by serving both paths from one definition.

Measurable maturity#

Breck et al. (2017), The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction, converts "is this system production-ready" into 28 scored tests across four axes: features and data, model development, ML infrastructure, and monitoring. The rubric is deliberately blunt — each test scores 0, 0.5 or 1 — and the paper reports that surveyed Google teams cluster at low scores on the monitoring axis specifically.

The highest-value tests in the rubric, by the authors' own reporting, are unglamorous: test that every feature is beneficial, test that the model is not stale, test for training-serving skew, and test that the serving path handles a rollback.

Underspecification#

D'Amour et al. (2020), Underspecification Presents Challenges for Credibility in Modern Machine Learning, gives the sharpest statistical account of why an in-distribution test score fails to predict deployment behaviour.

The argument: a training pipeline typically admits a large set of predictors with near-identical held-out risk. The pipeline does not constrain which member of that set you get — random seed, data ordering and hardware non-determinism do. Members of the set differ arbitrarily in behaviour under distribution shift and on subgroups.

The practical consequence is that seed variance is not noise to be averaged away in a paper. It is a lower bound on the deployment variance you should expect.

Data quality upstream#

Sambasivan et al. (2021), "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI, documents 92% of interviewed practitioners experiencing compounding downstream failures originating in data collection and annotation. Cascades are described as opaque, diagnosed late, and typically triggered by events far upstream of the model.

Cost accounting#

Three numbers that are routinely underestimated when planning ML systems:

• Inference dominates lifetime compute. A model trained once and served for a year at even modest traffic spends far more FLOPs on inference than on training. Optimise training cost only after inference cost is understood.
• Retraining is neither free nor reliably beneficial. Retraining cadence should be derived from measured performance decay, not from a calendar.
• Human labelling is often the binding constraint. Label latency, not model latency, sets how fast you can detect and correct a regression.

Papers#

• Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015 — papers.nips.cc/paper/5656
• Breck et al., The ML Test Score, IEEE Big Data 2017 — research.google/pubs/pub46555
• D'Amour et al., Underspecification Presents Challenges for Credibility in Modern Machine Learning, 2020 — arxiv.org/abs/2011.03395
• Sambasivan et al., Data Cascades in High-Stakes AI, CHI 2021 — research.google/pubs/pub49953
• Shimodaira, Improving predictive inference under covariate shift, JSPI 2000 — the origin of importance-weighted correction.
• Paleyes, Urma and Lawrence, Challenges in Deploying Machine Learning: a Survey of Case Studies, 2020 — arxiv.org/abs/2011.09926

What to learn next#

• Experiment tracking — how to make a good result repeatable next month.
• Docker for ML — freezing the library versions so a different machine cannot change the answer.
• Monitoring and model drift — catching the silent failure.