Golden outputs and regression tests

Setup#

pip install pytest

The pipeline step being frozen#

A small, deterministic text-cleaning function — the kind that runs before almost every text model sees a row. Deterministic means the same input always gives the same output, which is exactly what makes golden testing possible here.

"""Deterministic text-cleaning used before every model in this pipeline sees a row."""
import re

_WHITESPACE = re.compile(r"\s+")
_PUNCT = re.compile(r"[^\w\s]")


def clean(text: str) -> str:
    text = text.strip().lower()
    text = _PUNCT.sub("", text)
    text = _WHITESPACE.sub(" ", text)
    return text

Freezing the golden file#

Run once, by a human, and checked into version control. This script is never part of the test suite itself.

"""Run once by a human to freeze today's known-good output. Never run inside a test."""
import json
from clean import clean

INPUTS = [
    "  Hello,   World!  ",
    "Great product!!! Would buy AGAIN.",
    "N0T-a-typical-sentence...",
    "Multiple    spaces   here",
    "Already clean",
]

golden = {text: clean(text) for text in INPUTS}
with open("golden_clean.json", "w") as f:
    json.dump(golden, f, indent=2, sort_keys=True)
print(json.dumps(golden, indent=2))

python make_golden.py

{
  "  Hello,   World!  ": "hello world",
  "Great product!!! Would buy AGAIN.": "great product would buy again",
  "N0T-a-typical-sentence...": "n0tatypicalsentence",
  "Multiple    spaces   here": "multiple spaces here",
  "Already clean": "already clean"
}

Notice the third line: hyphens are stripped as punctuation with no space left behind, so "N0T-a-typical-sentence..." becomes one run-on word, "n0tatypicalsentence". That may or may not be what you want — but the golden file does not judge, it only remembers. Whether this is correct is a decision a human makes once, when freezing the file.

The regression test#

import json

import pytest

from clean import clean

with open("golden_clean.json") as f:
    GOLDEN = json.load(f)


@pytest.mark.parametrize("raw_text,expected", sorted(GOLDEN.items()))
def test_matches_the_frozen_output(raw_text, expected):
    assert clean(raw_text) == expected

pytest test_clean_regression.py -q

.....                                                                    [100%]
5 passed in 0.02s

Now break something on purpose#

Someone tries to tidy the punctuation regex, meaning to keep only letters and spaces, and writes [^a-z\s] instead of [^\w\s]:

# in clean.py
_PUNCT = re.compile(r"[^a-z\s]")

....F                                                                    [100%]
================================== FAILURES ===================================
_ test_matches_the_frozen_output[N0T-a-typical-sentence...-n0tatypicalsentence] _

raw_text = 'N0T-a-typical-sentence...', expected = 'n0tatypicalsentence'

    @pytest.mark.parametrize("raw_text,expected", sorted(GOLDEN.items()))
    def test_matches_the_frozen_output(raw_text, expected):
>       assert clean(raw_text) == expected
E       AssertionError: assert 'ntatypicalsentence' == 'n0tatypicalsentence'
E
E         - n0tatypicalsentence
E         ?  -
E         + ntatypicalsentence

test_clean_regression.py:13: AssertionError
=========================== short test summary info ===========================
FAILED test_clean_regression.py::test_matches_the_frozen_output[N0T-a-typical-sentence...-n0tatypicalsentence]
1 failed, 4 passed in 0.28s

[^a-z\s] — letters and spaces only — silently also strips digits, because \w (the original pattern) meant "letters, digits, and underscore" and a-z does not. Four of the five golden cases had no digits, so they still pass. Only the one case with a 0 in it catches the change. This is exactly why a golden set needs a handful of varied examples, not one — a single easy case would have missed this entirely.

Updating a golden file on purpose#

When the new output is genuinely the intended one, re-run make_golden.py, review the diff in the saved JSON file like any other code change, and commit it with a reason written in the commit message. That review step — a human looking at exactly what changed and why — is the entire point. A test that regenerates its own golden file on failure has no review step, and stops meaning anything.

Exact match versus tolerance#

String and integer outputs, as above, compare with plain equality. Floating-point outputs from a model rarely will, because the exact bits can shift by a tiny amount between library versions or hardware. Use a tolerance instead:

import math
assert math.isclose(current_score, golden_score, rel_tol=1e-4)

Set the tolerance loose enough to absorb harmless floating-point noise, and tight enough that a real change in behaviour still fails the test. There is no universal number — start tight, and loosen it only when a passing change genuinely needs it.

Common mistakes#

Storing the golden file somewhere it will not be reviewed. If it lives outside version control, nobody sees it change, and the whole safety net disappears silently.

One golden example. As shown above, the digit-stripping bug was invisible in four of five cases. Cover the boundary cases you actually care about — empty strings, punctuation, numbers, unusual characters — not only the easy ones.

Regenerating on every failure "to unblock CI". This is the single most common way golden tests stop protecting anything. If regenerating feels urgent, that pressure is exactly why the review step needs to survive it.

Testing golden outputs for something that is meant to change often. A golden test fits a deterministic, stable transform. A model's raw prediction on live, growing data is the wrong target — that belongs in monitoring, not a fixed golden file.

Try it yourself#

Add a sixth input to INPUTS containing an emoji, regenerate the golden file, and look at what clean() actually does to it. Decide, on purpose, whether that is the behaviour you want — that decision, made once and written down, is the entire value of this pattern.

What to learn next#

• Testing output you cannot predict — what to do when the output is not the same every time, so exact golden comparison stops working.
• CI/CD for machine learning — running this exact regression test on every push, automatically.
• Behavioural tests for models — checking specific abilities on purpose, the layer above a golden set.