Testing output you cannot predict

Setup#

pip install pytest

No API key needed for this lesson. The examples below use a small stand-in for a real model call — it picks randomly from a few templates, the same way sampling at a temperature above zero does, but runs offline in milliseconds. Swap reply() and reply_json() for your real API call; the tests do not change shape when you do.

"""A stand-in for a real LLM call: randomised, like sampling at temperature > 0,
but running fully offline so the lesson is CPU-runnable.
"""
import json
import random

OPENERS = ["Sure!", "Absolutely.", "Here you go —", "Happy to help."]
CLOSERS = ["Let me know if that helps.", "Hope that's useful!", "Anything else?"]


def reply(question: str, seed: int | None = None) -> str:
    rng = random.Random(seed)
    opener = rng.choice(OPENERS)
    closer = rng.choice(CLOSERS)
    return f"{opener} The answer to '{question}' involves a few steps. {closer}"


def reply_json(question: str, seed: int | None = None) -> str:
    rng = random.Random(seed)
    payload = {
        "answer": f"a short answer about {question}",
        "confidence": round(rng.uniform(0.6, 0.99), 2),
        "sources": rng.randint(1, 4),
    }
    return json.dumps(payload)

from fake_llm import reply
for s in [1, 2, 3]:
    print(reply("how do I reset my password", seed=s))

Absolutely. The answer to 'how do I reset my password' involves a few steps. Anything else?
Sure! The answer to 'how do I reset my password' involves a few steps. Let me know if that helps.
Absolutely. The answer to 'how do I reset my password' involves a few steps. Anything else?

Three different seeds, three different sentences. This is honest randomness, not a bug — a real model at a non-zero temperature behaves the same way, and no test should assume otherwise.

The tests#

import json

import pytest

from fake_llm import reply, reply_json

BANNED_WORDS = {"guarantee", "promise", "definitely"}


# --- Structural checks: shape and type, never exact text ---

def test_json_reply_has_every_required_field():
    data = json.loads(reply_json("refund policy", seed=1))
    assert set(data.keys()) == {"answer", "confidence", "sources"}


def test_confidence_is_always_a_valid_probability():
    for seed in range(20):
        data = json.loads(reply_json("refund policy", seed=seed))
        assert isinstance(data["confidence"], float)
        assert 0.0 <= data["confidence"] <= 1.0


def test_sources_is_always_a_positive_integer():
    for seed in range(20):
        data = json.loads(reply_json("refund policy", seed=seed))
        assert isinstance(data["sources"], int)
        assert data["sources"] >= 1


# --- Property checks: things true of every sample, not one fixed sample ---

def test_reply_is_never_empty_and_never_absurdly_long():
    for seed in range(20):
        text = reply("how do I reset my password", seed=seed)
        assert 10 <= len(text) <= 400


def test_reply_never_contains_a_banned_word():
    for seed in range(20):
        text = reply("how do I reset my password", seed=seed).lower()
        assert not any(word in text for word in BANNED_WORDS)


def test_reply_stays_on_topic():
    # a weak but cheap groundedness check: the question itself should be echoed
    for seed in range(20):
        text = reply("refund policy", seed=seed)
        assert "refund policy" in text

pytest test_llm_output.py -q

......                                                                   [100%]
6 passed in 0.07s

Line-by-line: why loop over 20 seeds#

Every property test above loops over a range of seeds instead of checking one call. A single lucky sample can pass by accident. Looping over twenty different random outcomes checks the property holds across the distribution of things the model can say, not only the one thing it happened to say this time. This is the direct nondeterministic equivalent of the boundary-value testing from unit testing machine learning code — there you tested specific values on purpose; here you sample many values on purpose, because you cannot list them all by hand.

Now break something on purpose#

Someone renames the confidence field to score while updating the prompt template — a real, common change when a prompt or schema gets "cleaned up":

# in fake_llm.py, reply_json becomes:
payload = {
    "answer": f"a short answer about {question}",
    "score": round(rng.uniform(0.6, 0.99), 2),   # renamed from "confidence"
    "sources": rng.randint(1, 4),
}

FF....                                                                   [100%]
================================== FAILURES ===================================
__________________ test_json_reply_has_every_required_field ___________________

    def test_json_reply_has_every_required_field():
        data = json.loads(reply_json("refund policy", seed=1))
>       assert set(data.keys()) == {"answer", "confidence", "sources"}
E       AssertionError: assert {'answer', 'score', 'sources'} == {'answer', 'c...e', 'sources'}
E         
E         Extra items in the left set:
E         'score'
E         Extra items in the right set:
E         'confidence'
E         Use -v to get more diff

test_llm_output.py:14: AssertionError
________________ test_confidence_is_always_a_valid_probability ________________

    def test_confidence_is_always_a_valid_probability():
        for seed in range(20):
            data = json.loads(reply_json("refund policy", seed=seed))
>           assert isinstance(data["confidence"], float)
E           KeyError: 'confidence'

test_llm_output.py:20: KeyError
=========================== short test summary info ===========================
FAILED test_llm_output.py::test_json_reply_has_every_required_field - Asserti...
FAILED test_llm_output.py::test_confidence_is_always_a_valid_probability - Ke...
2 failed, 4 passed in 0.28s

Even though the content of every reply is different on every run, the structural tests catch the schema change instantly and exactly, in two precisely named failures. This is the real value of structural checks: they are strict about shape while staying completely indifferent to wording.

Common mistakes#

Asserting on exact wording. assert text == "Sure! The answer..." will fail almost immediately, for no real reason, on the very next run. It is testing a property the system was never designed to have.

Testing with one seed and calling it done. One pass proves the property held once. It says nothing about the other 99% of what the model can produce. Loop across many samples, as above.

No temperature-zero option checked anywhere. Most real APIs let you request temperature=0, which makes output far more repeatable — not perfectly bit-identical across model versions, but close. Use it for tests that genuinely need repeatability, and reserve property-based tests for the parts that must handle real variation.

Skipping structural checks because "the model is usually good at JSON". Usually is not always. A schema check costs a few milliseconds and catches malformed output before it reaches a parser that was not written to handle it.

Try it yourself#

Add a test that calls reply() a hundred times with different seeds and asserts that at least two distinct strings appear. This checks the opposite thing from every test above — not that variation stays within limits, but that variation genuinely exists, which would catch a stand-in (or a real API misconfigured to always return a cached answer) that stopped actually sampling.

What to learn next#

• Recording and replaying model API calls — testing code that calls a model without paying for or depending on the real API every run.
• Evaluation gates in CI — turning checks like these into a pass/fail line that blocks a bad release.
• LLM-as-judge in production — using a second model to grade answers a rule cannot check.