Configs instead of constants scattered everywhere

Setup#

pip install scikit-learn

Verified with scikit-learn 1.7.2, numpy 1.26.4, Python 3.10, CPU. Seeded, so your numbers should match.

The pattern, complete in two files#

First, the config — every knob, named:

{
  "n_estimators": 200,
  "max_depth": 4,
  "seed": 0,
  "test_size": 0.25
}

Then a script that reads it, trains, and appends a run record joining config to result:

import json
import pathlib
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

cfg = json.loads(pathlib.Path("config.json").read_text())

X, y = load_iris(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=cfg["test_size"],
                                      random_state=cfg["seed"])
model = RandomForestClassifier(n_estimators=cfg["n_estimators"],
                               max_depth=cfg["max_depth"],
                               random_state=cfg["seed"]).fit(Xtr, ytr)
accuracy = round(model.score(Xte, yte), 4)

record = {"config": cfg, "accuracy": accuracy}
with open("runs.jsonl", "a") as f:
    f.write(json.dumps(record) + "\n")
print("accuracy:", accuracy)
print("recorded:", json.dumps(record))

accuracy: 0.9737
recorded: {"config": {"n_estimators": 200, "max_depth": 4, "seed": 0, "test_size": 0.25}, "accuracy": 0.9737}

The walkthrough#

The script contains no experiment-specific numbers. Change max_depth to 8? Edit the config, not the code. The script becomes stable infrastructure; the config becomes the experiment. Version control now shows experiments as one-line config diffs instead of scattered code edits.

runs.jsonl is the humble hero. One JSON line per run, config and result welded together. Grep it, load it into pandas, never lose an orphan number again. This is a two-line experiment tracker — when runs multiply, graduate to a real one (experiment tracking, MLflow); the principle stays identical.

The seed is a config value. The ten-seed protocol from the previous lesson becomes a shell loop that rewrites one field — no code edits between runs.

What belongs in the config? Everything you might ever vary, plus everything needed to re-run: data paths, split sizes, model settings, seeds, preprocessing choices. When in doubt, put it in. An unused config key is free; a hardcoded constant costs a future afternoon.

Levelling up, when you need it#

• Defaults plus overrides: a base config, with per-experiment files overriding a few keys. Beyond that, tools like Hydra manage config composition — worth it around the time configs start importing each other.
• Record the environment too: library versions and the git commit make the record fully re-runnable. When you cannot reproduce your own number shows the manifest pattern.
• Validate on load: assert required keys exist and values are sane before training, not ninety minutes in. A dataclass with type hints does this almost for free.

Common mistakes#

Command-line arguments as the config. --lr 0.001 --depth 8 --seed 3 works until the command disappears with your terminal history. Arguments are fine as overrides, but the resolved, final settings must be written into the run record either way.

Editing the config after the run. The record's copy is the truth; a mutated config.json rewrites history. Append-only records — as runs.jsonl is — make this mistake harmless.

Half-configs. Nine knobs in the file, the tenth hardcoded — the tenth is always the one that mattered. Grep your script for numeric literals as a config-completeness check.

Configs that contain code. When the config imports libraries and computes values, it inherits code's irreproducibility. Keep it data: JSON, TOML, YAML.

Try it yourself#

Run the script three times, editing max_depth to 2, then 4, then 8 between runs. Load runs.jsonl with pandas (pd.read_json("runs.jsonl", lines=True)) and print a depth-versus-accuracy table. You have built the input for making sense of two hundred runs.

What to learn next#

• Making sense of two hundred runs — what config-keyed records make possible.
• Experiment tracking — the industrial version of runs.jsonl.
• When you cannot reproduce your own number — the failure this lesson quietly prevents.