Building a pretraining corpus

Deduplication is the highest-leverage step, and it is measured#

Lee et al., 2022 (Deduplicating Training Data Makes Language Models Better) is the reference result. On C4 they found a single 61-word sequence repeated 61,036 times. After deduplication:

• Models emit memorised training text at least 10× less often.
• Held-out perplexity improves, despite training on strictly less data.
• Train–test overlap, which inflates evaluation numbers, drops sharply.

Two algorithms are used together. Exact substring matching via a suffix array finds every repeated span above a length threshold (they used 50 tokens) in $O(n \log n)$. MinHash + LSH finds document-level near-duplicates.

MinHash estimates the Jaccard similarity $J(A,B) = |A \cap B| / |A \cup B|$ of shingle sets. For a random permutation $\pi$,

$$ \Pr[\min \pi(A) = \min \pi(B)] = J(A, B) $$

so the fraction of matching entries across $k$ independent hashes is an unbiased estimator of $J$ with standard error $\approx 1/\sqrt{k}$. At $k=64$ that is about 0.125 — which is exactly why the toy pipeline's estimate of 0.30 against a true 0.29 was luckier than it deserved, and why production settings use 128 to 256 permutations.

LSH banding splits $k$ hashes into $b$ bands of $r$ rows ($k = br$). Two documents become candidates if any band matches exactly, giving detection probability

$$ P(J) = 1 - (1 - J^{r})^{b} $$

an S-curve with its steep region near $J \approx (1/b)^{1/r}$. Choosing $b$ and $r$ is choosing the threshold; the nominal similarity cutoff is a consequence, not an input.

Filtering: rules, then classifiers#

Rule-based. C4 (Raffel et al., 2020) kept lines ending in terminal punctuation, dropped pages with fewer than five sentences, dropped lines containing "javascript" or "lorem ipsum", and dropped any page on a bad-word list. Gopher (Rae et al., 2021) added the now-standard set: word count between 50 and 100,000, mean word length between 3 and 10, symbol-to-word ratio under 0.1, at most 90% of lines starting with a bullet, at least 80% of words containing an alphabetic character, and at least two of a stopword list.

Classifier-based. The step change came from training a scorer.

• DCLM (Li et al., 2024) trained a fastText classifier on OpenHermes-2.5 instructions plus high-scoring ELI5 posts as positives, and released 3.8T tokens. DCLM-Baseline 7B trained on 2.6T tokens beat comparable open models at 40% less compute.
• FineWeb-Edu (Penedo et al., 2024) had Llama-3-70B rate 460k pages for educational value, distilled the ratings into a small classifier, and kept the top-scoring 1.3T tokens out of FineWeb's 15T. The subset outperformed the full set on MMLU and ARC at equal token budget — a direct demonstration that fewer, better tokens beat more tokens.
• Nemotron-CC (Su et al., 2024) ensembled a DCLM-style and a FineWeb-Edu-style classifier — noting the two agree on only 368 of their top 1000 domains — and added LLM-rephrased synthetic text to restore the diversity that hard filtering removes. They report 6.3T tokens (4.4T real, 1.9T synthetic) and roughly +5 MMLU over Llama 3.1's data at a 15T-token horizon.

The disagreement between the two classifiers is the interesting number in that list. "Quality" is not one axis, and a single scorer bakes in one definition of it.

Mixture weights#

A corpus is a weighted mixture of sources, and the weights matter as much as the filters. The Pile (Gao et al., 2020) set them by hand. DoReMi (Xie et al., 2023) instead trains a small proxy model with group-DRO to learn weights that minimise worst-case excess loss across domains, then trains the large model with those weights — reporting a 2.6× speed-up to a baseline's final accuracy at 8B scale.

Repeating data#

Muennighoff et al., 2023 (Scaling Data-Constrained Language Models) is the reference for what happens when you run out of text: up to about 4 epochs, repeated tokens are worth nearly as much as new ones; past roughly 16 epochs the marginal value is close to zero. This directly bounds how hard you can filter — a filter that removes 95% of your corpus is only a win if the remaining 5% can be repeated enough times to fill the budget.

Decontamination and memorisation#

Carlini et al., 2021 and Carlini et al., 2023 (Quantifying Memorization Across Neural Language Models) established that memorisation scales with model size, with the number of duplicates of a sequence, and with prompt length. Deduplication is therefore a privacy control as well as a quality one.

Decontamination against benchmarks uses n-gram overlap (13-gram matching is the common convention, from GPT-3) or embedding similarity. Report the method and the removal rate; a paper that reports neither is not making a checkable claim.

Papers and tooling#

• Raffel et al., Exploring the Limits of Transfer Learning (C4), 2020 — arxiv.org/abs/1910.10683
• Gao et al., The Pile, 2020 — arxiv.org/abs/2101.00027
• Rae et al., Scaling Language Models: Methods, Analysis & Insights from Training Gopher, 2021 — arxiv.org/abs/2112.11446
• Lee et al., Deduplicating Training Data Makes Language Models Better, ACL 2022 — arxiv.org/abs/2107.06499
• Penedo et al., The RefinedWeb Dataset for Falcon LLM, 2023 — arxiv.org/abs/2306.01116
• Xie et al., DoReMi, 2023 — arxiv.org/abs/2305.10429
• Soldaini et al., Dolma, 2024 — arxiv.org/abs/2402.00159
• Li et al., DataComp-LM, 2024 — arxiv.org/abs/2406.11794
• Penedo et al., The FineWeb Datasets, 2024 — arxiv.org/abs/2406.17557
• Su et al., Nemotron-CC, 2024 — arxiv.org/abs/2412.02595
• datatrove — github.com/huggingface/datatrove

What to learn next#

• Reading a training run while it happens — what the corpus looks like once it is moving.
• Benchmark contamination in LLMs — the failure this pipeline exists to prevent.
• Bias in datasets — what your filters are quietly deciding.