Learn AI Create the Future — with Pranay
Main site

Tokeniser Internals

WordPiece

WordPiece merges the pair that is most surprising together, not the pair that is commonest. That one change in scoring is why BERT's tokens look different from GPT's.

Read these first

On this page 7
  1. The scoring rule, without any maths
  2. Why this changes the result
  3. The continuation marker
  4. How the encoder works
  5. Where you have seen this
  6. Remember this
  7. What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

WordPiece merges the pair whose two halves belong together, rather than the pair that appears most often.

Think about the phrase "chai" and "wala". Both are common on their own. Now think about "Bengaluru", where "Beng" and "aluru" almost never appear apart.

The second pair is more informative. Seeing "Beng" tells you a lot about what comes next. Seeing "chai" tells you much less about the next word.

WordPiece scores exactly that. It asks: do these two pieces occur together far more than their individual popularity would predict?

The scoring rule, without any maths

Take a candidate pair. Count how often the pair appears. Then count how often each half appears on its own.

If both halves are everywhere, being adjacent is unremarkable. If both halves are rare and yet they keep appearing together, that pairing is meaningful.

WordPiece prefers the second. BPE would prefer whichever pair appeared more often.

Why this changes the result

BPE glues common things to common things, which quickly produces frequent function words as tokens.

WordPiece finds cohesive units. It tends to surface morphemes: prefixes, roots and suffixes that carry meaning. That suits a model like BERT, which is asked to understand text rather than continue it.

The continuation marker

WordPiece marks any piece that is not the start of a word with two hashes.

   "playing"     ->   play  ##ing
   "unhappiness" ->   un  ##happiness
   "the"         ->   the

   "##ing" and "ing" are different entries in the vocabulary

That marker is doing real work. It lets the model tell "ing" starting a word from "ing" ending one, without a separate boundary symbol.

How the encoder works

Encoding is different from BPE, and simpler. Start at the left of a word. Take the longest piece in the vocabulary that matches from there. Move on, and repeat with the continuation marker attached.

There is one harsh rule. If any part of the word cannot be matched, the entire word becomes the unknown token. Not the failing part. The whole word.

That is why BERT-family models struggle with unusual scripts, emoji and unexpected symbols. One unmatched character discards everything around it.

Where you have seen this

  • BERT, and every model built on it, which is a very long list.
  • DistilBERT, ELECTRA, and most encoder-only classifiers.
  • Any time you see tokens printed with ## in front of them.
  • Older search-ranking and text-classification systems.

Remember this

  • WordPiece merges pairs whose halves rarely occur apart, not pairs that are common on their own.
  • ## marks a piece that continues a word rather than starting one.
  • Encoding is longest-match from the left, and one unmatched character makes the whole word unknown.

What to learn next

Developer — Code and libraries.

Setup

bash
python3 --version

No libraries needed. The training rule and the encoder are both short.

Training and encoding, from scratch

wordpiece.py
from collections import Counter

CORPUS = ("low low low low low lower lower newest newest newest "
          "newest newest newest widest widest widest new new newer")

def split_word(w):
    """Every character after the first is marked as a continuation."""
    return [w[0]] + ["##" + c for c in w[1:]]

def join(a, b):
    return a + b[2:] if b.startswith("##") else a + b

words = Counter(tuple(split_word(w)) for w in CORPUS.split())

def best_pair(words):
    pair_freq, piece_freq = Counter(), Counter()
    for word, n in words.items():
        for p in word:
            piece_freq[p] += n
        for a, b in zip(word, word[1:]):
            pair_freq[(a, b)] += n
    # WordPiece picks the pair whose halves are surprising together, not the commonest pair.
    scored = {p: c / (piece_freq[p[0]] * piece_freq[p[1]]) for p, c in pair_freq.items()}
    if not scored:
        return None, 0, 0
    best = max(scored, key=lambda p: (scored[p], p))
    return best, scored[best], pair_freq[best]

def apply(words, pair):
    a, b = pair
    out = Counter()
    for word, n in words.items():
        new, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and word[i] == a and word[i + 1] == b:
                new.append(join(a, b)); i += 2
            else:
                new.append(word[i]); i += 1
        out[tuple(new)] += n
    return out

# Start from every single character, then keep every piece the trainer ever builds.
vocab = {p for w in words for p in w}
print("pair chosen at each step, with its score and its raw count:")
for step in range(8):
    pair, score, freq = best_pair(words)
    if pair is None:
        break
    print(f"  {step+1:>2}. {pair[0]!r} + {pair[1]!r} -> {join(*pair)!r:<12}"
          f" score {score:.5f}  raw count {freq}")
    vocab.add(join(*pair))
    words = apply(words, pair)

vocab |= {"##" + c for c in "abcdefghijklmnopqrstuvwxyz"}
print(f"\nvocabulary size after 8 merges: {len(vocab)}")

def encode(word, vocab, unk="[UNK]"):
    """Longest match from the left. This is the whole encoder."""
    out, start, first = [], 0, True
    while start < len(word):
        end = len(word)
        piece = None
        while start < end:
            cand = word[start:end] if first else "##" + word[start:end]
            if cand in vocab:
                piece = cand; break
            end -= 1
        if piece is None:
            return [unk]                      # one bad character kills the whole word
        out.append(piece); start = end; first = False
    return out

print("\nencoding:")
for w in ["low", "lowest", "newest", "newer", "widen", "zebra"]:
    print(f"  {w:>7} -> {encode(w, vocab)}")
Output
pair chosen at each step, with its score and its raw count:
   1. 'w' + '##i' -> 'wi'         score 0.33333  raw count 3
   2. 'wi' + '##d' -> 'wid'        score 0.33333  raw count 3
   3. 'l' + '##o' -> 'lo'         score 0.14286  raw count 7
   4. '##s' + '##t' -> '##st'       score 0.11111  raw count 9
   5. 'lo' + '##w' -> 'low'        score 0.06250  raw count 7
   6. 'wid' + '##e' -> 'wide'       score 0.04762  raw count 3
   7. 'wide' + '##st' -> 'widest'     score 0.11111  raw count 3
   8. 'n' + '##e' -> 'ne'         score 0.05556  raw count 9

vocabulary size after 8 merges: 37

encoding:
      low -> ['low']
   lowest -> ['low', '##e', '##st']
   newest -> ['ne', '##w', '##e', '##st']
    newer -> ['ne', '##w', '##e', '##r']
    widen -> ['wide', '##n']
    zebra -> ['[UNK]']

Reading that output, against BPE

Compare merge 1 with BPE's merge 1 on the same corpus. BPE picked w + e, count 9. WordPiece picked w + ##i, count 3. WordPiece chose the pair that appeared three times over one that appeared nine times.

The reason is in the score column. i occurs only in widest, so w and ##i together are far more informative than their individual counts suggest. That is the entire difference between the two algorithms.

Steps 1, 2, 6 and 7 build widest piece by piece — wi, wid, wide, widest — from a word appearing three times, while newest at six occurrences is still in fragments. Rarity plus cohesion beats raw frequency.

Step 4 merged two continuation pieces, ##s + ##t into ##st. Both halves already had the marker, so the result keeps it. join handles this by stripping the second piece's marker only.

lowest encodes as low ##e ##st although the trainer never saw the word. Longest-match takes low, cannot extend, then continues with the marker attached.

zebra returns [UNK], and every character in it is in the vocabulary. Look at the encoder: z is a word-initial candidate, and only ##z was added by the alphabet line. There is no bare z entry, so the first match fails and the whole word is discarded.

That is not a contrived failure. It is exactly how BERT loses entire words containing one unusual character, and it is the strongest practical argument for byte-level tokenisers.

Checking against a real implementation

The tokenizers library trains WordPiece with the same scoring rule and no downloads:

wordpiece_lib.py
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders

tok = Tokenizer(models.WordPiece(unk_token="[UNK]"))
tok.pre_tokenizer = pre_tokenizers.Whitespace()
tok.decoder = decoders.WordPiece()
tok.train_from_iterator(
    ["low low low low low lower lower newest newest newest",
     "newest newest newest widest widest widest new new newer"],
    trainers.WordPieceTrainer(vocab_size=40, min_frequency=1,
                              special_tokens=["[UNK]"], show_progress=False))

print("vocabulary size:", tok.get_vocab_size())
for w in ["low", "lowest", "newest", "widen"]:
    print(f"  {w:>7} -> {tok.encode(w).tokens}")
Output
vocabulary size: 32
      low -> ['low']
   lowest -> ['low', '##est']
   newest -> ['newest']
    widen -> ['[UNK]']

Different vocabulary size and different splits, because the library trains to a target size rather than a fixed merge count and seeds the alphabet from the corpus alone.

Note widen returns [UNK] here where our hand-written version produced wide ##n. Print tok.get_vocab() and the reason is visible: the vocabulary contains n but not ##n, because no word in this corpus has an n anywhere except at the start. The encoder gets as far as wi ##d ##e, then needs ##n, fails, and discards the entire word.

Same all-or-nothing rule, a different gap. On a two-sentence corpus this happens constantly. On a real corpus it happens on unusual scripts, emoji and symbols, which is the failure that matters.

Common mistakes

Assuming ##ing and ing are the same token. They are separate vocabulary entries with separate embeddings. Building a vocabulary by hand and forgetting the marked variants gives an encoder that fails on most words.

Stripping ## before feeding text back to the model. The marker is part of the token string. Removing it changes the id.

Treating [UNK] as rare. On text outside the training domain, especially non-Latin scripts, [UNK] rates can be high, and every unknown discards a whole word. Measure your unknown rate on real data before assuming coverage is fine.

Confusing WordPiece with BPE because both merge pairs. The scoring rule and the encoder both differ. BPE replays an ordered merge list; WordPiece does greedy longest match against a flat vocabulary and needs no merge list at inference.

Try it yourself

Add a bare z to the vocabulary and re-encode zebra. Then work out which other single characters are missing from the word-initial set. That gap between "characters in the vocabulary" and "characters usable at the start of a word" is where most [UNK] results come from.

What to learn next

Researcher — Mathematics and papers.

Origin and objective

WordPiece comes from Schuster and Nakajima (2012), for Japanese and Korean voice search, and was popularised by Wu et al. (2016) in Google's neural machine translation system and then by Devlin et al. (2019) in BERT.

The stated objective is to choose merges that maximise the likelihood of the training corpus under a unigram language model over the vocabulary. For a corpus $C$ segmented into pieces, the log-likelihood under independence is

$$ \log P(C) = \sum_{p \in C} \log P(p) $$

Merging pieces $a$ and $b$ into $ab$ changes this. Under a maximum-likelihood estimate with counts, the gain from merging is approximately

$$ \Delta \approx \mathrm{count}(ab) \log \frac{\mathrm{count}(ab)}{\mathrm{count}(a)\,\mathrm{count}(b)} $$

The multiplier $\mathrm{count}(ab)$ weights by how often the merge applies; the logarithm is pointwise mutual information between the two pieces. Practical implementations, including huggingface/tokenizers, select on the ratio

$$ \mathrm{score}(a,b) = \frac{\mathrm{count}(ab)}{\mathrm{count}(a)\cdot\mathrm{count}(b)} $$

which is the developer implementation above. It preserves the ordering of the PMI term and drops the frequency weighting, which is why it can prefer a pair occurring three times over one occurring nine times.

Comparison with BPE's criterion

BPE maximises $\mathrm{count}(ab)$. WordPiece maximises approximately $\mathrm{count}(ab) / (\mathrm{count}(a)\mathrm{count}(b))$.

The denominator is the entire difference. BPE is biased toward merging high-frequency pieces with each other, since their co-occurrence counts are large by construction. WordPiece normalises that away and surfaces cohesive units.

Empirically this yields more morpheme-like pieces. Bostrom and Durrett (2020), Byte Pair Encoding is Suboptimal for Language Model Pretraining, compare BPE against unigram tokenisation under matched conditions and find unigram produces segmentations better aligned with morphology and yields better downstream results. Their argument applies with reduced force to WordPiece, which shares the likelihood motivation but keeps the greedy bottom-up construction.

Inference: greedy longest-match-first

Encoding is not a replay of merges. For a word $w$:

  1. Find the longest prefix of $w$ present in the vocabulary.
  2. Emit it, remove it, prefix the remainder with ##.
  3. Repeat until $w$ is consumed, or emit [UNK] for the whole word if any step fails.

Complexity is $O(m^2)$ per word of length $m$ in the naive form, and $O(m)$ with a trie. The all-or-nothing [UNK] rule is a property of the reference implementation, not a necessity of the algorithm, and it is the source of BERT's poor coverage on unseen scripts.

Song et al. (2021), Fast WordPiece Tokenization, give a linear-time algorithm using a modified Aho-Corasick automaton, reporting 8.2x speedup over the reference single-word implementation and 5.1x end to end.

Determinism, and the absence of regularisation

Like BPE, WordPiece assigns exactly one segmentation per word. It defines no distribution over segmentations, so it admits no direct analogue of subword regularisation. BPE-dropout has no clean WordPiece counterpart, because there is no merge sequence to drop from at inference.

Current usage

WordPiece is dominant in encoder-only models of the BERT lineage: BERT, DistilBERT, ELECTRA, MobileBERT, and the many domain-specific BERT variants. It is essentially absent from decoder-only generative models released after 2021, which use byte-level BPE or SentencePiece-unigram. The reasons are coverage — the [UNK] behaviour is unacceptable for open-ended generation — and the ecosystem's shift to byte-level guarantees.

Papers

What to learn next