Tokenization

Tokenization is cutting text into small pieces called tokens, because a model can only work with a fixed list of known pieces.

By Pranay Mahendrakar
16 min read
3 reading levels
Published 20 Aug 2026

Read these first

On this page 5

Why it exists
How it works
Where you have already seen it
Remember this
What to learn next

One lesson, three depths. Pick the one that fits you today — you can switch any time.

Beginner — No maths. Plain English.

Tokenization is cutting text into small pieces, called tokens, so that a model has something countable to work with.

Think about cooking. Before anything goes into the pan, you chop. A whole onion is useless to a recipe. Chopped onion is not.

The size of the chop matters too. Chop too big and the onion never cooks through. Chop too fine and it turns to paste. Text works exactly the same way, and finding the right chop size took researchers years.

Why it exists

A model cannot handle an infinite list. It needs a fixed vocabulary — a numbered list of every piece it is allowed to see. Every piece of text gets turned into numbers from that list.

So how do you chop? People tried three ways, and two of them fail.

Chop into whole words. This feels natural. It breaks the first time someone types a word you never listed. New slang, a person's name, a typo, a product code — all become "unknown". A model that reads "unknown" learns nothing from it.

It also breaks across languages. Hindi, Tamil and Turkish glue meaning onto the end of words. One English word can match a hundred word-forms in Tamil. Your list explodes.

Chop into single letters. Now nothing is ever unknown, because there are only so many letters. But a short sentence becomes hundreds of pieces. The model has to work far harder. It also has to rebuild the idea of a word from scratch each time.

Chop into pieces in between. This is the answer that won. Common words stay whole. Rare words break into familiar chunks.

How it works

The chopping rules are not written by a person. They are learned by counting.

The method starts with single letters, then repeatedly glues together the pair that appears most often. Do that a few thousand times and you get a vocabulary of useful chunks.

  Start:      u n b e l i e v a b l e

  After learning from lots of text, common pairs
  have been glued into bigger pieces:

  Result:     [ un ] [ believ ] [ able ]
                |        |         |
                v        v         v
              [ 402 ]  [ 8811 ]  [ 712 ]     <- the model sees these

Notice what happened. The model has never seen "unbelievable" as a whole. It still handles it, because it knows "un", "believ" and "able" from other words.

A common English word like "the" stays in one piece. A rare word like "Mahendrakar" might break into four or five pieces. Both work.

Where you have already seen it

Chatbot pricing. Chat services charge per token, not per word. A thousand tokens is roughly 750 English words.
Length limits. When a chatbot says your document is too long, it is counting tokens.
The letter-counting failure. Ask a chatbot how many times the letter "r" appears in "strawberry" and it may get it wrong. The model never saw the letters. It saw two or three chunks. Counting letters inside a chunk is genuinely awkward for it.
Non-English costs more. The same sentence in Hindi or Kannada often costs two to four times more tokens than in English. Same meaning, higher bill, slower reply.

That last point is not a small technical detail. A student in Bengaluru writing in Kannada pays more than a student writing in English. Same question, same answer, higher bill. Most tokenizers were built by counting English text, so English got the efficient chunks.

Remember this

A token is a chunk of text. Sometimes a whole word, often a piece of one.
Whole-word chopping breaks on new words. Letter-by-letter chopping makes sequences far too long. Subword chunks are the compromise everyone uses.
Token count decides your cost, your speed and your length limit — and it is unfair across languages.

What to learn next

Embeddings — how each token becomes numbers that carry meaning.
What is NLP? — the bigger picture this step sits inside.
Context windows — why token count is a hard limit.

Developer — Code and libraries.

Two things are worth doing here. First, see why splitting on spaces fails. Second, build a real subword tokenizer from scratch, so the thing stops being magic.

Both run on plain Python. No model download, no network call.

Setup

Nothing to install. The examples below use only the standard library.

bash

python --version    # 3.9 or newer

Splitting on spaces, and why it is not enough

naive.py

text = "AI isn't magic. It's maths, data and electricity."
pieces = text.split()

print(pieces)
print(len(pieces), "chunks")

Output

['AI', "isn't", 'magic.', "It's", 'maths,', 'data', 'and', 'electricity.']
8 chunks

Read the output slowly. Three separate problems are visible.

'magic.' and 'electricity.' carry a full stop inside them. To your vocabulary, magic and magic. are two unrelated entries. You have doubled your vocabulary and halved your evidence for each entry.

"isn't" stayed glued. Is that one token or two? Both answers are defensible, and different tokenizers choose differently.

And this whole approach assumes spaces separate words. Chinese and Japanese are written without spaces. Splitting on whitespace returns the entire sentence as one chunk.

Byte pair encoding, built from scratch

This is the algorithm behind the tokenizers in GPT, Llama and most open models. It is about thirty lines.

The idea: start with single characters, count every neighbouring pair, glue the most common pair into one piece, and repeat.

tiny_bpe.py

from collections import Counter

# Four words and how often each appears. </w> marks the end of a word,
# so "low" as a whole word stays distinct from "low" inside "lower".
words = {
    ("l", "o", "w", "</w>"): 5,
    ("l", "o", "w", "e", "r", "</w>"): 2,
    ("n", "e", "w", "e", "s", "t", "</w>"): 6,
    ("w", "i", "d", "e", "s", "t", "</w>"): 3,
}

def count_pairs(words):
    pairs = Counter()
    for word, freq in words.items():
        for i in range(len(word) - 1):
            pairs[(word[i], word[i + 1])] += freq   # weighted by word frequency
    return pairs

def apply_merge(words, a, b):
    merged = {}
    for word, freq in words.items():
        out, i = [], 0
        while i < len(word):
            if i < len(word) - 1 and word[i] == a and word[i + 1] == b:
                out.append(a + b)          # two pieces become one piece
                i += 2
            else:
                out.append(word[i])
                i += 1
        merged[tuple(out)] = freq
    return merged

for step in range(4):
    pairs = count_pairs(words)
    a, b = max(pairs, key=pairs.get)       # ties break in first-seen order
    print(f"merge {step + 1}: {a!r} + {b!r}  (seen {pairs[(a, b)]} times)")
    words = apply_merge(words, a, b)

print()
for word in words:
    print(" ".join(word))

Output

merge 1: 'e' + 's'  (seen 9 times)
merge 2: 'es' + 't'  (seen 9 times)
merge 3: 'est' + '</w>'  (seen 9 times)
merge 4: 'l' + 'o'  (seen 7 times)

lo w </w>
lo w e r </w>
n e w est</w>
w i d est</w>

Line by line

The merge list is the tokenizer. Those four lines printed at the top are the entire learned artefact. To tokenize a new word later, you apply the same merges in the same order. A production tokenizer is this list with fifty thousand entries instead of four.

</w> is an end-of-word marker. Without it, the tokenizer cannot tell a piece that ends a word from the same letters in the middle of one. Notice that merge 3 glued est to </w>, producing a suffix piece that only ever appears at the end of a word. That is the algorithm discovering the English suffix -est by counting alone. Nobody told it about superlatives.

max(pairs, key=pairs.get) breaks ties by first-seen order. Three different pairs each appeared 9 times at step one. Real implementations pin down tie-breaking explicitly, because a different tie rule produces a different vocabulary, and a different vocabulary makes your saved model unusable.

Frequency weighting matters. count_pairs adds the word's frequency, not 1. A pair inside a word that appears six thousand times should outrank a pair inside a word that appears twice.

Common mistakes

Counting characters and calling them tokens. len(text) is not your token count and neither is len(text.split()). For English, tokens land near len(text) / 4. For Devanagari or Tamil script, that estimate can be wrong by a factor of three. Measure with the actual tokenizer your model uses.

Forgetting that leading spaces are part of the token. In GPT-style byte-level tokenizers, "hello" and " hello" are different token IDs. Strip a leading space during preprocessing and your input silently stops matching what the model saw in training.

Mixing tokenizers between training and inference. A saved model and its tokenizer are one unit. Swap in a tokenizer trained on a different corpus and every ID points at the wrong embedding row. The model does not crash. It produces confident nonsense, which is far worse.

In this toy code, colliding keys overwrite. merged[tuple(out)] = freq assumes no two source words collapse to the same tuple. A real implementation sums the frequencies instead. It does not bite in this example. It will bite on a real corpus.

Try it yourself

Change range(4) to range(10) and re-run. Here is what you get:

Output

merge 1: 'e' + 's'  (seen 9 times)
merge 2: 'es' + 't'  (seen 9 times)
merge 3: 'est' + '</w>'  (seen 9 times)
merge 4: 'l' + 'o'  (seen 7 times)
merge 5: 'lo' + 'w'  (seen 7 times)
merge 6: 'n' + 'e'  (seen 6 times)
merge 7: 'ne' + 'w'  (seen 6 times)
merge 8: 'new' + 'est</w>'  (seen 6 times)
merge 9: 'low' + '</w>'  (seen 5 times)
merge 10: 'w' + 'i'  (seen 3 times)

low</w>
low e r </w>
newest</w>
wi d est</w>

Look at merge 8. The whole word newest</w> has become a single token.

That is a real failure mode, and it is worth understanding. With only four words in the corpus, BPE runs out of shared structure and starts memorising entire words. You have quietly rebuilt the word-level tokenizer that subwords were meant to replace.

Subword tokenization needs a large and varied corpus to work. On four words it degenerates. On four billion words the same algorithm produces pieces that get reused across thousands of different words, which is the entire point.

Then add a fifth word — ("s", "l", "o", "w", "</w>"): 4 — and re-run with range(4):

Output

merge 1: 'l' + 'o'  (seen 11 times)
merge 2: 'lo' + 'w'  (seen 11 times)
merge 3: 'low' + '</w>'  (seen 9 times)
merge 4: 'e' + 's'  (seen 9 times)

Every merge changed. One extra word reordered the entire vocabulary.

Tokenizer vocabularies are not stable under small corpus changes. That is why a tokenizer is trained once, frozen before pre-training begins, and shipped with the model forever.

What to learn next

Embeddings — turning these token IDs into meaning.
Hugging Face — using production tokenizers.
Context windows — where token count becomes a hard wall.

Researcher — Mathematics and papers.

The problem being solved

Given a corpus over an alphabet, choose a vocabulary V and a segmentation function s: string -> V* that trades two costs against each other.

Large V gives short sequences, so attention cost O(n^2) falls, but the embedding and softmax matrices grow as O(|V| * d), where d is model width.
Small V gives cheap embeddings but long sequences and a harder modelling task.

Fertility is the standard measurement: mean tokens per word, or per character, on a held-out corpus. Lower is better within a fixed |V|.

Frontier models sit around |V| between 32k and 256k. The recent drift upward is driven by multilingual fertility and by the fact that a wider vocabulary amortises well when d is already large.

Byte pair encoding

Originally a compression algorithm (Gage, 1994), brought to NMT by Sennrich, Haddow & Birch (2016).

text

V <- set of base symbols (characters, or all 256 bytes)
repeat k times:
    (a, b) <- argmax over adjacent symbol pairs of count(a, b)
    V <- V union {ab}
    replace every occurrence of (a, b) with ab

k is the number of merges, so |V| = |base| + k.
count(a, b) is corpus frequency of the ordered adjacent pair.

Training cost with the naive implementation is O(k * N), where N is corpus length in symbols. Practical implementations keep a priority queue over pair counts and an index of affected positions, giving roughly O(N log N) overall. Encoding a new string with a learned merge list is O(m log m) for a string of m symbols using a priority queue over merge ranks.

BPE is greedy and provides no optimality guarantee for the resulting segmentation. It optimises merge-time frequency, not held-out likelihood.

WordPiece

Schuster & Nakajima (2012), popularised by Wu et al. (2016) and used by BERT. Structurally identical to BPE, with a different merge criterion. Rather than raw frequency, it merges the pair that maximally increases the likelihood of the corpus under a unigram language model:

text

score(a, b) = count(a, b) / (count(a) * count(b))

count(x) is the corpus frequency of symbol x.

This is pointwise mutual information up to a monotone transform. It penalises pairs that are frequent only because both parts are individually frequent, which is exactly the failure mode of raw-frequency BPE on function words.

Unigram language model

Kudo (2018). This one inverts the direction: start from a large candidate vocabulary and prune it.

The model treats a segmentation x = (x_1, ..., x_m) as independent draws:

text

P(x) = product over i = 1..m of p(x_i),    subject to sum over all v in V of p(v) = 1

p(x_i) is the learned unigram probability of subword x_i.
V is the current vocabulary.

Training alternates two steps, an EM procedure:

E-step. For each word, compute the marginal or the best segmentation under current p, using the Viterbi algorithm over the lattice of possible splits. Cost is O(m * L) per word for word length m and maximum piece length L.
M-step. Re-estimate p from expected counts, then drop the lowest-contribution 10 to 20 percent of pieces by likelihood loss, until |V| reaches the target.

Two properties follow that BPE lacks. The vocabulary is chosen by a held-out-comparable likelihood objective rather than by greedy merge order. And because the model is probabilistic, you can sample alternative segmentations at training time. This is subword regularisation. It acts as data augmentation and consistently helps low-resource translation.

SentencePiece (Kudo & Richardson, 2018) is the implementation, not the algorithm. It offers both BPE and unigram. Its real contribution is treating the input as a raw Unicode stream, with no language-specific pre-tokenization and a visible space marker. This is what makes the same code work for Japanese and for English.

Byte-level fallback

Radford et al. (2019), GPT-2, apply BPE over UTF-8 bytes rather than Unicode code points. The base vocabulary is exactly 256 symbols, so the out-of-vocabulary rate is structurally zero for any input. No <UNK> token is needed at all.

The cost is that a single Devanagari code point occupies 3 UTF-8 bytes. Before any merges are learned, Indic text therefore starts at three times the sequence length of ASCII. Merges recover much of this only if the training corpus contained enough of that script.

Measured consequences

Petrov, La Malfa, Torr & Bibi (2023), Language Model Tokenizers Introduce Unfairness Between Languages (arXiv:2305.15425), measured tokenized length for parallel corpora across languages. The ratio between the longest and shortest tokenization of identical content exceeded 15 for some tokenizer and language pairs.

The consequences are concrete and compound:

Effect	Mechanism
Higher price	Billing is per token
Higher latency	Decoding time is linear in output tokens
Smaller usable context	The window is measured in tokens
Worse quality	Fewer characters per position means less signal per attention operation

Arithmetic is a second measured failure. Tokenizers that split numbers inconsistently — "1234" as one piece but "12345" as two — degrade multi-digit arithmetic. Right-to-left digit grouping and forced single-digit tokenization are the mitigations now used in several model families.

Character-level tasks fail for the same structural reason. A model that receives "strawberry" as two or three opaque IDs has no direct access to its letters and must have memorised the spelling separately.

Key references

Gage, P. (1994). A New Algorithm for Data Compression. C Users Journal 12(2). The original BPE.
Schuster, M. & Nakajima, K. (2012). Japanese and Korean Voice Search. ICASSP. WordPiece.
Sennrich, R., Haddow, B. & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909
Wu, Y. et al. (2016). Google's Neural Machine Translation System. arXiv:1609.08144
Kudo, T. (2018). Subword Regularization. arXiv:1804.10959
Kudo, T. & Richardson, J. (2018). SentencePiece. arXiv:1808.06226
Petrov, A. et al. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. arXiv:2305.15425

Current state and open problems

Tokenization is the least principled component of the modern stack, and it is load-bearing. The vocabulary is frozen before pre-training, cannot be changed afterwards without retraining the embedding and output layers, and silently determines cost fairness across languages.

Three directions are active.

Tokenizer-free models. ByT5 (Xue et al., 2022) operates directly on bytes and removes the problem entirely, at the price of far longer sequences. MEGABYTE (Yu et al., 2023) and the byte-latent line of work use hierarchical patching to make byte-level sequences affordable. None has displaced subword tokenization at frontier scale.

Learned or dynamic segmentation. Making the segmentation itself differentiable, or letting entropy decide patch boundaries, so compression adapts to content rather than to a frozen merge list.

Vocabulary transplantation. Swapping a tokenizer after pre-training, re-initialising affected embedding rows from the old ones, then briefly continuing training. This works well enough to be practically useful for adapting English-centric models to Indic scripts, and it is not yet well understood theoretically.

The honest summary: everyone agrees tokenization is the wrong abstraction, and nothing that removes it has yet been worth the compute at scale.

What to learn next

Embeddings — what the token IDs index into.
Attention — where the O(n^2) sequence-length cost lands.
Word2Vec — the objective that first made token vectors useful.

🟢 Beginner — No maths. Plain English.

Why it exists#

How it works#

Where you have already seen it#

Remember this#

What to learn next#

🟡 Developer — Code and libraries.

Setup#

Splitting on spaces, and why it is not enough#

Byte pair encoding, built from scratch#

Line by line#

Common mistakes#

Try it yourself#

What to learn next#

🔴 Researcher — Mathematics and papers.

The problem being solved#

Byte pair encoding#

WordPiece#

Unigram language model#

Byte-level fallback#

Measured consequences#

Key references#

Current state and open problems#

What to learn next#

Beginner — No maths. Plain English.

Why it exists

How it works

Where you have already seen it

Remember this

What to learn next

Developer — Code and libraries.

Setup

Splitting on spaces, and why it is not enough

Byte pair encoding, built from scratch

Line by line

Common mistakes

Try it yourself

What to learn next

Researcher — Mathematics and papers.

The problem being solved

Byte pair encoding

WordPiece

Unigram language model

Byte-level fallback

Measured consequences

Key references

Current state and open problems

What to learn next