Counting tokens before you send them

Setup#

pip install tokenizers "transformers==5.6.2"

Written against tokenizers 0.22.2 and transformers 5.6.2. The tokeniser below is trained inline so the script needs no network. For a real model you would load its own tokeniser instead, which is covered further down.

Measuring, rather than estimating#

import random
from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders
from transformers import PreTrainedTokenizerFast

WORDS = ("the a and of to in is was for with on at by from that this it as but "
         "delivery refund warranty invoice payment customer order shipping return "
         "account balance transfer statement").split()
rng = random.Random(0)
CORPUS = [" ".join(rng.choice(WORDS) for _ in range(14)) for _ in range(3000)]

tok = Tokenizer(models.BPE())
tok.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)
tok.decoder = decoders.ByteLevel()
ROLES = ["<pad>", "<|user|>", "<|assistant|>", "<|end|>"]
tok.train_from_iterator(CORPUS, trainers.BpeTrainer(
    vocab_size=600, show_progress=False, special_tokens=ROLES,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet()))
t = PreTrainedTokenizerFast(tokenizer_object=tok, pad_token="<pad>",
                            additional_special_tokens=ROLES[1:])

SAMPLES = {
    "plain English":      "the customer asked about the refund and the delivery",
    "code":               "for i in range(10): print(i * 2)",
    "a long number":      "invoice 4429183 for 128456 rupees",
    "Hindi (Devanagari)": "ग्राहक ने वापसी के बारे में पूछा",
    "punctuation soup":   "!!! ??? ... ---   <<<>>>",
}

print(f"{'sample':<20} {'chars':>6} {'tokens':>7} {'chars/token':>12}")
for name, s in SAMPLES.items():
    n = len(t(s)["input_ids"])
    print(f"{name:<20} {len(s):>6} {n:>7} {len(s) / n:>12.2f}")

print("\nthe 'four characters per token' rule of thumb, checked:")
for name, s in SAMPLES.items():
    n = len(t(s)["input_ids"])
    print(f"  {name:<20} guess {len(s) // 4:>3}, actual {n:>3}, "
          f"off by {abs(len(s) // 4 - n) / n:>6.1%}")

print("\ncounting a whole batch in one call:")
texts = list(SAMPLES.values())
enc = t(texts, return_length=True)
print("  lengths:", enc["length"], " total:", sum(enc["length"]))

print("\ncutting text to a budget without guessing:")
long_text = " ".join(CORPUS[:3])
cut = t(long_text, truncation=True, max_length=20, return_offsets_mapping=True)
print("  tokens kept:", len(cut["input_ids"]))
end = cut["offset_mapping"][-1][1]
print(f"  that is the first {end} characters:")
print("   ", repr(long_text[:end]))

print("\nthe overhead people forget: chat wrappers cost tokens too")
t.chat_template = ("{% for m in messages %}{{ '<|' + m['role'] + '|>' + m['content'] "
                   "+ '<|end|>' }}{% endfor %}"
                   "{% if add_generation_prompt %}{{ '<|assistant|>' }}{% endif %}")
msg = [{"role": "user", "content": SAMPLES["plain English"]}]
bare = len(t(SAMPLES["plain English"])["input_ids"])
wrapped = len(t.apply_chat_template(msg, tokenize=True,
                                    add_generation_prompt=True)["input_ids"])
print(f"  the message alone : {bare} tokens")
print(f"  inside a template : {wrapped} tokens  (+{wrapped - bare})")

sample                chars  tokens  chars/token
plain English            52      15         3.47
code                     32      25         1.28
a long number            33      24         1.38
Hindi (Devanagari)       32      84         0.38
punctuation soup         24      24         1.00

the 'four characters per token' rule of thumb, checked:
  plain English        guess  13, actual  15, off by  13.3%
  code                 guess   8, actual  25, off by  68.0%
  a long number        guess   8, actual  24, off by  66.7%
  Hindi (Devanagari)   guess   8, actual  84, off by  90.5%
  punctuation soup     guess   6, actual  24, off by  75.0%

counting a whole batch in one call:
  lengths: [15, 25, 24, 84, 24]  total: 172

cutting text to a budget without guessing:
  tokens kept: 20
  that is the first 118 characters:
    'customer shipping and it statement order delivery transfer invoice from for but for is it with delivery is to warranty'

the overhead people forget: chat wrappers cost tokens too
  the message alone : 15 tokens
  inside a template : 18 tokens  (+3)

Reading that output#

English is the only row the rule of thumb nearly fits, at 3.47 characters per token against a guess of 4, so 13 percent low. That is the best case.

Hindi costs 84 tokens for 32 characters, 0.38 characters per token. This tokeniser was trained on English only, so Devanagari falls to raw UTF-8 bytes at three bytes per character. A properly multilingual tokeniser does much better than this, and never reaches English parity.

Code and numbers are both around 1.3 characters per token. Punctuation and digits do not merge into long pieces, so both cost roughly one token per character.

The 90.5 percent error is the headline. Budget a Hindi-language product using the four-character rule and you are out by a factor of ten.

return_length=True counts a batch in one pass. Do not loop and call the tokeniser per item; it is slower and the batched form gives you the masks you need anyway.

Truncation with return_offsets_mapping tells you exactly where the cut fell. 20 tokens turned out to be 118 characters of that particular text. That mapping is how you truncate a document and still show the user precisely what was kept.

The chat wrapper added 3 tokens here because the role markers are registered as single ids. Leave them unregistered and the same wrapper costs 23 tokens, since <|user|> spells out character by character. Whether your markers are real vocabulary entries changes your overhead by an order of magnitude.

Counting for a specific model#

The script above trains its own tokeniser so it runs offline. In production you count with the tokeniser belonging to the model you are calling:

# Hugging Face models. Downloads the tokeniser files on first use.
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("<model-id>")
n = len(tok(text)["input_ids"])

# OpenAI models. pip install tiktoken; downloads the encoding file on first use.
import tiktoken
enc = tiktoken.encoding_for_model("<model-name>")
n = len(enc.encode(text))

No output block for these, deliberately. Both fetch files on first use, counts differ per model and per version, and printing invented numbers here would teach you to expect totals that will not match what you see.

For providers exposing a token-counting endpoint, that endpoint is authoritative and accounts for their exact prompt formatting. Prefer it over any local reimplementation when billing accuracy matters.

A budgeting checklist#

Count all of these, not only the user's message:

• system prompt, sent on every request
• tool and function schemas rendered into the prompt
• retrieved context, which is usually the largest item in a RAG system
• the full conversation history for multi-turn chats
• chat template markers and the generation prompt
• expected output length, usually priced higher than input

The item that surprises people is history. Re-sending an $n$-turn conversation on turn $n$ makes total tokens across a session grow with the square of turn count.

Common mistakes#

Counting with a different model's tokeniser. Vocabularies differ, so counts differ. A number from one model family is not a valid estimate for another.

Counting the message and not the wrapper. Shown above. For short messages the wrapper can be a significant fraction of the total.

Truncating by characters. Slicing a string mid-token gives the model a partial byte sequence. Truncate with the tokeniser and max_length.

Ignoring output tokens in the budget. Generation is usually the more expensive half, and it is the half you control least.

Assuming a token count is stable across versions. Tokenisers do change between model releases. Re-measure when you upgrade.

Try it yourself#

Take a real request from your own application: system prompt, tools, retrieved context, history and the user message. Count each part separately and print the breakdown. Almost everyone is surprised by which part dominates, and that breakdown is the map of where cost reduction is actually available.

What to learn next#

• Context window — the limit these counts are measured against.
• Choosing a vocabulary size — the upstream cause of your fertility.
• Adding tokens and resizing embeddings — one way to cut token counts on your own domain.