Streaming tokens correctly

Setup#

# no libraries needed for the first block
python3 --version

The whole problem is byte boundaries. Python's standard library already has the correct tool, and most streaming bugs come from not knowing it exists.

Incremental decoding, and what happens without it#

import codecs, json, sys, io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8")

TEXT = "आपका चाय ☕ तैयार है"
raw = TEXT.encode("utf-8")
# Byte-level tokenizers cut on byte boundaries, not on character boundaries.
CHUNKS = [raw[i:i + 5] for i in range(0, len(raw), 5)]

print(f"text            : {TEXT}")
print(f"characters      : {len(TEXT)}")
print(f"utf-8 bytes     : {len(raw)}")
print(f"pretend tokens  : {len(CHUNKS)} pieces of at most 5 bytes\n")

print("decoding each piece on its own:")
for i, c in enumerate(CHUNKS):
    try:
        print(f"   piece {i}: {c.decode('utf-8')!r}")
    except UnicodeDecodeError as e:
        print(f"   piece {i}: UnicodeDecodeError -> {e.reason}")

print("\nsame pieces through an incremental decoder:")
dec = codecs.getincrementaldecoder("utf-8")()
out = ""
for i, c in enumerate(CHUNKS):
    piece = dec.decode(c)                 # holds back an unfinished character
    out += piece
    print(f"   piece {i}: emitted {piece!r}")
print(f"   final: {out!r}")
print(f"   matches the original: {out == TEXT}")

print("\nthe same bytes as server-sent events, which is what the wire looks like:")
dec = codecs.getincrementaldecoder("utf-8")()
for c in CHUNKS[:4]:
    piece = dec.decode(c)
    if not piece:
        continue                          # nothing complete yet: send NOTHING
    frame = {"choices": [{"delta": {"content": piece}, "finish_reason": None}]}
    print("data: " + json.dumps(frame, ensure_ascii=False))
print('data: {"choices": [{"delta": {}, "finish_reason": "stop"}]}')
print("data: [DONE]")

text            : आपका चाय ☕ तैयार है
characters      : 19
utf-8 bytes     : 49
pretend tokens  : 10 pieces of at most 5 bytes

decoding each piece on its own:
   piece 0: UnicodeDecodeError -> unexpected end of data
   piece 1: UnicodeDecodeError -> invalid start byte
   piece 2: UnicodeDecodeError -> invalid start byte
   piece 3: UnicodeDecodeError -> invalid start byte
   piece 4: UnicodeDecodeError -> invalid start byte
   piece 5: UnicodeDecodeError -> invalid start byte
   piece 6: UnicodeDecodeError -> unexpected end of data
   piece 7: UnicodeDecodeError -> invalid start byte
   piece 8: UnicodeDecodeError -> invalid start byte
   piece 9: UnicodeDecodeError -> invalid start byte

same pieces through an incremental decoder:
   piece 0: emitted 'आ'
   piece 1: emitted 'पक'
   piece 2: emitted 'ा '
   piece 3: emitted 'चा'
   piece 4: emitted 'य '
   piece 5: emitted '☕ त'
   piece 6: emitted 'ै'
   piece 7: emitted 'या'
   piece 8: emitted 'र '
   piece 9: emitted 'है'
   final: 'आपका चाय ☕ तैयार है'
   matches the original: True

the same bytes as server-sent events, which is what the wire looks like:
data: {"choices": [{"delta": {"content": "आ"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": "पक"}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": "ा "}, "finish_reason": null}]}
data: {"choices": [{"delta": {"content": "चा"}, "finish_reason": null}]}
data: {"choices": [{"delta": {}, "finish_reason": "stop"}]}
data: [DONE]

Reading the output#

Nineteen characters take forty-nine bytes. Each Devanagari character costs three. The same sentence in English would be close to one byte per character. Every byte-level budget written by an English-speaking developer is wrong for Indian languages by roughly a factor of three, and that includes token budgets.

All ten pieces failed to decode on their own. Not one, not a couple — every single one. That is what a naive chunk.decode("utf-8") in a streaming loop does to Hindi.

The incremental decoder emitted nothing extra and everything correct. piece 0 gave one character out of five bytes and silently kept the remaining two. piece 5 gave three characters, including one that began in an earlier chunk. The reconstruction matches the original exactly.

Piece 6 emitted a combining mark on its own. The ै is a vowel sign that attaches to the previous consonant. It is a valid complete character, so the decoder correctly emits it, and a renderer will attach it to what came before. Byte-level correctness is not the same as visual stability, and a client that animates each delta separately can make text appear to shuffle. Byte correctness is the part you must get right; visual smoothness is a client-side choice.

The if not piece: continue line is the streaming rule. When the decoder holds bytes back it returns an empty string. Sending an empty delta frame is not harmful, but sending a broken one is, and the guard makes the intent explicit.

The second bug: building text from tokens#

import sys, io
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8")
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("TinyLlama/TinyLlama-1.1B-Chat-v1.0")
ids = tok("The chai is ready", add_special_tokens=False).input_ids

print("token ids:", ids)
print("per-token decode :", [tok.decode([i]) for i in ids])
print("joined           :", repr("".join(tok.decode([i]) for i in ids)))
print("whole-sequence   :", repr(tok.decode(ids)))

out, deltas = "", []
for n in range(1, len(ids) + 1):
    full = tok.decode(ids[:n])          # decode the whole prefix
    deltas.append(full[len(out):])      # send only what is new
    out = full
print("prefix-diff deltas:", deltas)
print("joined deltas    :", repr("".join(deltas)))

token ids: [450, 521, 1794, 338, 7960]
per-token decode : ['The', 'ch', 'ai', 'is', 'ready']
joined           : 'Thechaiisready'
whole-sequence   : 'The chai is ready'
prefix-diff deltas: ['The', ' ch', 'ai', ' is', ' ready']
joined deltas    : 'The chai is ready'

This downloads the tokenizer files only, around a megabyte, not the model weights. Run against transformers 5.6.2.

'Thechaiisready'. Every space is gone. This tokenizer marks a leading space with a special symbol inside the token, and decoding one token at a time strips it as leading whitespace. Decoding tokens individually and concatenating is wrong for every SentencePiece-derived tokenizer, which is most of the Llama, Mistral and Gemma families.

The prefix-diff deltas are correct. 'The', ' ch', 'ai', ' is', ' ready' — spaces attached to the right tokens, and joining them reproduces the sentence. Decoding the full prefix each step is $O(n^2)$ over a long generation, so production detokenizers keep a small window of recent tokens and diff within it. The principle is the same: decode a span, not a single token.

What a real streaming server holds back#

Three separate buffers, and each exists for its own reason:

1. Incomplete UTF-8 bytes — until the character is whole.
2. Possible stop-string prefixes — up to len(stop) - 1 characters, covered in stopping criteria.
3. Tokenizer context — a few recent tokens so the decoder knows how to join.

All three are hold-backs. All three are missing from most hand-rolled streaming loops.

The wire format#

Server-sent events, which is HTTP with a specific body shape:

Content-Type: text/event-stream
Cache-Control: no-cache
X-Accel-Buffering: no

data: {"choices":[{"delta":{"content":"The"}}]}

data: {"choices":[{"delta":{"content":" chai"}}]}

data: [DONE]

Each frame is data: plus one JSON object, terminated by a blank line. [DONE] is a convention rather than part of the SSE specification.

X-Accel-Buffering: no matters more than it looks. Nginx buffers proxied responses by default. A correct streaming server behind a default reverse proxy then delivers the whole answer in one lump. This is the most common "streaming does not work in production but works locally" cause.

Common mistakes#

Decoding each token alone. Shown above. Spaces vanish, and non-English text breaks worse.

No client-side cancellation. If the user closes the tab, the server should stop generating. Without this you pay for tokens nobody will read and you hold a KV cache slot.

Buffering in the framework. Flask, Django and Express all need explicit streaming responses. A generator that is silently collected into a string is easy to write by accident.

Streaming a response that must be parsed as a whole. For JSON output, streaming gives the user a growing blob of syntax. Stream it only if you have a partial-JSON parser on the client, and check finish_reason before parsing.

Measuring only total time. Time to first token and inter-token latency are what streaming changes. Total time is barely affected. Log all three separately.

Try it yourself#

Change CHUNKS to slice at 1 byte instead of 5. Every piece now fails the naive decode and the incremental decoder still reconstructs the sentence exactly. Then try a string of pure ASCII and watch the naive version work perfectly — which is precisely how this bug survives testing and ships.

What to learn next#

• Stop tokens and stopping criteria — the other reason a streaming server holds bytes back.
• Tokenization — why token boundaries and character boundaries disagree.
• Model deployment — putting the whole pipeline behind a real endpoint.