Reading logprobs

Setup#

pip install torch transformers

This one uses a real model. distilgpt2 is about 340 MB and runs on a CPU in a second or two. It downloads the first time and prints a progress bar that is not shown in the output below.

Getting the numbers out#

import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2").eval()


def next_token_table(prompt, k=5):
    ids = tok(prompt, return_tensors="pt").input_ids
    with torch.no_grad():
        logits = model(ids).logits[0, -1]          # the LAST position predicts the next token
    logprobs = F.log_softmax(logits, dim=-1)       # log of the probability, in nats
    top = torch.topk(logprobs, k)
    print(f"prompt: {prompt!r}")
    for lp, i in zip(top.values, top.indices):
        print(f"   {tok.decode(i)!r:<14} logprob {lp:>7.3f}   probability {lp.exp():>6.2%}")
    print()


next_token_table("The capital of France is")
next_token_table("The capital of Burkina Faso is")


def score(text):
    """Total and per-token log-probability of a whole string."""
    ids = tok(text, return_tensors="pt").input_ids
    with torch.no_grad():
        logits = model(ids).logits
    lp = F.log_softmax(logits[0, :-1], dim=-1)     # position i predicts token i+1
    chosen = lp.gather(1, ids[0, 1:, None]).squeeze(1)
    return ids[0], chosen


for text in ("The capital of France is Paris.",
             "The capital of France is Madrid."):
    ids, chosen = score(text)
    total = chosen.sum().item()
    ppl = torch.exp(-chosen.mean()).item()
    print(f"{text!r}")
    print(f"   total logprob {total:>8.3f}   mean {chosen.mean():>7.3f}   perplexity {ppl:>8.2f}")
    for i, lp in enumerate(chosen):
        print(f"      {tok.decode(ids[i + 1])!r:<10}{lp:>8.3f}")
    print()

prompt: 'The capital of France is'
   ' the'         logprob  -2.139   probability 11.77%
   ' home'        logprob  -2.498   probability  8.23%
   ' now'         logprob  -2.998   probability  4.99%
   ' a'           logprob  -3.009   probability  4.93%
   ' being'       logprob  -3.335   probability  3.56%

prompt: 'The capital of Burkina Faso is'
   ' the'         logprob  -2.593   probability  7.48%
   ' home'        logprob  -2.617   probability  7.30%
   ' now'         logprob  -3.086   probability  4.57%
   ' a'           logprob  -3.194   probability  4.10%
   ' being'       logprob  -3.198   probability  4.08%

'The capital of France is Paris.'
   total logprob  -30.065   mean  -5.011   perplexity   150.02
      ' capital' -14.300
      ' of'       -0.951
      ' France'   -4.620
      ' is'       -3.009
      ' Paris'    -5.891
      '.'         -1.293

'The capital of France is Madrid.'
   total logprob  -31.838   mean  -5.306   perplexity   201.62
      ' capital' -14.300
      ' of'       -0.951
      ' France'   -4.620
      ' is'       -3.009
      ' Madrid'   -7.926
      '.'         -1.032

Run against transformers 5.6.2 and torch 2.5.1 on CPU. Greedy scoring of a fixed string is deterministic, so these numbers reproduce. The last decimal can differ on other hardware, for the reason covered in temperature zero is still not deterministic.

Reading the output, which is more interesting than it looks#

distilgpt2 does not know that the capital of France is Paris. Its top five continuations are the, home, now, a, being. Its favourite is the, at under twelve percent. Small models are grammar engines with patchy facts, and this is what that looks like numerically.

The Burkina Faso prompt gives almost the same list. Same five tokens, same order, slightly lower confidence. The model has no country-specific knowledge in play at all — it is completing the pattern "The capital of X is ...". A model that does not know need not look different from one that does. That is why an eyeball check of the top token is a weak test.

Paris scored -5.891 and Madrid scored -7.926. Paris is about seven and a half times more likely. Both are unlikely in absolute terms — Paris is at roughly 0.28 percent. The right answer won, and it won without the model being remotely confident. Confidence and correctness came apart in a single example.

capital scored -14.300, by far the worst token in both sentences. Nothing in "The" predicts "capital". This is the most useful habit to build: the lowest-scoring tokens are usually places where information arrived from outside the model rather than places where the model was wrong.

Perplexity is 150.02 for the Paris sentence. Perplexity is the exponential of the average negative logprob. It reads as "about as unsure as choosing uniformly among 150 options each step". Lower is less surprised. The Madrid sentence scores 201.62, correctly ranking the two.

The first token has no score. Six tokens are listed for a seven-token sentence. Nothing predicts The, so it has no probability under a causal model. Off-by-one errors here are the most common bug in home-made scoring code, and they inflate scores in a way that is hard to notice.

Getting logprobs from an API#

Most providers expose them, with a cap on how many alternatives per position:

resp = client.chat.completions.create(
    model="...", messages=[...],
    logprobs=True, top_logprobs=5,
)
for tok in resp.choices[0].logprobs.content:
    print(tok.token, tok.logprob, [(a.token, a.logprob) for a in tok.top_logprobs])

No output block: the values depend on the provider and the model, and inventing them would teach you to expect something that will not happen. The shape of the response is what matters, and it is stable across OpenAI-compatible servers including vLLM.

Two limits worth knowing. Reasoning models often do not return logprobs for hidden reasoning tokens. And top_logprobs is capped, typically at 20, so you see the head of the distribution and never the tail.

Three things logprobs are genuinely good for#

Extraction confidence. For a field the model filled in, take the minimum token logprob across that field's tokens. Route anything below a threshold to a human. This is the highest-value use of logprobs in production, and it needs no extra model call.

Choosing between fixed options. For a classification with labels yes and no, compare the logprob of each label token directly instead of generating and string-matching. Cheaper, and it gives you a graded score rather than a bare answer.

Detecting a model out of its depth. A rolling average of per-token logprob across a conversation drifting downward means the model is on unfamiliar ground. Useful as a monitoring signal, not as a hard gate.

Common mistakes#

Comparing totals across different lengths. Longer text always has a lower total. Compare mean logprob per token, or perplexity, never the sum.

Comparing perplexity across tokenizers. A model that splits text into more tokens gets a different perplexity on the same string. Cross-model perplexity comparisons are only meaningful with an identical tokenizer.

Treating high confidence as truth. Models are frequently confident and wrong, and the confident errors are the ones that reach users. Logprobs measure fluency-of-fit, not factual support. For that you want RAG and a grounding check.

Reading logprobs after truncation. With top-p or top-k in play, the reported logprobs may come from the filtered distribution rather than the raw one, depending on the server. If you are using logprobs as a confidence signal, request them at temperature zero with no truncation.

Forgetting the off-by-one. logits[i] predicts token[i+1]. Every scoring bug in this area is this line.

Try it yourself#

Score "The capital of France is Delhi." and compare against Paris and Madrid. Then score all three under a larger model such as gpt2-medium and watch the gap widen enormously. The size of that gap is a rough measure of how much the model actually knows, as opposed to how well it forms sentences.

What to learn next#

• Speculative decoding — an algorithm built entirely on comparing two models' probabilities.
• Hallucination — the failure logprobs partly, and only partly, detect.
• Model evaluation — where confidence scores belong in a measurement plan.