Learn AI Create the Future — with Pranay
Main site

Error database

RuntimeError: probability tensor contains either inf, nan or element < 0 (generation)

The model produced invalid logits during text generation — most often a bfloat16-trained model loaded in float16. Load in bfloat16 or float32, and check your sampling settings.

The message you saw
RuntimeError: probability tensor contains either inf, nan or element < 0 (generation)

By Updated

The error

Output
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

It fires inside model.generate(...) when sampling the next token.

What it means

At each generation step, the model's output scores are turned into a probability distribution to sample from. That conversion produced garbage — infinity, not-a-number, or negatives — so sampling is impossible. The model's forward pass is numerically broken for this input, dtype or configuration. The sampler is the messenger, not the culprit.

Why it happens

The leading cause by a wide margin: a model trained in bfloat16, loaded in float16. The two 16-bit formats trade precision against range differently. bfloat16 keeps float32's range; float16's range is small, and activation values that were fine in training overflow to infinity at inference. One infinity becomes NaN in the softmax, and generation collapses.

Lesser causes: broken sampling parameters (a temperature of 0 or negative while sampling), corrupted or incompatible weights, inputs longer than the model's context window, and occasionally genuine bugs fixed in newer library versions.

How to fix it

1. Load the model in bfloat16 instead of float16.

python
import torch
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    dtype=torch.bfloat16,
    device_map="auto",
)

bfloat16 runs natively on Ampere GPUs (RTX 30 series, A100) and newer. On older GPUs or CPU, use torch.float32 — twice the memory, but numerically safe.

2. Isolate sampling from the model with a greedy run.

python
out = model.generate(**inputs, do_sample=False, max_new_tokens=50)

Greedy decoding needs no probability sampling. If this works, revisit your sampling arguments; if this also produces garbage or errors, the problem is dtype or weights.

3. Check the sampling parameters make sense. With do_sample=True, keep temperature strictly positive (0.7 is a sane default) and top_p in (0, 1]. For deterministic output, use do_sample=False and drop temperature entirely.

4. Update transformers. Several past versions had model-specific numerical issues that produce exactly this error; upgrading is cheap and occasionally the whole fix.

bash
pip install -U transformers

5. Re-download the weights if corruption is plausible. A truncated download can pass loading and fail at run time. Delete the model from the cache and pull it again — see the safetensors page for the mechanics.

How to prevent it

Match the loading dtype to the model card — cards state the training dtype, and modern model repos default to bfloat16. When a new model misbehaves, test greedy generation first; it separates "model is broken" from "my sampling settings are broken" in one line.