Learn AI Create the Future — with Pranay
Main site

Error database

SafetensorError: Error while deserializing header: HeaderTooLarge

The weights file is not a valid safetensors file — a truncated download, an LFS pointer stub, or a full disk mid-write. Check the file size against the repo, then delete and re-download.

The message you saw
SafetensorError: Error while deserializing header: HeaderTooLarge

By Updated

The error

Output
safetensors_rust.SafetensorError: Error while deserializing header: HeaderTooLarge

Siblings from the same cause family:

Output
safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer
safetensors_rust.SafetensorError: Error while deserializing header: InvalidHeaderDeserialization

What it means

A safetensors file starts with a small header describing the tensors inside. The loader read the first bytes of your file and they are not a plausible header — HeaderTooLarge means those bytes, interpreted as a header length, produced an absurd number. Translation: this file is not (or is no longer) a valid safetensors file. The format is fine, the library is fine; the bytes on disk are wrong.

Why it happens

Three ways bad bytes end up with a .safetensors name:

  • A truncated download. The connection dropped mid-file; what landed is the first N GB of a larger file. MetadataIncompleteBuffer is the typical signature.
  • A Git LFS pointer stub. A repo cloned without LFS leaves a ~130-byte text file where the weights should be — its ASCII beginning decodes into the nonsense HeaderTooLarge reports.
  • Disk full during download or save — the writer stopped early without cleaning up.

How to fix it

1. Compare the size on disk with the size on the model page.

bash
ls -lh model.safetensors

A 134-byte file is a pointer stub (see the related page). A 3.2 GB file where the repo says 4.9 GB is a truncation. Matching sizes with a bad header means genuinely corrupted content — rare, but re-downloading settles it.

2. For Hub-cached models, delete the bad copy and re-download. The reliable path:

bash
huggingface-cli delete-cache

Pick the affected model in the prompt, then re-download with resume support:

bash
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.3

Deleting the model's folder under ~/.cache/huggingface/hub/ by hand works too.

3. For git-cloned repos, pull the real LFS content.

bash
git lfs install
git lfs pull

4. Check free disk space before the retry.

bash
df -h ~

A near-full disk reproduces the truncation on every attempt — clear space first, including old cached models (huggingface-cli scan-cache shows what is eating the space).

5. For your own saved checkpoints, save to a temp name and rename on success. An interrupted save_pretrained leaves a plausible-looking partial file; writing to model.safetensors.tmp and renaming atomically afterwards means a crash leaves no impostor behind.

How to prevent it

Prefer resumable downloaders (huggingface-cli download, from_pretrained) over hand-rolled curl for multi-gigabyte files. Keep an eye on disk headroom on small VMs — weights plus cache plus datasets fill 50 GB faster than expected. And when a fresh model errors instantly at load, suspect the download before suspecting the code.