Securing the ML supply chain
Almost nobody trains a model completely from scratch anymore — which means almost every model quietly inherits risk from every pretrained weight, dataset and package it was built on.
- 11 min read
- 3 reading levels
- Published
Read these first
On this page 5
One lesson, three depths. Pick the one that fits you today — you can switch any time.
Beginner — No maths. Plain English.
The ML supply chain is every pretrained model, dataset and package a project depends on but did not build. A problem in any one link can quietly reach the final product.
Think about a restaurant's supply chain for a single dish. The restaurant buys sauce from a supplier. That supplier bought tomatoes from a farm. That farm bought pesticide from another company. Suppose any single link in that chain is contaminated. The final plate served to a customer can be affected. This happens even though the restaurant itself did nothing wrong, and never touched the contaminated ingredient.
Modern machine learning works exactly the same way. Almost nobody notices the chain, until something in it breaks.
Why it exists
Training a large model completely from scratch is expensive and slow. Most real projects start from something built by someone else. A pretrained model downloaded from a public hub. A public dataset scraped from the internet. A code package installed with one command. This is efficient. It is also how the previous two lessons' risks — poisoned data, stolen model behaviour — extend into everything you depend on.
A pretrained model can already contain a backdoor, planted before you ever downloaded it. A popular code package can be swapped for a malicious lookalike with a nearly identical name. This trick is called typosquatting — hoping someone installs numpi instead of numpy by mistake. And, less visibly, the very file format many models ship in has a real, well-documented security problem of its own.
How it works
Your model
|
+-- built on: a pretrained model you downloaded, not trained
| |
| +-- built on: a dataset you didn't collect or fully audit
|
+-- built with: a dozen software packages you installed with one command
|
+-- built on: dozens more packages THOSE packages depend on
A problem introduced ANYWHERE in this chain can reach your final,
deployed model -- without you ever writing a single unsafe line
of code yourself.Securing your own code is necessary. It is no longer sufficient. So much of a modern ML project was never written by you at all.
A real example you have seen
Malicious packages turn up regularly on public repositories (PyPI for Python, npm for JavaScript). They are designed to look like a popular, trusted package. This is a recurring, real security story. Python's ML ecosystem gets targeted this way specifically, because so many projects install a long dependency chain without close review.
Remember this
- Almost every real ML project depends on pretrained models, datasets, and packages it did not build itself.
- A vulnerability introduced anywhere upstream in that chain can reach your final model, even if your own code is clean.
- This risk compounds with every dependency added. That is exactly why supply chain security has become its own discipline.
- Trusting a pretrained model or package in production needs a real security review, not only a popular name and a high download count.
What to learn next
- Data poisoning and backdoors — how a pretrained model can already be compromised before you download it.
- Model cards — a practical step toward documenting exactly what a model was built from.
- Responsible deployment — the broader practice this section's nine lessons all feed into.
Developer — Code and libraries.
Setup
python3 --versionStandard library only — this lesson demonstrates a real, well-documented risk in Python's own pickle module, which many older ML model file formats are built directly on top of.
Minimal runnable code
Many .pkl, and older .pt (PyTorch) files are, under the hood, Python pickle files. Pickle was never designed to be safe against untrusted input — loading one can run arbitrary code, not only read data. This is demonstrated here with a completely harmless payload, purely to show the mechanism.
import pickle
class LooksLikeModelData:
"""Nothing about this class name or the file it produces looks dangerous."""
def __reduce__(self):
# __reduce__ tells pickle: "when someone LOADS me, call this function with these args."
# A real attacker would put something destructive here.
# This one is deliberately harmless -- it only prints a message.
return (print, ("<< arbitrary code ran while 'loading a data file' >>",))
payload = pickle.dumps(LooksLikeModelData())
print("what gets downloaded is plain bytes, e.g.:", payload[:35], "...")
print()
print("loading it the way many tutorials casually load a downloaded .pkl checkpoint:")
pickle.loads(payload) # <-- code executes HERE, at load time, not at download time
print("...loading finished. Nothing about the file's name or extension warned us.")what gets downloaded is plain bytes, e.g.: b'\x80\x04\x95Q\x00\x00\x00\x00\x00\x00\x00\x8c\x08builtins\x94\x8c\x05print\x94\x93\x94\x8c4<' ... loading it the way many tutorials casually load a downloaded .pkl checkpoint: << arbitrary code ran while 'loading a data file' >> ...loading finished. Nothing about the file's name or extension warned us.
What actually happened
pickle.loads(payload) was supposed to do nothing more than "load some data." Instead, it ran print(...) — arbitrary Python code, chosen entirely by whoever created the pickle file, not by the person loading it. This happened purely because the object defined a __reduce__ method, a completely standard, documented Python feature meant for legitimate uses like copying complex objects — and it is exactly what a malicious pickle file abuses.
This one printed a harmless message. A real malicious payload could run any command the loading process has permission to run — reading files, sending data over the network, or installing further malware — the moment pickle.load() or pickle.loads() is called, disguised as an ordinary "load my downloaded model checkpoint" step.
- The danger is not in downloading the file. It is specifically in the moment code calls
pickle.load()or.loads()on it — a fact easy to miss, since most tutorials show downloading and loading as one uneventful step. payload[:35]shows the raw bytes are not human-readable — there is no way to inspect a pickle file by eye and confirm it is safe before loading it.
Common mistakes
Downloading and loading a pretrained model without checking its format. torch.load() on an untrusted .pt file historically used pickle underneath by default, inheriting exactly this risk. Current PyTorch supports torch.load(path, weights_only=True), which restricts what can be reconstructed during loading — check your framework's current documentation for the safe-loading option, since this area has been actively improving.
Treating a popular model hub as a guarantee of safety. Hosting a file is not the same as verifying its contents are safe to deserialize. Reputable hubs have added scanning (Hugging Face runs automated pickle-import scanning on uploaded files, for example), but scanning catches known bad patterns, not everything.
Copy-pasting an install command without checking the package name carefully. Typosquatted packages rely on exactly this — a name one character off from a well-known package, installed by a tired developer moving fast.
Assuming this is only a "deep learning" problem. The exact same risk applies to any pickled scikit-learn model, pandas DataFrame, or other Python object shared as a file — pickle's behaviour does not care what the object represents.
Try it yourself
Look up the current recommended safe-loading option for whichever framework you use most (weights_only=True in PyTorch, or the safetensors format, are two real, current examples). Confirm it is set correctly the next time you load a model file from an untrusted or third-party source — this is a real, five-minute habit worth building, not only a lesson exercise.
What to learn next
- Data poisoning and backdoors — a backdoor can ride along inside a pretrained model exactly like this risk can.
- Model cards — documenting a model's provenance, part of a real supply-chain defence.
- Structuring an ML repo — where dependency and artifact management decisions get made in practice.
Researcher — Mathematics and papers.
The pickle deserialization vulnerability, formally
Python's pickle protocol serializes not only data but instructions for reconstructing arbitrary objects, including a REDUCE opcode that, on unpickling, calls a specified callable with specified arguments — exactly the mechanism exploited above. Because the callable and arguments are attacker-controlled in a malicious pickle stream, pickle.load() on untrusted input is equivalent, in security terms, to executing untrusted code — this has been documented in Python's own official pickle module documentation for years, stating plainly that it is "not secure" against erroneous or maliciously constructed data.
This is not a bug to be patched; it is an inherent property of the format's design, which is why the ML ecosystem's response has been to move away from pickle-based formats for model weights rather than to attempt to "fix" pickle itself.
The safetensors response
safetensors (Hugging Face, 2022) is a file format designed specifically as a safe replacement for pickle-based weight files: it stores only tensor data and shape/dtype metadata, with a format that cannot, by construction, contain executable instructions, and is verifiably fast to load via memory-mapping. Its adoption across the Hugging Face Hub, and increasingly as a default output format for major training frameworks, is a direct, ecosystem-level mitigation for the exact risk demonstrated above.
Model and dataset provenance
Beyond the file-format risk, supply-chain integrity for ML increasingly borrows frameworks from traditional software supply-chain security:
- SBOM-equivalent documentation — model cards (Mitchell et al., 2019) and dataset datasheets (Gebru et al., 2018) formalise what a model or dataset was built from, analogous to a Software Bill of Materials in traditional software supply-chain practice.
- Cryptographic signing — signing model artifacts (e.g. via Sigstore-style transparency logs, increasingly discussed for ML artifacts specifically) lets a consumer verify a downloaded model matches exactly what its publisher released, unmodified in transit or at rest.
- In-toto and SLSA frameworks — supply-chain integrity frameworks originally built for traditional software builds, being actively adapted (as of recent years) to cover ML training pipelines, where the "build" includes data collection and training runs, not only code compilation.
Dependency confusion and typosquatting, formally
Typosquatting attacks register a package name similar to a popular one (torhc for torch), relying on installer typos. Dependency confusion (Birsan, 2021) is a related but distinct attack: if an organisation uses an internal package name on a private index, and an attacker publishes a same-named package with a higher version number on the public index, misconfigured tooling can silently prefer and install the attacker's public package instead of the intended private one — a vulnerability class discovered to affect numerous major technology companies' internal build pipelines when first publicly disclosed.
Cost
Auditing every transitive dependency of a modern ML stack (a single pip install torch pulls in dozens of further packages, each with their own dependencies) is not fully tractable by hand; practical mitigation relies on automated tooling — dependency vulnerability scanners (e.g. pip-audit, GitHub's Dependabot), pickle-content scanners (e.g. Hugging Face's picklescan), and pinned, hash-verified dependency files — rather than manual review, mirroring how traditional software supply-chain security operates at scale.
Key references
- Python Software Foundation.
pickle— Python object serialization (official documentation) — states the security limitation directly. - Mitchell, M., et al. (2019). Model Cards for Model Reporting. ACM FAT*.
- Gebru, T., et al. (2018, revised 2021). Datasheets for Datasets. Communications of the ACM.
- Birsan, A. (2021). Dependency Confusion: How I Hacked Into Apple, Microsoft and Dozens of Other Companies. (independent security research, widely cited as the origin of the dependency-confusion attack class).
- OWASP. Machine Learning Security Top 10 — a current, actively maintained community reference covering supply-chain risk alongside the other attack classes in this section.
Current state
Supply-chain security for ML is younger and less standardised than traditional software supply-chain security, but converging quickly toward it: safetensors adoption, model-hub automated scanning, and SBOM-equivalent model documentation have all seen substantial growth in recent years, driven directly by real incidents involving malicious models uploaded to public hubs. Treat any model or dataset download from an unfamiliar or unverified source with the same caution recommended for installing an unfamiliar software package, because the underlying risk is the same.
What to learn next
- Model cards — the documentation practice this lesson's provenance discussion builds on.
- Structuring an ML repo — practical dependency and artifact management.
- Data poisoning and backdoors — revisit this lesson, now with the full picture of how upstream risk reaches a downloaded model.