The ML Model-File Supply Chain: What Runs When You Load a Model, and How Scanners Hold Up

Reference writeup. Defensive and educational: it explains the attack surface and demonstrates detection. It contains no operational exploit payloads.

The one-sentence problem

Loading a machine-learning model can run code on your machine, because several of the file formats the ecosystem uses were designed to reconstruct arbitrary Python objects, and reconstructing an object can call arbitrary code. A model you downloaded to do inference can instead open a shell.

This is not theoretical. Model hubs host millions of files, most people load them with a single load() call, and the dangerous formats look identical to the safe ones from the outside. The purpose of this note is to make the surface legible: which formats execute code and why, what a real scanner catches, and what to actually do about it.

Part 1: why loading a model can execute code

The risk lives in deserialization. A file format is dangerous-by-design when parsing it can construct attacker-chosen objects or invoke attacker-chosen callables. It is safe-by-design when parsing it can only produce inert data (numbers, strings, tensors).

Pickle, and everything built on it. Python's pickle is the root of most model-file risk. A pickle stream is a small program for a stack machine, and one of its opcodes (REDUCE) calls a callable named earlier in the stream. That is the whole exploit primitive: a crafted file names a dangerous callable and calls it at load time, before any of your code runs. Anything pickle-backed inherits this: joblib (scikit-learn's default), PyTorch's legacy torch.load path (a zip that contains a pickle), numpy.load(..., allow_pickle=True), and pandas read_pickle. If a format is "a pickle in a wrapper," it is code-execution by design.

Keras / HDF5 Lambda layers. A Keras model can contain a Lambda layer whose body is serialized Python. Rebuilding the model to run inference reconstructs and can execute that body. The .h5 or Keras-zip container looks like a normal model; the code path is the Lambda.

GGUF chat templates (the newer one). Modern local-LLM files (GGUF, used by llama.cpp and friends) embed a Jinja2 chat template in metadata. Whatever renders that template at inference time is a template engine, and template engines are a classic server-side-template-injection surface. A hostile template is an inference-time code path, not a load-time one, which makes it easy to miss.

The safe-by-design formats. safetensors was created precisely to end this: it is a JSON header of tensor shapes plus a raw tensor buffer, with no object-reconstruction path at all. ONNX and raw protobuf graphs are data graphs (custom operators are the caveat, not the default). If a format can only yield tensors and metadata, loading it cannot run code.

Design-risk by format

format	typical use	code-exec on load?	why
pickle / `.pkl`	generic Python objects	YES, by design	`REDUCE` calls arbitrary callables
joblib	scikit-learn models	YES	pickle-backed
numpy `.npy` `allow_pickle=True`	arrays / object arrays	YES if allow_pickle	object arrays are pickled
PyTorch legacy `torch.load`	`.pt` / `.pth` / `.bin`	YES	zip containing a pickle
Keras HDF5 / Keras-zip	Keras models	YES	`Lambda` layer body
GGUF chat template	local LLMs	inference-time	embedded Jinja2 template rendering
ONNX / protobuf	portable graphs	mostly no	data graph (custom-op caveat)
safetensors	weights	NO	header + raw buffer, no object path

The uncomfortable part: from a directory listing, weights.safetensors (safe) and weights.pkl (code) are the same shape of thing, and an attacker can rename one to look like the other.

Part 2: how a real scanner holds up

The standard defensive control is a model scanner that inspects a file before you load it. ModelAudit (promptfoo) is a mature open-source example, with 45 format scanners. The obvious question a defender should ask of any such tool is: can I get a known-malicious file past it just by repackaging? Attackers do not hand you a file named evil.pkl; they rename it, wrap it, or dress it up as a safe format.

Method

Fully defensive and reproducible. I took a single, pre-existing, known-malicious pickle fixture (it invokes posix.system, which ModelAudit flags as rule S201 "dangerous call") and applied only benign packaging, rename, and nesting operations to it, then re-scanned each variant. No new exploit was authored; the same known-bad payload is simply delivered in different wrappers. Output is a detection matrix (was it scanned, top severity, which rules fired). Benign real models in each format were scanned too, to check for false positives.

Results

Evasion by repackaging did not work. Zero false negatives.

evasion vector	tested as	still detected?
rename to a model extension	`.bin .ckpt .pt .pth .model .data`	YES, S201 critical
rename to a safe-format extension	`.safetensors .gguf .onnx`	YES, S201 + S901
no extension at all	`noext`	YES, S201
unknown extension	`.xyz`	YES, S201
archive wrapper	`.zip`, `.tar.gz`	YES, recurses in
nested archive	zip-inside-zip	YES, recurses in
PyTorch-style container	zip with `archive/data.pkl`	YES, S201
directory embed	scan a folder	YES, finds the member
benign models	npy, safetensors, real `torch.save`, data-only pickle	clean, no false positives
malformed / empty	random bytes as `.safetensors`, empty `.pkl`	flagged malformed / clean, not fail-open

Two behaviors are worth calling out:

Detection is content-based, not extension-based. The scanner reads magic bytes, so stripping or faking the extension changes nothing. This is the correct design and it is worth verifying in any scanner you rely on, because an extension-based tool would be trivially defeated by mv.
Masquerade raises an extra integrity flag (S901). When a pickle wears a .safetensors or .gguf extension, ModelAudit fires S201 (the dangerous pickle) and S901 (file type does not match its claimed format). The "safe format" costume is itself treated as suspicious. Likewise, random bytes claiming to be safetensors are flagged rather than passed as clean, so the tool fails closed on content it cannot parse instead of failing open.

The honest conclusion is that a mature model scanner is robust against naive evasion. The residual risk in the ML supply chain is therefore mostly not "the scanner can be fooled by a rename." It is process and format choice, which is Part 3.

Addendum: robustness under adversarial pickle construction

The repackaging tests above defeat naive evasion (rename, wrap, nest, spoof the format). A stronger question is whether the pickle construction itself can be varied to slip a dangerous call past the opcode scanner. So we ran a broader adversarial sweep: 175 malicious-pickle fixtures, each carrying only a benign marker argument (detection keys off the dangerous global, not the argument), spanning

dangerous-global classes: os.system, posix.system, subprocess.call/Popen, builtins.exec/eval/__import__, importlib.import_module, runpy, pty.spawn, and more, and
construction styles: direct __reduce__, __reduce_ex__, GLOBAL+REDUCE, STACK_GLOBAL, pickle protocols 0 / 2 / 5, nested and tuple reduce, module-path obfuscation and aliasing, and find_class-style indirection.

Result: all 175 detected. Zero false negatives, zero errors, firing S201 / S106. The detection is opcode-level and content-based, so varying the protocol, the reduce style, or obscuring the module path did not hide the dangerous global. The sweep is reproducible from a committed generator.

Combined with the repackaging results, the honest conclusion holds on both layers: a mature model scanner is robust to delivery-layer obfuscation (rename, wrap, nest) and construction-layer obfuscation (reduce style, protocol, module-path tricks) of the classic pickle vector. The residual supply-chain risk is process and format choice, not scanner evasion.

Part 3: what to actually do

Prefer safe-by-design formats. Use safetensors for weights. If a repo only ships pickle or pickle-backed formats, treat that as a smell and, where possible, convert in a sandbox and re-publish as safetensors.
Treat every pickle-backed format as code. .pkl, .joblib, legacy .pt/.pth/.bin, and numpy object arrays are executables wearing a data costume. Set allow_pickle=False on numpy loads.
Scan before load, and make the scan content-based. Do not trust the extension. A scanner that keys off .pkl is defeated by mv model.pkl model.bin. Verify your scanner reads magic bytes and recurses into archives.
Fail closed. If a scanner cannot parse a file, that is a reason to block, not to wave it through. Confirm your tool flags "unparseable" rather than reporting "clean."
Mind the newer inference-time paths. GGUF chat templates and Keras Lambda layers execute later than load time; make sure whatever renders a chat template or rebuilds a Lambda is inside the same trust boundary as your model-loading policy.
Pin and verify provenance. Hash-pin model files, verify signatures where available, and do not auto-pull "latest" into a trusted environment.

Reproducing this

The experiment is a scanner run over a repackaged known-bad fixture plus benign controls. The detection matrix, the design-risk table, and the scanner-coverage list are all reproducible with an open-source scanner and standard library packaging calls. No offensive payload is required to reproduce the detection results; the malicious behavior itself stays as an inert, already-flagged sandbox fixture and is never reproduced or published here.

This note is published as reference material for defenders. It names attack surfaces at the conceptual level and demonstrates detection; it does not provide operational exploit code.

More security experiments and build notes live in the experiments index.