How a Model on Hugging Face Became a Reverse Shell
A case study in OWASP LLM03: Supply Chain
Per the official OWASP entry, LLM supply chains are susceptible to vulnerabilities that can affect the integrity of training data, models, and deployment platforms — risks that can result in biased outputs, security breaches, or system failures. Unlike prompt injection, this isn’t about what a user or an attacker sends the model at runtime. It’s about what’s already inside the model, or its surrounding infrastructure, before it’s ever deployed.
A real, verified case: roughly 100 malicious models on Hugging Face. In February 2024, JFrog’s security research team built a scanning system to examine models hosted on Hugging Face — one of the largest public repositories for pretrained AI models — and found around 100 with genuinely malicious functionality, not false positives. One specific example: a PyTorch model uploaded by an account called “baller423” (since deleted) contained a payload that, once loaded, opened a reverse shell connection to an external IP address belonging to the Korea Research Environment Open Network. As JFrog’s senior researcher David Cohen put it, the payload “grants the attacker a shell on the compromised machine, enabling them to gain full control over victims’ machines... all while leaving victims utterly unaware of their compromised state.”
This worked because of how a common model file format behaves. Many PyTorch models are saved using Python’s pickle format, which can embed and execute arbitrary code the moment the file is loaded — not when the model generates a prediction, but simply when someone imports it into their environment. A data scientist doing exactly what the job requires — searching for a well-reviewed, reasonably-benchmarked pretrained model to save time on a deadline — can end up executing an attacker’s code before they’ve asked the model a single question. Hugging Face does scan uploads and flags suspicious ones as “unsafe,” but flagged models aren’t blocked from being downloaded, only labeled — the decision to proceed is left to the user.
A related, more subtle pattern: Model Namespace Reuse. Unit 42 (Palo Alto Networks) documented a different angle on the same underlying problem: when a model's original author deletes their Hugging Face account, the namespace — the identifier other projects reference to pull that exact model — becomes available for a new, unrelated party to register. Unit 42 didn't just describe this theoretically; they demonstrated it by re-registering abandoned namespaces and successfully achieving reverse shell injection against orphaned models found in two major cloud platforms: Google's Vertex AI Model Garden and Microsoft's Azure AI Foundry Model Catalog. Any downstream code, CI pipeline, or production system still pointing to that namespace by name now silently pulls whatever the new owner uploads instead — no code change required on the victim's end. Legit Security independently discovered the same vulnerability, naming it "AI Jacking" and estimating tens of thousands of developers potentially affected. Google's response after disclosure is worth noting on its own: they now run daily scans specifically for deleted-author namespaces to prevent orphaned models from ever reaching Vertex AI. Trust was extended to a name, not to a verified, unchanging artifact.
Mitigations focus on verification and containment, not detection of a clever attack: vet data sources and suppliers directly rather than trusting a model card’s claims at face value, maintain a software bill of materials (SBOM) — and for ML specifically, tools like ML-BOM — to track exactly which models, datasets, and adapters are in use and where they came from, apply traditional software supply chain practices like dependency scanning and patch management to the ML stack as well, use digital signatures and file hash verification rather than relying on a namespace staying stable forever, and conduct your own adversarial testing on third-party models instead of trusting published benchmarks alone. None of this catches a sophisticated attack in progress — it’s designed to prevent the untrusted artifact from ever entering your environment in a form capable of executing anything.
The Hugging Face case and the Model Namespace Reuse pattern share the same root cause, even though they look different on the surface: trust was extended based on appearance — a tidy model card, a familiar-looking name — rather than verified provenance. Nothing about either required the model to misbehave in generation. The compromise happened before the model ever produced a single output.