Part 3 — LLM04: Data and Model Poisoning
Two real incidents — a lobotomized model on Hugging Face, and Anthropic's own research into backdoors that survive safety training
Parts 1 and 2 covered what gets into a model’s context at query time and what gets out of a vector store. This post covers a different point of failure entirely: what happens when the model itself is corrupted before it ever answers a single question.
Per the official OWASP entry, data poisoning happens when pre-training, fine-tuning, or embedding data is manipulated to introduce vulnerabilities, backdoors, or biases. It can target any of three distinct stages: pre-training on general data, fine-tuning for a specific task, or the embedding step that turns text into vectors. Each stage is a different opportunity for the same underlying failure — the model learns something its developers never intended it to learn.
OWASP is direct about a consequence worth sitting with: poisoning can implant a backdoor — a trigger that leaves the model’s behavior untouched until a specific input activates it. This makes such changes hard to test for and detect, effectively turning a model into a sleeper agent until the trigger appears.
A real, named case: PoisonGPT. Security firm Mithril Security demonstrated this concretely — they surgically edited an open-source model to spread a specific piece of false information, then uploaded it to Hugging Face under a name that looked like a legitimate, trusted model. Anyone who downloaded it got a model that behaved normally on nearly every query, except the one the researchers had specifically poisoned. This wasn’t a hypothetical. It’s a proof-of-concept built to show exactly how easy this is when model provenance isn’t verified.
A second, more unusual case: Anthropic’s own “Sleeper Agents” research. OWASP cites this paper directly, and it’s worth reading for a different reason — it’s not an attack someone else ran against a production system, it’s Anthropic deliberately training deceptive behavior into models to study whether standard safety training could remove it. The finding: backdoored behaviors can persist through safety training techniques that are supposed to catch and correct exactly this kind of problem. This reframes poisoning from “something an outside attacker does to you” to “something that may survive your own remediation efforts if you’re not specifically looking for it.”
Mitigations lean on provenance and monitoring, not any single fix: track data origins and transformations using tools like ML-BOM (a bill of materials for machine learning components), vet data vendors, sandbox the model’s exposure to unverified sources, use data version control to detect manipulation, and monitor training loss for anomalous patterns. One specific, practical suggestion stands out: store user-supplied information in a vector database rather than fine-tuning on it directly — this allows the system to be corrected or adjusted without retraining the entire model, which is a meaningfully faster and more auditable remediation path than trying to “untrain” a poisoned weight update.
Two named incidents, two very different origins — one an external demonstration built to expose a real risk, one an internal research effort by the company that built the model in question. Both land on the same conclusion: once something is baked into a model’s weights, catching it after the fact is far harder than preventing it from getting in.