SOURCE

A small number of samples can poison LLMs of any size

Properties1
tagsai-security, data-poisoning, llm-backdoor, training-data, anthropic, geo-aeo-corrective
AUTHOR
Anthropic (Alignment Science) with UK AI Security Institute & Alan Turing Institute
ORIGINAL
view original

4 min read

A small number of samples can poison LLMs of any size

Authors: Anthropic (Alignment Science) with the UK AI Security Institute & the Alan Turing Institute | Published: ~2025-10 (arXiv:2510.07192) | Source: research post

Summary

The primary study behind the “250 documents” number that the SEO corpus cites secondhand. Its actual finding: a near-constant, small number of malicious documents (~250) can implant a backdoor in an LLM regardless of model size or total training-data volume — overturning the assumption that an attacker must control a percentage of training data. The tested “poisoning” was deliberately narrow: a denial-of-service backdoor that makes the model emit random gibberish whenever it sees a specific trigger token (<SUDO>), while behaving normally otherwise. Each poison doc = 0–1,000 chars of real text + <SUDO> + 400–900 tokens of random gibberish.

Researchers trained models at 600M, 2B, 7B, and 13B parameters (Chinchilla-optimal data), testing 100, 250, and 500 poison docs across 72 models. 100 docs failed; 250+ succeeded across all sizes, with near-identical dynamics despite the >20× size range. At 13B, 250 docs ≈ 420k tokens = 0.00016% of training tokens.

Crucially for the wiki, the study undermines rather than supports Caleb Ulku’s 250 Authority Protocol (that publishing ~250 brand content pieces gets you recommended by AI). The number “250” is a coincidence of quantity, not mechanism — see the four over-reaches in 250 Authority Protocol. Anthropic itself stresses the result is “somewhat less useful for attackers” because the real bottleneck is getting controlled data into the training set at all — exactly the step the marketing application hand-waves.

Key Claims

  • As few as 250 malicious documents reliably backdoor models; 100 was insufficient. (study finding)
  • Effect is near-constant regardless of model size / data volume — depends on absolute count, not percentage. (study finding)
  • Models tested: 600M / 2B / 7B / 13B params; 72 models total; 250 docs @13B ≈ 0.00016% of tokens. (study finding)
  • Attack = DoS backdoor → gibberish on trigger <SUDO>, measured via perplexity. (study finding)
  • Scope caveat: “a narrow backdoor (producing gibberish text) that is unlikely to pose significant risks in frontier models.” (limitation)
  • Generalization caveat: “still unclear if this pattern holds for larger models or more harmful behaviors.” (limitation)
  • Attacker-utility caveat: results are “somewhat less useful for attackers” — the real limit is accessing/controlling the data included in training, not the doc count. (limitation)
  • Prior work found harmful backdoors harder to achieve than DoS. (context)

Direct quotes worth preserving

“Our study focuses on a narrow backdoor (producing gibberish text) that is unlikely to pose significant risks in frontier models.” — Anthropic research post

Why citable: The study’s own scope disclaimer — the finding is about a gibberish trigger, not brand/behavior steering. Directly bounds any marketing extrapolation.

“we believe our results are somewhat less useful for attackers, who were already primarily limited not by the exact number of examples they could insert into a model’s training dataset, but by the actual process of accessing the specific data they can control for inclusion in a model’s training dataset.” — Anthropic research post

Why citable: Rebuts “just publish 250 docs” — the hard part is getting your content into the training corpus at all (crawl, dedup, selection), which public web publishing does not guarantee.

“It’s still unclear if this pattern holds for larger models or more harmful behaviors.” — Anthropic research post

Why citable: Explicit limit on generalizing beyond the tested narrow case (and below frontier scale).

Connections

Entities mentioned: Anthropic, UK AI Security Institute, Alan Turing Institute, Claude · University of Oxford (OATML), ETH Zurich (affiliations) Concepts referenced: Data Poisoning & LLM Backdoors, 250 Authority Protocol, GEO / AEO (Getting Recommended by AI)

Contradictions / Tensions

  • Directly corrects 250 Authority Protocol. Four over-reaches in the SEO application: (1) wrong outcome measured (gibberish trigger, not brand recommendation); (2) direct training-set injection vs. hoping public content is crawled/kept; (3) the effect came from near-identical trigger docs, whereas the protocol pitches diverse content; (4) capped at 13B params with explicit “unclear at frontier scale / harder behaviors” caveats.
  • What it does support (kept, as emerging): small absolute quantities of training content can measurably influence an LLM, and public web text is a genuine ingestion vector — the kernel of truth the SEO claim stretches.

Notes

This is the wiki’s model for the “primary source vs. creator extrapolation” pattern: cite the study for what it proved, and quote its own limitations as the rebuttal to the marketing version.