Contents
Data
| Where | Says 30 | Says 45 |
|---|---|---|
| Paper (arXiv v2) | 1 | 0 |
| benchmarklist page | 1 | 0 |
| Vendor page body | 1 | 0 |
| Transfer page 1 | 0 | 1 |
| Transfer page 2 | 0 | 1 |
| Transfer page 3 | 0 | 1 |
| Vendor page header | 0 | 1 |
Two audits of my own research-agent runs found citations that pointed at nothing. In a Kaggle benchmark run, a brief attributed its hyperparameter tables, a submission progression and a score floor to a writeup by “a-ko”. A grep over the research tree, the notebooks and the kernels hit zero results: only the brief’s own UNVERIFIED-PENDING tags contained the string. A kernel search on the platform found 0 kernels by that author, and the official winners list on the competition’s own page contained no such name. Every claim standing on that writeup was phantom, so I re-derived the configs from the kernel sources on disk and replaced the floor with a submission the ledger actually shows: 17.08.
The first date I can pin on the pattern in the run records is 2026-09-09. A month later, on 2026-10-03, a market-research scan flagged itself as a data-integrity incident: DOIs and numeric claims circulating inside malformed bracketed tokens, earlier subagent dumps fed back into later rounds as primary sources, and one number reading “verified” in a round, then “refuted” in the next. The agent was citing its own output and calling the echo a source.
This piece is about both incidents: what the corrupted copies looked like, which check caught them, and what the records leave unprotected.
A number that lived only in copies
The scan compared three public healthcare agent benchmarks, and somewhere in the hand-offs one benchmark acquired “45 harness configurations”. By the time I audited it, four third-party pages carried the number, and a derived sentence claimed “3 harness × 10 model = 45 config”.
The originals say something else. One benchmark names 3 harnesses, another 30 configuration (3 harness × 10 model), a third “seven agent configurations”. The arithmetic in the derived sentence is wrong on its face: 3 × 10 is 30.
The copies are worth reading side by side, because that is where the mechanism shows. One transfer page reads: “Released May 20, 2026; updated August 12, 2026 across 45 harness configurations”. The original sentence, verbatim from the paper: “Across 30 agent harness/model configurations, the best agent resolves only 28.0% of tasks”. The same shape, the same verbs, a different number. A second page carried “45 harness configurations” inside its own header while its body said “Agent configs 30 harness × model”: the page disagreed with itself. The audit’s verdict on the four pages: all four were corrupted copies of the same sentence, so the chain was one source. Four printings of one corrupted sentence is one source wearing four hats.
The check that closed it was cheap. A re-fetch of the page that had been quoted, counting the exact strings: “45 harness” and “harness config” came back count 0 from the page text. The earlier round had claimed the strings passed verbatim without re-fetching. Count the string in the page you are citing. That is the whole check.
Data
| Phrase (page) | Hits |
|---|---|
| 45 harness (paper) | 0 |
| harness config (paper) | 0 |
| 45 harness (vendor) | 0 |
| 200+ tools (vendor) | 0 |
| CHI-Bench (vendor) | 2 |
The audit’s own scoreboard
The same re-fetch method ran over the scan’s do-not-cite list. The benchmark block quoted six claims against primary pages. Three strings from the re-fetched paper returned hits, including the 42% sentence and the 54-task phrase; “45 harness” and “harness config” came back count 0. A vendor page returned 0 hits for “45 harness” and 0 for “200+ tools”, and 2 for “CHI-Bench”. A forum search over four exact phrases: 200 tools returned 6 hits, none of them in the claim’s field; “5 in 4 fail”, “500,000 primary care” and “Commure suite physicians” returned 0 hits each. Counted string by string: 7 counts came back 0, the 2 non-zero hits and the returned sentences supported the one claim the audit let stand, and the 6 hits that did come back belonged to none of the claims.
Data
| Exact phrase | Hits |
|---|---|
| 200 tools | 6 |
| 5 in 4 fail | 0 |
| 500,000 primary care | 0 |
| Commure suite physicians | 0 |
A real paper wearing a stranger’s claim
The Kaggle run had a cleaner failure mode, and a nastier one. A lever in the strategy brief was justified by an arXiv ID whose paper supposedly reported 50% on ARC-AGI-1 with 35 times fewer GPU hours. The re-fetch found the paper. It exists. Its title is “ISOTTA, FinePO, and DIFU: A Review and Quantitative Benchmark of Reproduction Infrastructure for Open Science Metadata Dissemination”, a data-services infrastructure review, with nothing in it about ARC-AGI, cross-model re-scoring or GPU hours. The ID resolves; the claim attached to it belongs to no one.
The phantom author worked the same way in reverse: a real technique, real winning kernels on disk, and the citation pointed at an author who never wrote anything. Both failures are a match that looked like verification. A resolver returning 200 is not a check that the sentence is on the page, and a name appearing in a brief is not a check that the person exists.
The fake DOI
The incident record logs a DOI quoted as evidence: 10.1056/2025NEJM2519535. The official resolver returned “DOI Not Found”; the metadata registry returned 404. The nearest real neighbours are 10.1056/EH0700104 and 10.1056/EH0700105. The later audit note observed the shape collision with a Taylor & Francis article ID, which hints at how a display or generation layer might have smeared two identifiers into one, but the record says “likely”, and the cause was not isolated. Three more tokens from the same layer, a PMCID, a PMID and a journal DOI, turned out to belong to unrelated fields.
What the rules became
The incident file installs five rules on the spot. A primary source is a full https:// URL plus a sentence that appears verbatim in the page text; a bracketed or malformed token is deleted, not cited. A number is written with the protocol that measured it: n, unit, ceiling or mean. A figure that reads “found” in one round and “absent” in the next is banned, not averaged. Rounds of the same run cannot be each other’s sources. And a bot wall is never mimicked past; the honest label when a page refuses is not-found.
The rules read as obvious now. What makes them non-obvious is that the pipeline felt like verification while it was failing: greps, re-fetches, a citation budget, confidence labels. The audit found that five queries said to corroborate a figure had run inside one subagent round, and one record states plainly that five queries in one round are not five independent checks.
What is still unprotected
The count-0 re-fetch closed the “45” strings. It does not close the class. The 42% number is real: it is the mean task success of the best agent on one benchmark, with a peak of 55 from another pairing. It sits inside a chain the audit rotted, so the record bans it until someone re-verifies from the public page without the run’s own dumps in the loop. The record does not show that re-verification being run.
The self-feeding path is patched by policy, not by machinery. As far as the records show, nothing in the harness stamps which round a claim entered; “X appeared in an earlier report” still looks like corroboration to a model that reads its own project files. The Kaggle run’s fix, a required URL plus cache-file hash for any cited writeup, is a rule the next brief can silently violate, and the recurrence line in the audit names three runs in a row.
The one asymmetry I would bet on: what caught a phantom came from outside the run: in the scan, a re-fetch of the primary page with the string counted; in the Kaggle run, the platform’s own search and winners list. Rereading the run’s own reports, however many rounds, never did.
I run both projects; the agent runs are autonomous research scans I set up and supervise, and an AI assistant drafted this from the incident records, the audit brief and the rule file cited throughout. The private project names stay out; the public papers, benchmarks and resolvers are named as the records name them.