---
title: "When research agents cite themselves: phantom sources, fake DOIs and a 45 that never existed"
kind: essay
date: 2026-11-14
author: Gürkan Özkan
url: https://gurkanozkan.ai/writing/when-research-agents-cite-themselves/
markdown: https://gurkanozkan.ai/writing/when-research-agents-cite-themselves.md
description: Two audits of my own agent runs found a writeup author who never existed, a DOI that 404s at both resolvers and a 45 that lived only in corrupted copies of one sentence. The check that caught them counted strings in the page I was citing.
topics:
  - ai
tags:
  - agents
  - research
  - citations
  - evaluation
  - data-integrity
concepts:
  - data-leakage
  - llm-agents
  - model-evaluation
---

# When research agents cite themselves: phantom sources, fake DOIs and a 45 that never existed

**Chart:** Cover: Seven pages, one sentence: three printed the original number, four printed a corrupted copy of it, and counting the string in the cited page showed which. Source: my audit notes on the market-research scan (R6b, R7, V5) and the incident record of 3 October 2026.

*Printings of the sentence: which pages say 30 and which say 45*

| Where | Says 30 | Says 45 |
| --- | ---: | ---: |
| Paper (arXiv v2) | 1 | 0 |
| benchmarklist page | 1 | 0 |
| Vendor page body | 1 | 0 |
| Transfer page 1 | 0 | 1 |
| Transfer page 2 | 0 | 1 |
| Transfer page 3 | 0 | 1 |
| Vendor page header | 0 | 1 |

Two audits of my own research-agent runs found citations that pointed at nothing. In a Kaggle benchmark run, a brief attributed its hyperparameter tables, a submission progression and a score floor to a writeup by “a-ko”. A grep over the research tree, the notebooks and the kernels hit zero results: only the brief’s own UNVERIFIED-PENDING tags contained the string. A kernel search on the platform found 0 kernels by that author, and the official winners list on the competition’s own page contained no such name. Every claim standing on that writeup was phantom, so I re-derived the configs from the kernel sources on disk and replaced the floor with a submission the ledger actually shows: 17.08. {#p-0n8y2y}

The first date I can pin on the pattern in the run records is 2026-09-09. A month later, on 2026-10-03, a market-research scan flagged itself as a data-integrity incident: DOIs and numeric claims circulating inside malformed bracketed tokens, earlier subagent dumps fed back into later rounds as primary sources, and one number reading “verified” in a round, then “refuted” in the next. The [agent](https://gurkanozkan.ai/concepts/llm-agents/) was citing its own output and calling the echo a source. {#p-52tyc0}

This piece is about both incidents: what the corrupted copies looked like, which check caught them, and what the records leave unprotected. {#p-gjn7pl}

## A number that lived only in copies {#a-number-that-lived-only-in-copies}

The scan compared three public healthcare agent benchmarks, and somewhere in the hand-offs one benchmark acquired “45 harness configurations”. By the time I audited it, four third-party pages carried the number, and a derived sentence claimed “3 harness × 10 model = 45 config”. {#p-3m1nl4}

The originals say something else. One benchmark names 3 harnesses, another 30 configuration (3 harness × 10 model), a third “seven agent configurations”. The arithmetic in the derived sentence is wrong on its face: 3 × 10 is 30. {#p-ordvgd}

The copies are worth reading side by side, because that is where the mechanism shows. One transfer page reads: “Released May 20, 2026; updated August 12, 2026 across 45 harness configurations”. The original sentence, verbatim from the paper: “Across 30 agent harness/model configurations, the best agent resolves only 28.0% of tasks”. The same shape, the same verbs, a different number. A second page carried “45 harness configurations” inside its own header while its body said “Agent configs 30 harness × model”: the page disagreed with itself. The audit’s verdict on the four pages: all four were corrupted copies of the same sentence, so the chain was one source. Four printings of one corrupted sentence is one source wearing four hats. {#p-8hjqg3}

The check that closed it was cheap. A re-fetch of the page that had been quoted, counting the exact strings: “45 harness” and “harness config” came back count 0 from the page text. The earlier round had claimed the strings passed verbatim without re-fetching. Count the string in the page you are citing. That is the whole check. {#p-4dooq2}

**Chart:** The re-fetch counted 0 for each corrupted string it looked for; the one phrase that came back was a name, not a claim. Source: my audit notes on the market-research scan (R6b, R7, V5) and the incident record of 3 October 2026.

*String counts in the re-fetched paper and vendor page*

| Phrase (page) | Hits |
| --- | ---: |
| 45 harness (paper) | 0 |
| harness config (paper) | 0 |
| 45 harness (vendor) | 0 |
| 200+ tools (vendor) | 0 |
| CHI-Bench (vendor) | 2 |

## The audit’s own scoreboard {#the-audits-own-scoreboard}

The same re-fetch method ran over the scan’s do-not-cite list. The benchmark block quoted six claims against primary pages. Three strings from the re-fetched paper returned hits, including the 42% sentence and the 54-task phrase; “45 harness” and “harness config” came back count 0. A vendor page returned 0 hits for “45 harness” and 0 for “200+ tools”, and 2 for “CHI-Bench”. A forum search over four exact phrases: 200 tools returned 6 hits, none of them in the claim’s field; “5 in 4 fail”, “500,000 primary care” and “Commure suite physicians” returned 0 hits each. Counted string by string: 7 counts came back 0, the 2 non-zero hits and the returned sentences supported the one claim the audit let stand, and the 6 hits that did come back belonged to none of the claims. {#p-j2hqhh}

**Chart:** Four exact phrases searched in the forum: three came back empty, and the six hits for the fourth were about something else. Source: my audit notes on the market-research scan (R6b, R7, V5) and the incident record of 3 October 2026.

*Hits of four exact phrases in the forum search*

| Exact phrase | Hits |
| --- | ---: |
| 200 tools | 6 |
| 5 in 4 fail | 0 |
| 500,000 primary care | 0 |
| Commure suite physicians | 0 |

## A real paper wearing a stranger’s claim {#a-real-paper-wearing-a-strangers-claim}

The Kaggle run had a cleaner failure mode, and a nastier one. A lever in the strategy brief was justified by an arXiv ID whose paper supposedly reported 50% on ARC-AGI-1 with 35 times fewer GPU hours. The re-fetch found the paper. It exists. Its title is “ISOTTA, FinePO, and DIFU: A Review and Quantitative Benchmark of Reproduction Infrastructure for Open Science Metadata Dissemination”, a data-services infrastructure review, with nothing in it about ARC-AGI, cross-model re-scoring or GPU hours. The ID resolves; the claim attached to it belongs to no one. {#p-3zclde}

The phantom author worked the same way in reverse: a real technique, real winning kernels on disk, and the citation pointed at an author who never wrote anything. Both failures are a match that looked like verification. A resolver returning 200 is not a check that the sentence is on the page, and a name appearing in a brief is not a check that the person exists. {#p-42b7qk}

## The fake DOI {#the-fake-doi}

The incident record logs a DOI quoted as evidence: `10.1056/2025NEJM2519535`. The official resolver returned “DOI Not Found”; the metadata registry returned 404. The nearest real neighbours are `10.1056/EH0700104` and `10.1056/EH0700105`. The later audit note observed the shape collision with a Taylor & Francis article ID, which hints at how a display or generation layer might have smeared two identifiers into one, but the record says “likely”, and the cause was not isolated. Three more tokens from the same layer, a PMCID, a PMID and a journal DOI, turned out to belong to unrelated fields. {#p-ckmdgk}

## What the rules became {#what-the-rules-became}

**Figure:** The loop that made echoes look like sources, and the one place where a count could break it: the primary page. Source: my audit notes on the market-research scan (R6b, R7, V5) and the incident record of 3 October 2026.

A subagent dump in round 1 holds a number; round 2 cites that dump as a primary source; the brief then reads the number as verified; the earlier report is read back in as corroboration. The only check that broke the loop counted the string in the primary page itself.

The incident file installs five rules on the spot. A primary source is a full `https://` URL plus a sentence that appears verbatim in the page text; a bracketed or malformed token is deleted, not cited. A number is written with the protocol that measured it: n, unit, ceiling or mean. A figure that reads “found” in one round and “absent” in the next is banned, not averaged. Rounds of the same run cannot be each other’s sources. And a bot wall is never mimicked past; the honest label when a page refuses is not-found. {#p-b1oocb}

The rules read as obvious now. What makes them non-obvious is that the pipeline felt like verification while it was failing: greps, re-fetches, a citation budget, confidence labels. The audit found that five queries said to corroborate a figure had run inside one subagent round, and one record states plainly that five queries in one round are not five independent checks. {#p-o4cwlo}

## What is still unprotected {#what-is-still-unprotected}

The count-0 re-fetch closed the “45” strings. It does not close the class. The 42% number is real: it is the mean task success of the best agent on one benchmark, with a peak of 55 from another pairing. It sits inside a chain the audit rotted, so the record bans it until someone re-verifies from the public page without the run’s own dumps in the loop. The record does not show that re-verification being run. {#p-sgguhw}

The self-feeding path is patched by policy, not by machinery. As far as the records show, nothing in the harness stamps which round a claim entered; “X appeared in an earlier report” still looks like corroboration to a model that reads its own project files. The Kaggle run’s fix, a required URL plus cache-file hash for any cited writeup, is a rule the next brief can silently violate, and the recurrence line in the audit names three runs in a row. {#p-02zrji}

The one asymmetry I would bet on: what caught a phantom came from outside the run: in the scan, a re-fetch of the primary page with the string counted; in the Kaggle run, the platform’s own search and winners list. Rereading the run’s own reports, however many rounds, never did. {#p-hex2ff}

*I run both projects; the agent runs are autonomous research scans I set up and supervise, and an AI assistant drafted this from the incident records, the audit brief and the rule file cited throughout. The private project names stay out; the public papers, benchmarks and resolvers are named as the records name them.* {#p-ap9z4e}