The corpora
Compile success and DOI resolution run on the full set of 85 completed manuscripts generated by the current pipeline — averaging roughly 3,700 words each, spanning machine learning, clinical medicine, physics, chemistry, and bioinformatics. Nothing is hand-picked: every completed paper is included, method-heavy and prose-heavy alike.
The invented-citation figure comes from a separate, purpose-built corpus: 12 fresh cases generated across all three plans, 216 sections in total. It was written deliberately outside the first corpus’s comfort zone — 11 of the cases are non-computational (wet-lab immunology, field ecology, epidemiology, cryo-EM, paleoclimate, a clinical trial), because the older corpus is almost entirely machine learning. We keep the two sets separate here rather than quoting one corpus’s citation count beside another’s compile count.
What counts as a pass
Citations resolve
Every DOI in a paper’s references is looked up live against the DOI registry (the DOI Handle API) — a DOI counts only if it actually resolves to a registered record. It measures whether a reference points at a real, registered paper; it does not measure whether the sentence citing it is supported by that paper.
Compile success
We take the exact .tex a user exports and compile it to a PDF with Tectonic, unmodified. A paper counts only if it produces a PDF with no manual fixing. A missing figure image no longer aborts the document — it degrades to a labelled placeholder — so a single failed diagram can’t cost you the whole PDF.
No invented citations
Every in-text [n] marker in 216 generated sections was resolved against the pool of papers actually retrieved for that manuscript. A citation counts as invented if it points at no retrieved paper. This is a hallucination check, not a claim-support check: it proves the reference exists and was really fetched, not that the sentence citing it is supported by it.
What the numbers don’t cover
Compile success reached 100% on this run, up from 95.3% on the previous one. The four papers that failed before did so for reasons worth naming: two were defects in our own exporter — a paper with more than one author emitted a malformed author block, and maths inside a reference title lost its delimiters — and two were drafts that put a maths symbol in ordinary prose. All four are fixed and covered by regression tests. A perfect score on 85 papers still means the true rate could be as low as 95.7%; it is not a promise that nothing will ever fail.
The citation figure has a narrower denominator than it looks. Across the corpus there are 1,509 references, of which 284 carry no DOI at all — preprints, datasets, theses and older books that were never assigned one. Those cannot be checked against the DOI registry, so they are excluded rather than counted as passes. The 98.5% is 1,207 / 1,225 DOIs resolve.
Zero invented citations is a strong result, but it is a narrow one, and it is largely structural: the model is handed a numbered list of papers already retrieved and can only cite from it, so there is no opportunity to invent one. Read it as evidence the architecture forecloses hallucinated references — not as a claim that every citation is apt. Whether the sentence is actually supported by the paper it cites is a different measurement, and one we do not publish, because it depends on who does the grading: three independent LLM judges scoring the same claims disagreed by nearly 28 percentage points. A number that moves that much with the grader describes the grader.
A resolving DOI proves the reference points at a real, registered paper. It does not prove the sentence citing it is supported by that paper — that is a separate measurement, and we would rather name the gap than let one number stand in for both.
Reproduce it yourself
Every number here comes from a machine-readable results file — the exact per-paper compile outcome and DOI-resolution count behind each figure. Download it and check any claim.
Download raw results (results.json)Full corpus · August 2026.