I had AI read my field

Hacker News by 10 min read 138x views
I had AI read my field

Share Post

Theoretical astrophysics · a entire site · audited by machine

158,059 document and 554,670 citations. Every document alongside complete content screened, 659 of the most crucial recomputed and checked, and all verified error followed through the document that citation it. This is the map, and what it says concerning whether discipline corrects itself.

Why

Errors that nobody fixed

I spent five years in this site doing my PhD at Caltech, and a few months of that went to errors in the writings that were already known. The equations my code was built on came from Bishop, Gómez, Lehner, Maharaj & Winicour, “High-powered gravitational news” (1997), the document the subfield's trait codes descend from. People knew it had mistakes. The fixes were scattered through several 10,000 lines of intersecting email threads that contradicted all other, and the document itself was never corrected. The finest I could do was accurate the document in my own papers: an appendix environment out the complete set of equations, made consistent, and a parenthetical “with a sign error in s3 corrected” for anyone study closely.

That is the bounds of individual writings review. Mine was not deep: my three thesis document citation small than a hundred document between them, in a site of tens of thousands. Nobody re-derives all equation they cite, referees inspect one document once, and nobody is paid to go rear and fix the original. So all pupil who builds on a damaged document pays the identical tax, one at a time, and the correction lives in somebody's inbox. A device can peruse the entire field, recompute it, and prosecute all error to all document that copied it. So I had one do it. It established errors in my own papers too.

The map

Every dot is a paper

Height is time: the 1960s at the bottom, this twelvemonth at the top. Each faint row is a citation. The brilliant dots are the document everything alternatively leans on, classified the way Google classified the first web.

The kernel is TAPIR, Caltech's theoretical astrophysics group, anywhere I did my PhD: 4,093 document by its 326 members since 2004. Add all document they citation and all document that cites them and you get 158,059 document joined by 554,670 citations. The 49,301 drawn current are the ones tied to TAPIR at smallest three times over.

A hundred years in ten seconds

Watch it grow

The map holds 499 document from the 1970s, 2,035 from the 1990s and 18,976 from the 2010s. The old layers are the survivors, the document group operating today motionless cite. Everything alternatively from 1975 has already been forgotten, correct or wrong.

Most of what this site knows was written in the final twenty years, by group who are motionless about to answer email. Hold that thought.

Subfields

A subfield is a column

Nobody told the layout what numerical relativity is. Papers that citation all another drag together, so a subfield shows up as a column: the identical few dozen group penning concerning the identical issue twelvemonth following year, all tier position on the last.

There are 32 columns, from accelerated broadcast bursts to the Milky Way's halo.

One column

Numerical relativity

Solving Einstein's equations on a device until two dreary holes merge. 278 document in this column, 71 of them TAPIR's, busiest in 2010.

Every subfield has a page: the questions it is organised about and whether all is open, contested or settled, who wrote the chief document and where, how accelerated it publishes, and which of its errors we traced.

Each open inquiry additionally carries a emblem for what today's AI could do concerning it. Of 155, a example judged 8 answerable now from data and codes that already exist, 119 ones AI would speed up, and 28 stuck waiting on a scope or a detector.

Numerical relativity → All 32 subfields

Reading everything

Every document gets screened

Reading a document carefully expenses a few dollars of example time. Screening one expenses nothing, so all document I could get the complete content of got screened: 50,269 papers, 421 myriad words, checked for the fingerprints of the worst kinds of trouble. That covers 76% of the document drawn here. Nothing before 1991 is on arXiv and journal-only document aren't open, so the remainder got lone the checks that run on a repository record. (Gold marks TAPIR's own papers.)

  • Paper-mill language: none.
  • Chatbot residue and hidden instructions to AI reviewers: none.
  • Text shared between document alongside no author in common: 43 pairs in 34,286 papers, led by successive study catalogues and data releases. The remainder are queued for a individual look.
  • Retractions: 13 among 184,733 document alongside a DOI. None is TAPIR's.
  • TODO and FIXME notes shipped to arXiv: 62 papers.

Astrophysics is comparatively clean. Whatever is incorrect alongside this literature, it isn't fraud. The one needle that moves is vocabulary: “delve”, “crucial” and allies are 2.5 times as average in 2025 as in 2016.

Reading carefully

The 200 that matter get checked row by line

Citations are brutally unequal, and a few hundred document transport the field. So the 200 most-cited TAPIR document of 2006–2019 (gold) all got their own AI reviewer alongside the LaTeX source, tools to run its own calculations, and orders to recompute everything it could: re-derive the equations, test the limits, rebuild the numbers from the paper's own tables. It worked blind, alongside no admission to errata or afterward papers.

2,623problems flagged, 84 major

112leaned on by 3+ afterward papers, re-derived by a second reviewer

56stand as errors (34 in full, 22 in part)

50 · 6judgement calls · thrown out

A emblem is not a finding. Only what a second reviewer upholds is called an error here.

What equal assessment caught

Thirteen fixed, fifty-six through

About 136 of the 200 were revised between the archetypal preprint and the published paper, so we peruse those archetypal preprints the identical way. 13 verified errors were in the preprint and gone from the diary version, fixed by the referee or by the authors on a second appearance (purple).

56 went through assessment and into the diary (gold). On this evidence equal assessment removes concerning one error in five. At smallest two of the thirteen were copied by afterward document anyway, from the old preprint.

One error, followed

A aspect of four, 2005–2014

Smith, Kamionkowski & Cooray change the gravitational waves remaining complete from cost increase into an energy density a detector could see. Two factors of one fractional went missing, so the answer is four times too big: a 6 in the denominator anywhere the textbook has 24. This one was double-checked. The paper's numbers in the paper's equation reproduce its result; the identical numbers in the norm equation provision a fourth of it.

Watch the years run. Red document copied the aspect of four. The green one stated it was wrong, in 2008. The copying carried on until 2014: six copies, one objection.

See all 872 →

All 872

Five copies for all criticism

Then we mold the net wider. The identical recompute-and-verify check, run on 459 additional still-cited document throughout the map, mostly extremely cited ones that a cheaper archetypal peruse had flagged, upheld 816 additional errors. That makes 872 in 433 papers. 159 of them were copied, incorrect value and all, into at smallest one afterward paper. Counting papers:

367 document copied an error. 68 criticised one.

Only 47 of the 872 were always criticised in print, and 21 of those kept being copied afterwards, by 51 additional papers. These are floors: we peruse lone the citing document whose content we hold.

And 872 is a flat for the site too. Only 4% of the extremely cited document on the map got the thorough care so far. Scaled up from the document that did, the apt figure of undetected errors in this field's document cited 100 or additional times is concerning 34,000 (25,000–48,000). How that figure is reached →

How lengthy an error lives

Half are motionless being copied 7 years later

How productive is equal assessment and publishing at detecting and expunging errors from the literature? We can really measure it.

0%25%50%75%100%024681012 years42% at twelvemonth 9

Share of copied errors motionless being copied, by years since the document appeared. Band: 95% interval. 33 errors are old adequate to be observed at twelvemonth 10.

The curve is motionless at 52% at twelvemonth 7 and crosses one fractional at twelvemonth 8; at twelvemonth 10 it is at 39%. Counting document that lean on the affected outcome without showing which value they used, it crosses at twelvemonth 9. The errors we traced most thoroughly final longer, crossing at twelvemonth 10: a shallower hunt merely finds small delayed copies.

What eventually stops an error? Of the 56 errors in the 200 document peruse most closely, 52 are motionless in the latest arXiv version, anywhere a fix is free. Of the 200 document peruse most closely, the repository records an erratum for 2. And of the 87 errors that did die out, 67 were never criticised by anyone. They faded whenever the subfield moved on, which retires correct answers and incorrect ones at the identical rate.

How far we got

4% of the document that matter, so far

Every document alongside open complete content was screened and skimmed. The costly check, the one that recomputes, has reached 659 papers, starting alongside the most cited. Each bar is the document on the map alongside that many citations, filled by how far they have been read.

All document on the mapFull content screenedHaiku skimSonnet readOpus recompute, verified

02,0004,0006,0008,00010,00012,00014,000Papers0 citations: 2,541 papers; Full content screened 1,463 (58%); Haiku skim 27 (1%); Sonnet peruse 0 (0%); Opus recompute, verified 0 (0%)2,5411–2 citations: 2,448 papers; Full content screened 1,618 (66%); Haiku skim 41 (2%); Sonnet peruse 0 (0%); Opus recompute, verified 0 (0%)2,4483–9 citations: 4,968 papers; Full content screened 3,493 (70%); Haiku skim 545 (11%); Sonnet peruse 0 (0%); Opus recompute, verified 0 (0%)4,96810–29 citations: 8,147 papers; Full content screened 6,286 (77%); Haiku skim 3,231 (40%); Sonnet peruse 26 (0%); Opus recompute, verified 0 (0%)8,14730–99 citations: 13,265 papers; Full content screened 10,759 (81%); Haiku skim 9,002 (68%); Sonnet peruse 742 (6%); Opus recompute, verified 10 (0%)13,265100–299 citations: 11,077 papers; Full content screened 8,972 (81%); Haiku skim 8,823 (80%); Sonnet peruse 1,047 (9%); Opus recompute, verified 324 (3%)11,077300–999 citations: 5,051 papers; Full content screened 3,591 (71%); Haiku skim 3,535 (70%); Sonnet peruse 512 (10%); Opus recompute, verified 256 (5%)5,0511,000–2,999 citations: 1,324 papers; Full content screened 718 (54%); Haiku skim 706 (53%); Sonnet peruse 67 (5%); Opus recompute, verified 60 (5%)1,3243,000+ citations: 403 papers; Full content screened 122 (30%); Haiku skim 117 (29%); Sonnet peruse 10 (2%); Opus recompute, verified 8 (2%)40301–23–910–2930–99100–299300–9991,000–2,9993,000+Times the document has been cited

Coverage in item →

A site outgrows its errors. It rarely corrects them.

All of this would disbursal concerning $2,720 in example period bought at catalog price, approximately $3 per verified error. In custom it ran inside a Claude Max subscription. The entire project, this site and the others, took concerning 4.9 milliard tokens and concerning 44 hours of my period complete 41 days. It is one field, peruse by a first-generation method, alongside limits spelled out on the method page. And it is not finished: the profound inspect has reached lone 4% of the extremely cited papers on the map, most cited first. The remainder of the queue is a prosperity decision, and so is the next field.

Errors were established by one AI meeting and re-derived by a second, autonomous one, and all was confirmed in the type the diary published. Five were additionally double-checked stage by stage against the source. No individual has checked any of them. As a test, 40 drawn at random went to a third meeting that saw lone the document and the passage: it established 33 of the 40 wrong, so several named current volition not last scrutiny. If we have your document wrong, say so and we volition fix it and say that we did.

Other Article Hacker News
↑
Close Right Ads
Close Left Ads