We ran hundreds of theses through an updater

"Is my literature outdated?" is asked constantly and answered almost entirely by instinct. We wanted to know what the question means mechanically — so we ran an update pass over a corpus of real theses and looked at what it found.

This post is what that exercise taught us, including the parts that changed how the product works. The precise proportions publish with the first Ref[In] Index in August 2027, and we are deliberately not pre-announcing numbers that will be quoted for years before their methodology is public.

"Outdated" is six different things

The first finding was definitional, and it reframed the whole exercise. A reference can be out of date in ways that have nothing to do with each other, and only one of them breaks anything a reader would notice.

1  SUPERSEDED VERSION
   the preprint became a journal article; the
   report reached a second edition
   → your citation points at a real thing that
     is no longer the version of record

2  MOVED
   the DOI resolves elsewhere, the URL 404s
   → the only kind a reader notices immediately

3  CORRECTED
   the record's metadata changed — a title typo
   fixed, an author added
   → your entry is now subtly wrong and nothing
     announced it

4  RETRACTED
   the work was withdrawn
   → the citation still works. The retraction and its relevance need review.

5  OVERTAKEN
   the finding stands but has been substantially
   revised by later work
   → nothing mechanical detects this at all

6  STILL FINE
   old, and correct, and cited for exactly what
   it says
   → the largest category, and the one people
     worry about most

Only the second is visible without checking. Numbers one, three and four require resolving the reference against a live record. The fifth requires reading. And the sixth is the one a supervisor's comment usually lands on and almost never should.

Five of the six leave your citation working perfectly. That is why nobody finds them by re-reading their own bibliography.

How the corpus works

We test against real theses rather than constructed examples, for the reason set out when we first wrote about testing at that scale: a corpus you invent is a measurement of your own assumptions. Real documents contain things nobody designing a test case would think to include.

An update pass over that corpus does something a single run cannot. On one thesis, an odd result is an anecdote — you fix it and move on. Across hundreds, the same odd result appearing repeatedly is a class, and a class is something you can build for.

What follows are four classes that emerged this way, each of which changed the product. None was predicted; all four were obvious in retrospect.

What an update pass actually does

Worth separating from a check, because they answer different questions and people conflate them.

A CHECK asks
  does this citation correspond to an entry?
  is this entry cited?
  → answerable from the document alone
  → the answer does not change over time

AN UPDATE asks
  is this entry still the best description of
  the work it names?
  → answerable only against the outside world
  → the answer changes while you sit still

That second row is why an update pass has to be re-run and a check does not. A cross-check performed today and repeated in a year returns the same result on an unchanged document. An update performed today and repeated in a year does not, and the difference between the two runs is precisely the thing this exercise set out to measure.

Mechanically, the pass takes each reference, resolves it against live bibliographic records, and compares what comes back with what the entry says. Where they agree, nothing happens. Where they disagree, the disagreement is classified — into the six kinds above — and shown with both versions side by side. Nothing is rewritten; the output is a list of differences, each of which a person accepts or declines.

Finding one: the correction we were refusing to show

We had a rule that discarded any proposed year change when the new year did not appear literally in the surrounding text. It was built for a real failure — a citation to one organisation being matched to a differently-named body in a different year.

Across the corpus, that rule was rejecting almost every genuine correction. A corrected year is, by definition, not the year written in the text: (2620) for 2020, transposed digits, a year written one way in a citation and another in its entry. The rule was constructed so that the entire class of human error the product exists to catch could never be surfaced.

Intended to block   a different work being proposed
                    under a different year

Actually blocked    every real typo correction, because
                    a corrected year never appears in
                    the text it corrects

Result              the person who could have fixed it
                    at a glance never saw the row

It was removed and replaced with disclosure: a row whose year changed now shows the original alongside, and a human accepts or rejects it. The protection it provided turned out to be redundant — nothing is applied without a click — so it was not defending the document. It was deciding, before anyone saw the row, that a correction was not worth showing.

Finding two: the same work, cited two ways, constantly

We expected abbreviated citations to be a minor case. In the corpus they are ordinary — a work cited in full in one chapter and as "et al." in another, or under a first author in one place and a co-author in another, or with a surname transliterated differently across four years of writing.

What made this a design problem rather than a matching problem is that the naive fix is dangerous. Grouping citations that "look like" the same work will, at corpus scale, eventually group two genuinely different papers by the same author in the same year — and a wrong merge is far more destructive than a duplicate left alone, because it removes a distinction and leaves no record.

So the corpus produced a rule rather than an algorithm: group only what can be proven, propose the rest to a person, and never merge on a resemblance. Four named shapes are surfaced as candidates; none is resolved automatically.

Finding three: absence is harder than presence

The third class was the most uncomfortable, because it is about our own reporting rather than anyone's document.

Saying "this citation has no matching entry" is a claim about a whole document, and at corpus scale we found both failure directions. A citation the extractor read badly produces a false gap. And a test loose enough to avoid false gaps will, in dense citation prose, quietly suppress genuine ones — a neighbouring work's year satisfying a proximity check and hiding a reference that really was never cited.

The corpus made the asymmetry unarguable. A false finding costs a reader a few minutes. A suppressed real finding is a defect they were never told about, and they will never learn it was there. So the looser test was switched off and left in place, uncalled, until a measurement can decide whether it earns its cost — rather than being enabled on the strength of a plausible argument.

Finding four: the category we did not expect to matter

Retraction was on the list as a completeness item — one of six kinds of staleness, presumably rare. At corpus scale it behaves differently from the others, and it changed how we report it.

The distinguishing feature is that a retracted reference is not a broken citation. It resolves. The identifier works, the record exists, the formatting is immaculate. What has changed is only that the record now says the work was withdrawn — and a document citing it as supporting evidence is standing on something that has been taken away.

A MOVED REFERENCE
  the link fails
  → the reader knows immediately
  → cost: inconvenience

A RETRACTED REFERENCE
  the link works
  the record is real
  the entry is correctly formatted
  → nothing in the document indicates a problem
  → cost: a claim resting on withdrawn evidence

Reported together, the second disappears into
the first.

Which is why it is surfaced as its own finding rather than pooled with unresolved references. Nobody skimming a list of forty broken links stops on the one that is not broken.

A retraction needs contextual review. A paper analysing a retraction may appropriately cite the withdrawn work; a paper relying on its findings as current evidence may need a correction or qualification. Read the notice, identify the claim that depends on it, and consult the relevant editorial guidance. The lookup is a prompt for review, not an automated finding of misconduct or a reason to delete every mention of the work.

The coverage problem, stated plainly

One class of result caused more head-scratching than any other, and it is not a defect in anybody's thesis: references that no database we can reach happens to hold.

Bibliographic infrastructure is excellent at journal articles and uneven at everything else. A 2019 paper in a mainstream journal resolves instantly. A 1974 monograph, a national statistics bulletin, an ISO standard, a doctoral thesis from another institution, a government report that predates the department's current website — these are real, citable, frequently important, and often invisible to an automated lookup.

UNRESOLVED, because the work does not exist
  → the finding that matters

UNRESOLVED, because nothing we can query
  indexes that kind of material
  → a fact about the databases, not the thesis

Both arrive as "could not confirm".
Only the reader knows which they are looking at.

This is the strongest argument we have for reporting the reason alongside the result rather than a verdict. "Unresolved" with no explanation invites a reader to assume the worst about a perfectly good reference to a government report, and that assumption is wrong far more often than it is right.

It also biases any corpus measurement in a direction worth declaring: disciplines that cite monographs, archives and grey literature will look worse on a mechanical check than disciplines that cite journal articles, and the difference is coverage rather than quality. Any figure that does not separate the two is describing bibliographic infrastructure while claiming to describe scholarship.

What the corpus does not tell you

Three honest limits, because a corpus exercise invites more confidence than it earns.

It says nothing about any individual thesis. A class that appears repeatedly across hundreds of documents may be entirely absent from yours. Corpus findings are about the shape of a problem, not a prediction about your list.

It cannot see category five at all. Whether a finding has been overtaken by later work is a judgement about a literature, and no amount of resolving identifiers approaches it. The most important kind of outdatedness is the one this entire exercise is blind to, and any tool claiming otherwise is describing something else.

And it is shaped by what our engines can reach. Bibliographic databases are excellent at journal articles and patchy at monographs, standards, theses and grey literature. A corpus finding about "unresolvable references" is partly a finding about coverage, which is why an honest report says could not confirm rather than does not exist.

What it means for one thesis

Three things worth taking from a corpus exercise into a single document.

The staleness you can find mechanically is not the staleness you were asked about. A supervisor saying "your literature is dated" usually means category five — overtaken — and no tool detects that. What a tool does find is one to four, which are invisible to reading and which nobody has checked.

The oldest references are the least checked and the most load-bearing. Across the corpus, the references that had decayed most were disproportionately the early theoretical ones, because they were added first and re-examined least. They are also the ones holding up the framing.

And most of your old references are fine. The largest category by a distance is number six. A reference list is not a perishable good, and treating age as a defect produces worse scholarship — the corpus made that clearer, not less clear.

What we changed, in one list

The point of running a corpus rather than a case is that the output is a set of design decisions rather than a set of fixes. Four, from this exercise.

Disclosure replaced suppression. Where a rule was preventing a class of correction from being seen, it came out and the original value is now shown beside the proposal. The principle underneath: a person who can see both can decide; a person shown nothing cannot.

Resemblance stopped being sufficient. Nothing merges on similarity. Candidates are proposed with both records visible, and a wrong recommendation costs a decline rather than a distinction.

Reasons travel with results. "Unresolved" alone was actively misleading, so every finding now carries what was checked and what came back. This is more work to produce and considerably more useful to read.

And the asymmetric errors are now treated asymmetrically. Where a false positive costs minutes and a false negative costs a defect nobody hears about, the design accepts the first to avoid the second — deliberately, and stated so that it can be argued with.

A single thesis tells you what to fix. A corpus tells you what to build differently, which is a slower and considerably more useful answer.

What we will and will not claim

We are being careful about numbers here, deliberately, and it is worth saying why.

A proportion quoted before its methodology is published gets quoted for years — detached from its denominator, its corpus definition and its caveats. The figures from this work belong with the Index, where the method, the exclusions and the confidence are all stated and where anyone can re-run the measurement on the same public documents and disagree with us.

What we can say without a number is the shape, and the shape is the useful part: most references are fine, the ones that are not are largely invisible to reading, and the category people worry about is the one least likely to be a problem. Everything precise arrives in August 2027.