From 1 thesis to 5,000: how we test

A citation checker is only as honest as the documents it has been made to face. Ours faces a jury: a corpus of real theses and papers that every change must pass before it ships. The jury started as one document. It grew to ten, then to 105 — where it stands today — and it is being collated toward 1,000, on its way to 5,000. This is the story of that corpus: what each stage of growth taught us, and the two times it caught our own mistakes before any user could meet them.

The first thesis

Every checking tool begins the same way: with one document that the builders know intimately. Ours did too. One real thesis, read until we could recite its reference list — every citation found, every entry matched, every quirk handled. It felt like rigour. It was actually the most dangerous moment in the product's life, because a system tested against one document doesn't learn the document's rules. It learns the document's habits. Its style, its formatting hand, its author's particular way of being wrong.

One thesis can't tell you which of its features are universal and which are accidents. For that, you need a second document — and the second document's job is to break everything.

One to ten: the defects get names

It did. And so did the third, and the seventh. Growing the corpus from one to ten was less like adding test data and more like fieldwork: each new document arrived carrying some way of being a thesis that we hadn't imagined, and each one turned an anonymous failure into a named defect class we could hunt for everywhere. From those first ten came classes we still test daily:

• The duplicate bibliography — the same reference listed twice, or a whole reference section repeated after edits, silently doubling the list.
• The multi-section bibliography — references split by chapter, which a naive reader treats as several small lists rather than one document-wide one.
• The document with no reference list at all — which a checker must report honestly rather than crashing or inventing.
• The numbered thesis — Vancouver, with everything that implies; we've written that case file up separately.
• The almost-citation — fragments that look like citations and aren't, which taught us to separate chaff from loss.

Ten documents sounds small. But ten well-chosen documents — different disciplines, styles, formatting hands — already span more reality than a thousand synthetic test cases, because nobody can synthesise the ways real writers improvise.

Ten to 105: the before/after discipline

Somewhere past ten documents, a corpus stops being a pile of examples and becomes an instrument. The method that makes it one is simple to state and unforgiving to live with: before any change ships, the whole corpus is run twice — once on the current system, once on the changed one — and every count is compared. Citations found, references parsed, matches made, orphans, misses, items set aside. Any document whose numbers move gets explained, line by line, before the change goes anywhere.

document        before   after     verdict
thesis-017      214/198 → 214/198   unchanged
thesis-042      156/149 → 156/149   unchanged
thesis-063      302/280 → 302/278   ← 2 references lost. EXPLAIN
                                     BEFORE ANYTHING SHIPS.

The point of the discipline is that it is indifferent to intention. A change that was supposed to improve matching, and does, will also occasionally nudge something else — and the diff doesn't care what we hoped. It reports what moved. The corpus has caught us twice this way: two separate occasions where a change we believed in produced a small, wrong movement in documents it wasn't meant to touch. Both times the diff flagged it; both times we found a real error in our own reasoning; both times it was fixed before any user ever saw it. A test suite checks that code does what we intended. The corpus checks whether what we intended was right.

Unit tests ask: does the code do what we meant? The corpus asks the harder question: was what we meant correct?

Underneath the diff sits a bookkeeping rule we've described in its own post: every item a document contains must end up somewhere, and the somewheres must add up. Conservation is what makes corpus runs comparable — if totals must always balance, then any imbalance between before and after is a defect by definition, not a judgement call.

What a corpus catches that tests can't

The recurring lesson of 105 documents is that real writing is stranger than anything we would have thought to test. A sample of what only the corpus surfaced — each now a defect class with its own regression checks, several with their own posts on this blog:

• Author names shredded by formatting until initials migrate to the wrong surname.
• The same author spelled two ways in one document — a hyphen's difference, a match's failure.
• Sentences like "as Smith argued in 1967" that look exactly like citations and must not be treated as them.
• Two works by shared authors cited in sequence, where the second citation carries only a bare year.
• Institutions drifting into author fields, ALL-CAPS names shouting through parsers, et al. in every style's dialect.

None of these came from imagination. Every one came from a document — which is the whole argument for corpus growth. The defect classes we know about are, by construction, the ones our documents have shown us. The ones we don't know about are waiting in documents we haven't read yet.

105 to 1,000: what changes at scale

That's why the collation to 1,000 theses and papers is under way now. It isn't about bragging rights; it's about arithmetic. A defect class that appears in one document in five hundred is effectively invisible at 105 — we'd have less than a coin-flip's chance of having seen it even once. At 1,000, the rare classes start appearing reliably. At 1,000, discipline coverage also stops being anecdotal: enough medicine to exercise numbered styles hard, enough social science for author–date's every dialect, enough engineering, humanities and health science for the mixtures in between.

Scale also changes what we can say. With hundreds of documents, "most theses have a handful of orphan references" is an impression. With a thousand, it's a distribution — and distributions are what let a report tell you honestly whether your document is typical or an outlier, something we already lean on when we describe what a healthy report looks like.

To 5,000: calibration

The target beyond that is 5,000, and it has one purpose above the rest: calibration. The Ref[In] Score summarises a document's REFerence [IN]tegrity on a fixed scale, and its bands are only as meaningful as the population behind them. Scored against 5,000 real documents, "strong alignment" stops being our opinion and becomes a position in a measured distribution — comparable across documents, defensible in front of a supervisor, stable over time. (Our sister product runs the same philosophy on the currency of references — how much of a real thesis is outdated is its own evidence story.)

What we don't do with your documents

A corpus of real writing carries obligations, so three commitments, stated plainly. Documents in the corpus are handled anonymised: when their stories appear on this blog, they appear as defect classes, with identifying details altered or omitted. The corpus is used to test the system — we do not use it, or your uploads, to train AI models; the AI features in CitationLab call external models with narrow, gated questions and every suggestion remains yours to approve. And the checking that the corpus certifies is deterministic first: the same document produces the same counts, every run, which is the property that makes before/after testing possible at all.