Why we never tighten extraction

The most frequent suggestion we get is also the one we always decline: make the extractor stricter so the review screen comes back clean. Here is why the answer has been no every time, and what we do instead.

The request is reasonable on its face. A user opens the review screen, sees rows that are obviously not citations, and concludes the extractor is imprecise. Tighten it, and the noise goes away.

It does. So does something else, and that is the whole argument.

The two errors, and why only one of them is recoverable

An extractor can be wrong in two directions. It can collect something that is not a citation, or it can fail to collect something that is.

These are not equivalent mistakes, and no amount of engineering makes them equivalent.

  COLLECTED, NOT A CITATION      NOT COLLECTED, WAS A CITATION

  appears on your screen         appears nowhere
  finds no reference             is never looked for
  shows as missing               shows as nothing
  you recognise it               you cannot recognise
  you dismiss it                 what you cannot see

  cost: a moment of your time    cost: a real gap, reported
                                 as no gap at all

The right-hand column is the one that ends a viva badly. A citation the extractor never took is not flagged, not counted, and not mentioned — the report says the chapter is clean, and it is wrong in the one direction a report must never be wrong in.

A tool that misses quietly is worse than no tool, because it also supplies confidence.

Tightening trades a visible cost for an invisible one

Every rule you add to make the extractor stricter is a rule that some real citation will eventually fall foul of. Not many — that is what makes it dangerous. A rule that excluded a tenth of real citations would be caught in testing immediately. A rule that excludes one in four hundred passes every test we have and fails on somebody's thesis in a year's time.

And the failure is silent by construction. There is no error, no warning, no row on any screen. The user's experience is of a tool that worked perfectly, right up until an examiner asks about a source that is in the text and not in the bibliography.

So the trade is: a screen that looks messier, in exchange for a class of error that cannot be detected by the person it happens to. We are not willing to make that trade, and the fact that the cost we keep is the visible one is exactly why the request keeps arriving.

The citations a resemblance rule drops first

It is worth being concrete about what falls through, because "some real citations" is easy to wave away in the abstract. The ones that go first are the ones that least resemble the tidy example a rule is written against.

  (Ministry of Works and         an organisation as author —
   Housing, 2018)                no surname anywhere in it

  (Adeyemi, in press)            a citation with no year at
                                 all to anchor it

  (Okonkwo, 2019;                broken across a line, so the
   Adeyemi, 2020)                two halves look like fragments

  (Nakamura et al., 2017,        a page range and a suffix, on
   pp. 44-61b)                   a shape a rule may not expect

None of those is exotic. An organisation as author is ordinary in policy research; a citation with no year is ordinary in a thesis citing work still in press; a line break falling inside a bracket is ordinary in any two-column layout. Each one is a citation a careful reader would recognise instantly and a tightened rule would decline.

They also cluster. A thesis in public health cites organisations constantly, so a rule that struggles with them does not lose one citation from that thesis — it loses a systematic slice, all from the same chapter, all invisible.

What we tighten instead

The rule is narrow: collection stays generous; the screen is where judgement lives. Every improvement goes into the pass that separates, never into the pass that collects.

The difference matters because the two passes fail differently. If the screen is wrong, the row is still there — set aside, visible, one click from being reinstated. If collection is wrong, there is no row to be wrong about. One mistake is in front of you; the other does not exist as far as the software is concerned.

That is why the screened pile is never hidden, and why a reinstatement is recorded with who made it. Both properties only mean something if the row survived collection in the first place. We wrote about what that screen looks like from the other side in the checker said 90% of my citations failed.

The version of this we did accept

One refinement passes the test, and naming it is the clearest way to show where the line sits. Reference-list spans are excluded from in-text collection.

When the extractor walks the document, it knows where the bibliography begins and ends, and it does not harvest in-text citations from inside it. A reference entry contains a surname and a year, so without that exclusion every entry in your bibliography would be collected a second time as though it were a citation in the body.

That is a tightening, and we made it — because it removes a class of row that cannot be a real in-text citation, on the strength of where it sits rather than what it looks like. It excludes by structure, not by resemblance. Nothing real can fall through it, which is the only kind of tightening that is safe.

Every request we have declined has been the other kind: a resemblance rule, excluding rows that look unlikely. Looking unlikely is not the same as being wrong, and the difference is where somebody's missing reference lives.

What this does not claim

Generous collection is not a substitute for good extraction. The extractor is expected to read the document correctly — to handle a superscript, a bracketed range, an organisation as author, a citation broken across a line. That work is continuous and it is not what this post is about.

And none of it makes a busy screen a finding about your writing or a suggestion of misconduct. A long screened pile usually means tables, a numbered style, or careful cross-referencing of your own figures. It is a property of the document's shape, and the report treats it as exactly that. The related read is what a healthy report actually looks like.