The seven steps of a Ref[In] annotation

Everything up to this point has been diagnosis: what is cited, what is listed, what is missing, what resolves. Annotation is where that becomes a document — an annotated paper, an annotated bibliography, a verified link on every reference, and the exports you actually take away.

It runs as seven stages in a fixed order, on a frozen session, by one worker. This post is why each of those constraints exists, because every one of them was added after something went wrong without it.

It will not start until the ledger balances

The first thing to understand is not a stage at all. Annotation is gated, and the gate is arithmetic rather than judgement.

Throughout the check, a ledger tracks every item read out of your document against every item accounted for in the result. Annotation cannot begin until that ledger balances — not "mostly balances", not "has been reviewed". The ruling behind it is unusually blunt, and the word doing the work is literally:

"I did not just mean that if all the extraction bucket was not poured into review then review should not have run — I meant this literally, that if the ledger was not balanced then it should not run."

Read from the document                          412

  matched to a reference                        268
  in the bibliography only                      121
  set aside as not a citation                    23
                                                ───
  accounted for                                 412     balanced

Versus

  accounted for                                 389     23 unexplained
                                                        → annotation does
                                                          not begin

The distinction that matters: a non-empty ledger is not a balanced one. A stage that received 900 of 1,000 items has a ledger; it does not have a balance. Building an annotated document over an unexplained gap would produce something that looks complete and is not — and an artefact that certifies an unbalanced ledger certifies nothing.

Then everything before it freezes

At the moment you press the button, the earlier phases go read-only. Cached, unchanging, incapable of being edited by anything — including by you — until the run finishes.

That sounds restrictive and it fixes a specific class of fault. These stages used to be started from four different places earlier in the pipeline, which meant they ran against a citation list that was still free to change underneath them. Link enrichment would be built against one version of your list and an export against another. Both would complete successfully. Neither would agree with the other, and nothing anywhere would say so.

Two stages that each succeeded, against two different versions of the same document, produce a result that is wrong in a way no error message can describe.

The seven stages

1  PAPER          your document, annotated in place —
                  every citation marked with what it
                  resolved to

2  BIBLIOGRAPHY   the reference list, annotated — each
                  entry with its status and its evidence

3  PRECOMPUTE     everything the later stages will need,
                  worked out once instead of per export

4  ENRICHMENT     a real, verified link for EVERY
                  reference — one lookup per entry

5  VERIFICATION   the checks over the assembled result

6  EXPORTS        every export format, built together

7  EMAIL          your files — and only once they exist

Stages one and two are the artefacts most people came for: the paper with its citations marked up, and the bibliography with a status on every entry. They come first because everything after them is derived from them.

Stage three, pre-compute, is unglamorous and exists for a boring reason: several exports need the same derived values, and computing them once rather than once per format is the difference between a run that takes minutes and one that takes considerably longer.

Stage four is the expensive one. Link enrichment makes a network call per reference — on a real thesis that is over six hundred lookups, which is why it is a stage of its own rather than a step inside another. It is also the stage most people do not expect, because it runs over every reference in your list rather than only the ones the check flagged.

Stage five verifies the assembled result rather than its parts. The parts were checked as they were built; this asks whether what came out the other end is internally consistent.

Stage six builds every export together, from one settled state, so that the Word document, the PDF and the reference exports cannot disagree with each other — the same reason the freeze exists, applied at the end instead of the beginning.

What stage five actually verifies

"Verification" is doing a lot of work as a word, so it is worth saying what it checks and what it cannot.

It checks the assembled result against itself. Every citation marked in the annotated paper should correspond to an entry in the annotated bibliography; every link attached during enrichment should belong to the entry it is attached to; every count reported should equal the thing it counts. These are internal consistency questions, and they are the ones that catch a defect introduced by the assembly rather than present in your document.

"Does this citation have an entry?"
   answered in the CROSS-CHECK, long before annotation

"Does this entry describe a real work?"
   answered in ENRICHMENT, by lookup against records

"Do the artefacts we just built agree with each other?"
   answered in VERIFICATION — and only here

What verification cannot do is re-open the earlier questions, because the session is frozen and the earlier answers are the input. That is the trade the freeze makes: the result is internally consistent and reproducible, and it is consistent with the document as it stood when you pressed the button rather than as it stands now.

There is one finding worth separating from all of this. Enrichment can report that a reference resolves to no record it can reach — and, as we discuss when reading any single result, that has five quite different causes. Four are mechanical. The fifth is that the work does not exist, and citing work that was never written is academic misconduct. The annotation stage does not adjudicate which of the five you are looking at; it attaches the evidence and the reason, and leaves the reading to you.

And the email goes last, deliberately

Stage seven is sending your files, and it is ordered rather than merely intended. The rule, in the code's own words: an email must never leave without the file it announces.

It is a small thing that is disproportionately damaging when it goes wrong. An email saying your documents are ready, arriving before the documents exist, converts a successful run into a support conversation and a user into someone who no longer believes the next message either. Making it the last stage rather than a notification fired alongside the work is the whole fix.

Why one worker stays with your session

A single runner walks the stage list from start to finish. If another worker is free it may take the next session — but never the middle of this one.

The reason is partly correctness and mostly honesty. A worker that switches sessions halfway through looks identical, from the outside, to a worker that has died: the stage list stops advancing, the panel stops narrating, and both users watching their own runs see something that appears stuck. Keeping one runner on one session until it is done means that when progress stops, something is actually wrong — which is what makes the progress display worth reading at all.

How a stage reports itself, and why a tick was not enough

This is the part we got wrong first, and the correction is the most useful thing in this post if you are building something similar.

Originally a stage was either settled or not — a boolean. Which meant a stage that finished and a stage that died landed in the same bucket, and a dead pre-compute rendered in the panel as a tick. The run had failed, quietly, and the interface was reporting success.

COMPLETE   the work happened

DEAD       the job settled without doing it
           → looks identical to COMPLETE under a boolean

LIVE       something is provably still working on it

UNKNOWN    never started, or nothing left that can answer

A tick over a dead stage is the same lie as a four-hour spinner, told in a friendlier voice. Both tell a user that the system knows what is happening when it does not, and the friendly version is worse, because the spinner at least prompts someone to ask.

The same reasoning produces two stall clocks rather than one. A run may legitimately sit inside a single stage for a long time — six hundred lookups is not quick — so the clock that condemns a run is deliberately slower than the one that condemns an individual job. Past a hard limit, a claim to be alive stops being evidence of anything and the run ends rather than spinning indefinitely.

What an annotated document actually contains

The word "annotated" is doing real work here and is worth unpacking, because it is not a summary bolted onto the front.

The annotated paper is your document, with every in-text citation marked in place. Each mark carries what that citation resolved to — the entry it matched, the record behind that entry, and where the two agreed. A reader can move through your argument and, at any citation, see the evidence for it without leaving the sentence.

The annotated bibliography is your reference list, with a status on every entry rather than only the ones that gave trouble. An entry that matched cleanly says so and shows what it matched against; an entry that could not be resolved says which of the reasons applied. The entries you never worried about are annotated too, which is the point — a list where only the problems are marked tells a reader nothing about the rest.

Okonkwo, C. B., & Ferreira, L. (2022). Workflow variance
in regional centres. Health Systems Review, 14(3), 201–218.

  cited at        §4.2, §5.1, §5.4
  resolved        record retrieved · authors, year and
                  title agree
  link            direct, verified
  status          matched

That cited at line is small and does more than it looks. It answers the question a supervisor or examiner actually has when they stop on a reference — where did this get used? — which ordinarily requires searching the document. It is also the line that makes an orphan obvious: an entry with nothing beside cited at is a reference in your list that your text never uses.

And every reference carries a link, direct where one could be verified and a search fallback where it could not. The distinction is deliberate rather than a shortfall: a direct link that goes to the wrong paper is worse than a search link that goes to a results page, because the first is confidently wrong and the second is honestly incomplete.

When a stage fails

Seven stages that must run in order raises an obvious question: what happens when one of them does not finish?

The run stops at that stage and says so, naming it. It does not skip ahead, and this is the whole reason the order is enforced rather than merely intended — a pipeline that continued past a failed pre-compute would build exports from values that were never computed, and those exports would be produced successfully, delivered by email, and wrong.

What you get instead is a run that stopped at stage three, with stage three named, and no email. That is a worse experience in the moment and a much better one an hour later, when the alternative would have been a document you had already started relying on.

DEAD          the job settled without doing the work
              → the run stops here, and says so
              → nothing downstream is attempted

LIVE, slow    something is provably still working
              → the run continues to wait
              → six hundred lookups take as long as
                six hundred lookups take

The difference between these two is the difference
between a failure and a queue, and reporting them as
the same thing is how a user is taught to ignore
progress displays.

Two clocks govern the waiting rather than one. A run may legitimately sit inside a single stage for a long time, so the clock that condemns an entire run is deliberately slower than the one that condemns an individual job. And past a hard limit, a claim to be alive stops being evidence of anything: no stage of this pipeline takes two hours, so beyond that the run ends rather than spinning indefinitely on a job sitting in a lane nothing consumes.

Why it is a step you choose

Annotation does not start automatically when a check finishes, and that is deliberate in two directions.

It is expensive — six hundred lookups is the bulk of the compute in a whole session — so running it on every check would mean charging everyone for an artefact most people do not need at that moment. A candidate diagnosing their reference list in week one does not want an annotated export; they want to know what is missing.

More importantly, it is the wrong moment. Annotation freezes your session and builds a document from it, which is only worth doing once the list is in the state you intend to submit. Running it before the reference work is finished produces a beautifully assembled artefact of an unfinished bibliography. The button exists so that you decide when the document is ready to be fixed in place, rather than the pipeline deciding for you.

Freezing a document is only useful when you are ready for it to stop changing.

What you have at the end

When all seven complete, the session reaches its third and final level. The first is Organising — the cross-check has run and you know what is missing. The second is Finalise — the reference work has been done. The third is Export, and it means something specific: the files exist.

That is the distinction worth ending on. Levels one and two are claims about your document. Level three is an artefact — a paper and a bibliography in which every citation carries what it resolved to, every reference carries a verified link, and every claim the earlier levels made can be checked by someone who was not there when you made it.

A report you can hand to a supervisor is only useful because of what stands behind it, and what stands behind it is seven stages that ran in one order, on a frozen document, over a ledger that balanced before any of them began.