Duplicate references: how they multiply

A duplicate reference is one work appearing twice, and it is more interesting than it sounds — because most duplicates are not two copies of the same text. They are two descriptions of the same work that disagree, which is why searching for identical entries finds almost none of them.

Two different problems that both get called duplicates

DUPLICATE ENTRIES
  the same work listed twice in the bibliography
  → usually from a merge, or two imports of one
    record in different house styles
  → symptom: one of them is never cited, so it
    reads as an orphan

DUPLICATE CITATIONS
  the same work cited in the text under two
  different forms
  → "Al-Mansour (2019)" and "Al Mansour (2019)"
  → symptom: one form matches an entry and the
    other appears to be missing

Different problems. Both present as something
else, which is why neither gets noticed.

Notice that neither announces itself as a duplicate. A duplicate entry shows up as an orphan; a duplicate citation shows up as a missing reference. You find them while investigating something else.

Four mechanisms, and they are genuinely different

These are the shapes worth knowing, because the fix differs in each and three of them cannot be found by comparing strings.

1  SPELLING VARIANT
   two entries whose first-author surnames are
   close but not equal — an accent, a hyphen, a
   transliteration
   Al-Mansour / Al Mansour / Almansour
   → they never shared a key, so nothing groups
     them

2  A DROPPED CO-AUTHOR
   two records of one work where one author list
   is a prefix of the other
   Okafor & Lindqvist (2021)
   Okafor, Lindqvist & Duarte (2021)
   → one import lost a name

3  AN ABBREVIATION
   the same work, once in full and once as
   "et al."
   → correct in several styles, and it means the
     text names more works than the list holds

4  TWO NAMES FOR ONE ENTRY
   two citations whose surnames are BOTH authors
   of a single reference-list entry
   → the work was cited under the first author in
     one chapter and a co-author in another

Only the first is close to a string comparison problem, and even there the variants are far apart as text while being obviously the same name to a person. The fourth is invisible to any comparison of citations against each other, because the two citations share nothing — you can only see it by looking at the reference entry that lists both people.

Duplicates are not repeated text. They are disagreeing descriptions of one thing, which is a much harder object to find.

Where they come from

A merge. The dominant cause in co-authored work: two libraries, two house styles, one manuscript. Both records survive because neither is a character-for- character match for the other.

Two imports of one record. You saved a paper from a publisher's page in year two and from a database in year four. The metadata differs slightly — an initial, a journal abbreviation, a page range — and your manager treats them as distinct.

A preprint that grew up. You cited the preprint, then the published article, without noticing they are the same work. This one is a genuine duplicate that looks entirely legitimate, because the year, venue and sometimes the title really are different.

Manual entry alongside an import. You typed a reference in a hurry, then later imported it properly. Both stay.

Why we do not merge duplicates automatically

This is worth explaining, because a tool that finds duplicates and does not remove them looks incomplete until you consider the failure mode.

Merging is destructive and asymmetric. Two entries that are genuinely one work, left separate, produce an untidy list and one orphan — annoying, visible, fixable in a minute. Two entries that are genuinely different works, merged, destroy a distinction: every citation to one of them now points at the other, and nothing in the document records that a merge ever happened.

LEFT UNMERGED, wrongly
  the list has two entries for one work
  → an orphan appears
  → you see it, and fix it in a minute

MERGED, wrongly
  two different works are now one entry
  → every citation to the lost work silently
    points at the surviving one
  → nothing records the merge
  → and the evidence needed to notice is gone

So candidates are surfaced and never merged on their own. A deterministic pass groups what it can prove, and anything requiring judgement — the four mechanisms above — is proposed to a person with both records shown, who chooses merge or keep separate. Where an AI is involved, it can only confirm or reject a pairing between two entries that already exist in your document; it cannot introduce a source, and it cannot complete the merge.

The result is that the worst case is a wrong recommendation you decline, rather than a distinction quietly removed from your bibliography.

The particular trap: same author, same year, different work

The reason none of this can be automated safely is that the strongest available signal — same first author, same year — is also the signature of a completely ordinary situation.

Productive researchers publish several papers a year. Two entries reading "Okafor, T. (2021)" are as likely to be two genuine works as one duplicated one, and the only way to tell is to look at the titles and, ideally, resolve both against real records. That is why year suffixes exist, and why adding 2021a and 2021b the moment you notice the situation saves work later.

The duplicate that is not a duplicate

Before clearing anything, one distinction that catches people going the other way: several things look like duplicates and are two genuinely different objects.

DIFFERENT EDITIONS
  Okafor (2015), 2nd ed.
  Okafor (2021), 3rd ed.
  → if you cited both, both belong. Page numbers
    and content differ.

TRANSLATION AND ORIGINAL
  → two publications of one text, and which you
    read matters to a reader

PREPRINT AND ARTICLE, CITED FOR DIFFERENT THINGS
  → rare, but real: if you are discussing what
    changed between them, you need both

CHAPTER AND WHOLE BOOK
  → an edited volume and one chapter in it are
    different works with different authors

The test is not whether the entries describe the same research. It is whether they describe the same publication — because a citation points at a document, not at an idea. Two editions are two documents, and a reader following your citation needs to reach the one you actually used.

The edition case is the one worth checking deliberately, because merging two editions loses page numbers that no longer correspond, and a quotation attributed to page 84 of the wrong edition is the kind of error an examiner does find.

How to clear them, in order

One: sort by first author and read down the list. Adjacent entries by the same author are where duplicates live. This finds spelling variants and dropped co-authors quickly, because they sort next to each other or very nearly.

Two: check the ones you cannot immediately separate. Two entries with the same author and year and different titles are two works. The same author, same year and a title differing by a subtitle or a word is worth resolving against a real record before you decide.

Three: fix the citations, not just the list. Deleting the surplus entry leaves every citation that pointed at it unmatched. Find them first, repoint them, and only then remove the entry — otherwise you have converted a duplicate into a missing reference.

Four: check the preprint pairs deliberately. Search your list for the same title appearing twice with different years or venues. These do not sort together and will not look like duplicates until you notice the titles match.

What a duplicate actually costs

Mostly, credibility rather than correctness. A reference list with three visible duplicates suggests to an examiner that the list was assembled rather than curated, and the inference — fairly or not — extends to whether anything else was checked.

None of it is a misconduct question, and it is worth saying because "duplicate" sounds worse than it is. Listing one work twice in different formats is bookkeeping; nobody gains anything by it and no claim in the thesis is affected. The only version that would raise a different question is deliberately inflating a reference list to appear more widely read — which is a different act with a different intent, and not what anybody is doing when their co-author's library merged badly.

The practical harm is smaller and more specific: a duplicate splits the evidence for a claim. If you cite one work under two entries, a reader counting your sources sees two supporting references where there is really one — and so, occasionally, do you.