Skip to content
The Daily Triptych158 / 365
Pairwise near-duplicate structure

Cell shading marks estimated overlap between documents. Two dark off-diagonal blocks are near-duplicate clusters; pale cells are unrelated pairs a deduplicator leaves untouched.

Try it in the local lab

Exact-duplicate pass on a local text folder

Before near-duplicate sketching, collapse byte-identical files with ordinary hashing. This is safe on your own data and shows how much of a crawl is pure repetition.

$ mkdir -p ~/dedup-lab && cd ~/dedup-lab
$ mkdir -p corpus_raw corpus_unique
$ python3 - <<'PY'
from pathlib import Path
raw = Path('corpus_raw')
raw.mkdir(exist_ok=True)
samples = {
  'a.txt': 'The same mirrored bulletin.\n',
  'b.txt': 'The same mirrored bulletin.\n',
  'c.txt': 'A genuinely different note about weather.\n',
  'd.txt': 'The same mirrored bulletin.\n'
}
for name, text in samples.items():
    (raw / name).write_text(text, encoding='utf-8')
print('wrote', len(samples), 'files into corpus_raw')
PY
$ find corpus_raw -type f -exec sha256sum {} \; | sort > hashes.txt
$ python3 - <<'PY'
from pathlib import Path
seen = {}
unique_dir = Path('corpus_unique')
unique_dir.mkdir(exist_ok=True)
kept = dropped = 0
for line in Path('hashes.txt').read_text().splitlines():
    digest, path = line.split(None, 1)
    src = Path(path)
    if digest in seen:
        dropped += 1
        print(f'drop exact copy: {src.name} == {seen[digest]}')
        continue
    seen[digest] = src.name
    dest = unique_dir / src.name
    dest.write_text(src.read_text(encoding='utf-8'), encoding='utf-8')
    kept += 1
print(f'kept {kept}, dropped {dropped}; unique files in corpus_unique/')
PY
$ wc -l hashes.txt && ls corpus_unique

This only removes exact byte matches. Near-duplicates need shingling and MinHash (or a dedicated tool such as a suffix-array pass) and are slower; run those on a copy of data you own, never on corpora with licences you cannot satisfy.

II · THE IDEA · ARTIFICIAL INTELLIGENCE

Deduplication of Massive Training Datasets

data · Near-duplicate removal · arXiv:2107.06499 · Quality, speed, less memorisation

▶ Listen · narrated

A model trained twice on the same article does not learn it twice as well. It mostly burns compute — and becomes more willing to quote the article back.

At a glance

What it is
Finding and removing near-duplicate documents and repeated sequences before training
Two levels
Whole documents that nearly match, and long identical substrings inside documents
Main aims
Cleaner data, less wasted compute, weaker verbatim memorisation
Typical tools
Hashing for exact matches; MinHash-style sketches and suffix arrays for near matches
Side benefit
Reduces train–test overlap when benchmarks were scraped from the same web

Think of a choir handed the same photocopied lyric sheet twenty times mixed into a pile of different songs. Rehearsal time spent on the twentieth copy of that sheet does not teach a new melody; it only makes that one song absurdly familiar. Web training text is full of those photocopies: mirrored pages, reposted articles, identical footers.

Deduplication is the pass that finds those copies and keeps roughly one. Exact copies are found by fingerprinting whole files. Near copies — same article with a different headline strip — are found by comparing compact sketches of their word chunks. A further pass looks inside files for long repeated paragraphs shared across many pages and keeps those paragraphs once.

After that, training spends more of its effort on text that is actually different. Models also become less prone to recite a training paragraph word for word, because they no longer saw it on every other batch. The method cannot catch two articles that say the same thing in entirely new words; it only catches overlap you can show in the strings themselves.

Look closer

  1. Duplicate is not only identical

    Exact byte-for-byte copies are the easy case: a hash collapses them. The harder mass is near-duplicates — the same news wire with a different byline, a Wikipedia page mirrored with a banner, a boilerplate licence repeated across thousands of files. Those still teach almost nothing new, yet they survive naive unique-line filters because a few tokens differ.

  2. Sequences inside documents matter too

    Even after document-level filtering, long identical spans can remain: quotes, templates, repeated code headers, or the same paragraph pasted into many pages. Matching on substrings catches that second layer. A model that sees one long span many times is the model most likely to emit it later when prompted near its start.

  3. Evaluation can be contaminated by the same copies

    If a benchmark example, or a close paraphrase of it, sits in the training crawl, reported gains partly measure leakage rather than generalisation. Deduplicating training text against evaluation sets — not only against itself — is how that particular illusion is reduced. The residual risk is paraphrases too distant for string methods to catch.

The story

Large language models are trained on text gathered at web scale. That text is not a tidy library. The same article is republished, scraped, archived, and lightly rewritten across countless URLs. Template sites generate near-identical pages by the thousand. When those copies are left in place, gradient steps are spent re-learning material the model has already fitted, and rare genuine signal is diluted by sheer repetition of the common copies.

Deduplication is the family of procedures that try to notice this before training begins. At document level, each page is reduced to a compact sketch — often a bag of hashed n-grams in the MinHash style — so that pairs with high estimated Jaccard overlap can be clustered and collapsed to a single survivor. Exact duplicates fall out even more cheaply with cryptographic or rolling hashes. What remains is closer to a set of distinct documents than to a multiset of whatever the crawler happened to fetch most often.

A second pass works inside documents. Suffix-array or related string methods find long repeated substrings that cross document boundaries: the same paragraph appearing in dozens of otherwise different files, or a fixed legal notice glued to every download. Those spans can be dropped or retained only once. The distinction matters because a model does not experience “a document”; it experiences tokens in context. A repeated span is repeated supervision regardless of the URL it came from.

The motives are practical rather than aesthetic. Training on a leaner corpus finishes sooner for a given compute budget, or lets more diverse text fit in the same budget. Models trained after aggressive deduplication tend to show stronger quality on held-out work that was not itself a hidden copy of the train set. They also tend to emit long training passages verbatim less often — a direct comfort when the corpus may contain personal or copyrighted text that nobody intended the model to recite.

None of this is free of judgement calls. How near is near? Too strict a threshold discards genuine variants that still carry signal; too loose a threshold leaves the sludge in place. Different domains — code, news, forums, books — have different natural rates of repetition, so a single global cutoff is a compromise. And string overlap is blind to paraphrase: two articles that restate the same facts in fresh prose will usually both survive, which is often desirable and occasionally not.

What the procedure does cleanly is attack the copies that hashing and string matching can prove. In web-derived corpora those copies are not a curiosity at the margin. They are a structural feature of how text circulates online, and leaving them untouched quietly reshapes what the model is actually optimised to do.

Why it mattered then

As soon as training sets grew from curated collections into automatic crawls, duplication stopped being a minor data-cleaning chore and became a first-order training choice. Crawls amplify whatever is easiest to republish. Without an explicit deduplication stage, a training run silently overweight whatever the web had already copied most — not whatever was most informative. The line of work around deduplicating training data made that mechanism visible and showed that removing the copies improved the models that resulted, rather than merely shrinking the files on disk. It also reframed memorisation: regurgitation is not only an architectural quirk, but partly a consequence of how often a span was shown.

Why it matters now

Open and proprietary training mixtures still lean on web text, code hosts, and mirrored archives. Every new crawl reintroduces the same republishing patterns. Deduplication remains one of the few pre-training interventions that simultaneously trims cost, hardens evaluation against leakage, and reduces the rate at which models reproduce long training stretches. For anyone assembling a local mixture for open-weight fine-tuning or continued pre-training, the same logic applies at smaller scale: repeated README files, duplicated issue threads, and scraped mirrors waste context and warp the loss just as they do in a frontier run. The tools are ordinary; the decision to run them is still what separates a corpus from a pile of downloads.

The surprising detail

The same near-duplicate that wastes training compute can also inflate benchmark scores. When an evaluation example, or a near copy of it, appears in the train set, a model can look capable on that example without having generalised. Deduplication against the evaluation data is therefore not only hygiene for training efficiency — it is part of whether a reported number means what readers think it means. String methods still miss paraphrased leakage, so even a clean report is a lower bound on the problem, not a proof of innocence.

What is disputed

Magnitudes — how much of a given crawl is duplicate, how large the quality or memorisation shifts are — depend on the corpus, the overlap threshold, and the model scale. Treat those as results tied to particular experimental setups, not as universal constants. A second listed source on doubly special relativity is unrelated to this topic and contributes no evidence here.

Remember this

Deduplication does not make text wiser; it stops the same text from counting as new evidence over and over.

Test yourself

You have already removed exact duplicate files by hash. Training loss still seems to overfit a handful of boilerplate passages that appear inside otherwise different pages, and a benchmark score looks oddly high on a news-summary task. What second form of deduplication addresses both symptoms, and what can it still miss?

Go deeper

Image: Original diagram, The Daily Triptych. Licence: Original work. Source.

← Back to day 158