Definitions

Definitions

The pipeline

Results

1. Dedup is cheap with hashing; the work is choosing what counts as a duplicate and verifying candidates.
2. Defects matter in proportion to a model's ability to memorise, so evaluate cleaning on your real model.
3. Leakage can make a model look 17 points better than it is (memoriser, my toy).
4. A student cannot learn what its teacher cannot show, and more teacher-labelled data copies the gap.
5. Self-training removes rare cases first; keep real data in the mix.

Design rules

• Define a duplicate (level, exactness, threshold) before writing code, and verify candidates with the real similarity.
• Split by group (source document, user, conversation, duplicate cluster), never by random row.
• Check contamination between training data and every evaluation set, and report clean-subset results.
• Normalise, then deduplicate, then filter, ordering steps from cheapest to most expensive.
• Find label errors with out-of-fold predictions and review flagged rows by hand.
• Scrub personal data with broad patterns and a second check, and keep a record of what was removed.
• Verify synthetic data with tests, schema checks, judges or human samples, and keep only what passes.
• Mix real and synthetic data; track the rare cases explicitly.
• Keep lineage: record the source, the generating model and the filters for every row.
• Measure every step on a held-out set with the model you will train.

Interview questions, with the follow-ups

Common mistakes

• Random row split of data with duplicates.
• Deduplicating only exact matches.
• Skipping the verification step after LSH.
• Cleaning without measuring on the model you will train.
• Treating a flagged label as wrong without review.
• Ignoring lineage of synthetic data and benchmark overlap.
• Training only on synthetic data with no real anchor.
• Using a teacher with a blind spot and assuming more data fixes it.
• Scrubbing personal data with one regex and calling it done.
• Reporting benchmark scores without contamination checks.

What the example does not do

• Synthetic tokens, not language: a bag-of-words signal with independent words; real text has syntax, context and ambiguity.
• Small scale: about 2,500 rows; real dedup runs on billions and needs distributed LSH.
• Simple models: Naive Bayes is robust to noise; 1-nearest-neighbour is a stand-in for a memorising network, not a language model.
• Planted defects: near-duplicates are generated by five edits; all have Jaccard of at least 0.6, which is why brute force matched the ground truth exactly.
• The label-noise detector works well because my classes are close to separable bags of words.
• The collapse simulation re-estimates a categorical distribution; it is not the neural-network experiments in the published studies.
• No embedding dedup, no toxicity or copyright filters, no human review costs.
• Optional real experiment (I did not run this): take a few thousand rows of your own agent traces, find near-duplicates with datasketch or this module's code, compare a random split with a group split on a small fine-tune, and measure the score gap.

For the curious Measure each cleaning step on the model you will train

Dedup and markup stripping moved Naive Bayes by under a point but the memorising model by several. MinHash with 16 bands of 4 rows found 99.6 percent of near-duplicate pairs from 499 candidates, against 3,073,960 pairs for all-pairs comparison. A student model trained on a blind teacher inherited its blind spot. All of this is synthetic text with planted defects and tiny models, not LLM training.

What you must be able to say (one page)

• Data is the lever. Data curation rests on three criteria: quality, coverage and quantity. A small amount of high-quality data can beat a large amount of noisy data: the Yi creators found 10K carefully crafted instructions better than hundreds of thousands of noisy ones, and LIMA fine-tuned a 65B model on 1,000 curated examples to get answers equal or preferred to GPT-4 in 43 percent of cases, though less robust than product-grade models. I am quoting both from memory and did not re-check them.
• What "high quality" means: relevant, aligned with the task, consistent, correctly formatted, unique and compliant. Note "aligned", not "correct": for a task that wants concise answers, a long correct answer is bad data.
• Duplicates skew the distribution, waste compute and leak test data into training: when duplicated data is split into train and test, an example can land in one and its copy in the other. There are three detection families: pairwise comparison, hashing (MinHash, Bloom filters) and dimensionality reduction, and asks you to define "duplicate" first: whole document, paragraph or sentence; exact or 80 percent overlap.
• My dedup experiment (a synthetic pool of 2,480 rows built from 2,000 originals with 222 exact and 258 near duplicates, all defects planted so counts are exact): exact matching after normalising case and markup found 63 percent of the 501 true duplicate pairs (it misses a changed word). MinHash plus LSH banding with 64 hashes over word-bigram shingles, verified by true Jaccard similarity at 0.6: 8 bands x 8 rows 83.4 percent recall, 418 candidate pairs; 16 x 4 99.6 percent, 499; 32 x 2 100 percent, 7,707, against 3,073,960 pairs for all-pairs comparison. Keeping one per duplicate cluster left 2,001 rows (true originals: 2,000).
• The S-curve explains the settings. A pair with Jaccard similarity s becomes a candidate with probability 1 - (1 - s^r)^b; the sharp threshold sits near (1/b)^(1/r): about 0.77 for 8 x 8 and 0.5 for 16 x 4. In my data, pairs with s in 0.7 to 0.8 became candidates 49 percent of the time with 8 x 8 and 100 percent with 16 x 4 (theory 57 and 99.6 percent). Choose bands and rows from the similarity you want to catch, then verify candidates.
• What none of these catch: paraphrases (same meaning, different words). That needs embeddings, which cost more and make their own errors.
• Cleaning and filtering: strip markup, remove personal and non-compliant data, drop low-quality rows. Databricks reported that removing extraneous Markdown and HTML improved its model's accuracy by 20 percent while cutting input length by 60 percent (I did not verify this). Inspect data by eye; annotator fatigue can lower quality late in a session (reported in the annotation literature; I did not check the source). My regex scrubber removed all planted emails and phones from the 131 rows that had them; real pipelines need broader detection.
• Finding wrong labels with a model. Five-fold cross-validated Naive Bayes flagged rows where the given label had low probability. Flagging below 0.05: 218 rows, precision 74 percent, recall 77 percent of the 210 planted wrong labels; below 0.1: precision 68, recall 85; below 0.2: 53 and 90; below 0.5: 35 and 95. Review flagged rows rather than trust a threshold blindly.
• Do cleaning steps help? It depends on the model. On a clean external test set, Naive Bayes (a smooth model) changed by under 1 point from dedup or markup stripping (0.892 raw, 0.894 stripped, 0.888 after dedup) and gained a little from the label filter on 300 rows (0.824 to 0.842). The memorising 1-nearest-neighbour model moved more: 0.480 to 0.526 with the label filter on the full pool. Lesson: measure on the model you will train; a robust model hides data defects, a memoriser exposes them.
• Leakage inflates a memoriser, not a smooth model (mean of 10 splits): with a random row split, 1-nearest-neighbour scored 0.652 on the held-out part against 0.481 with a group split (duplicates kept together) and 0.494 on an external clean test; Naive Bayes scored 0.870, 0.856 and 0.888. The inflation grows with a model's capacity to memorise, which is why large language models make contamination a real risk.
• Contamination checks: the n-gram approach (GPT-3's contamination analysis used 13-token overlaps) is accurate but needs the training data; the perplexity approach is cheaper but less accurate. My 4-word n-gram check flagged 170 of 496 test rows with precision 94 percent and recall 100 percent (159 truly leaked). Reporting accuracy on the clean subset put the memoriser at 0.463 against 0.633 on all rows, matching its external score; it is good practice to disclose the share of contaminated benchmark data and report results on the clean subset.
• Synthetic data and imitation. Reasons to use it: quantity, coverage, quality, privacy and distillation. A rigorous example is Llama 3's coding pipeline (generate problems, solutions and unit tests, run linters and tests, ask the model to fix failures, keep only passing examples; about 2.7 million examples, from memory). Its four limits: quality control, superficial imitation, model collapse and obscured data lineage.
• My imitation test (Naive Bayes teacher and student, 4,000 unlabelled rows labelled by the teacher, tested on clean data; I am not claiming it is how LLM distillation behaves): a good teacher (600 balanced labels) scored 0.866 and its students 0.884 (hard labels) and 0.888 (soft labels), above the 0.784 of 150 human labels. A blind teacher that saw only 4 examples of one intent got 0.08 on that intent; its students got 0.15 and 0.25 on it (human-label model: 0.745), and overall 0.742 and 0.762, below the human-label baseline, even after adding 150 human labels (0.764, intent 4 still 0.255). A student inherits the teacher's blind spots; more unlabelled data does not fix them.
• Model collapse, toy version (my own simulation, not the published studies): re-estimate a 50-type Zipf-like distribution from 200 samples of itself each generation. Types still alive after 1, 10 and 30 generations: 39, 17 and 9 with no real data; with 10 percent real data each generation 39, 28 and 27; 30 percent: 39, 35 and 35; 50 percent: 39, 37 and 37 (50 would be the full set). Rare types vanish and never return; mixing in real data stops the loss (Gerstgrasser et al., 2024, as I recall), though there is no definitive recommended ratio and my numbers are specific to my toy.
• What to say in an interview: "I curate for quality, coverage and quantity, and measure each step on the model I will train. I deduplicate with normalisation and MinHash-LSH, then verify; I split by duplicate group; I check contamination with n-grams and report clean-subset results; I find label errors with cross-validated predictions and review them; I use synthetic data with verification (tests, judges) and mix in real data, remembering that a student inherits its teacher's blind spots."
• Honest limits: synthetic text with planted defects, simple models (Naive Bayes, 1-nearest-neighbour), small scale; no LLM pretraining or fine-tuning data; no embedding-based dedup; my noise detector benefits from knowing the data is nearly separable.