Part 1 of 4 · 1 min read
Overview
Agent tuning data (tool-call traces, conversations, preference pairs) is full of near-duplicates (retries, templates), leaked evaluation items and wrong labels, and it is often produced by another model. A model trained on it inherits each defect, and an evaluation that overlaps with training will look better than the agent is. Dataset checks are the cheapest fixes in this level.
About the example, and what is real
Real: the data generator with planted defects, normalisation, exact and MinHash-LSH deduplication with verification, a PII scrubber, a cross-validated label-error detector, an n-gram contamination check, Naive Bayes and 1-nearest-neighbour classifiers, a distillation comparison, a collapse simulation, and the 25 tests in code/mini-dataset. Assumed: the text is made of invented tokens (five intents with eight signal words each, plus 40 filler words), 60 percent of signal words match the label, and the defect rates (10 and 12 percent duplicates, 8 percent wrong labels, 5 percent personal data). Not real: no natural language, no web-scale data, no model pretraining; real near-duplicates are messier and the label signal is not a bag of independent words.