Collect failures first: Debugging prompts

Collect failures first, then look

Rule one: never debug from a single example. Run the whole case set (module 5) and look at all the failures together.

rep = run_suite(BASE, CASES, complete)      # 30 cases, 24 pass
clusters(rep)
# [('expected account, got billing', ['t21', 't22', 't24']),
#  ('expected bug, got billing',     ['t14', 't15', 't16'])]

clusters groups failures by signature ("expected X, got Y"). Six failures became two groups of three, and both say the same thing: tickets whose real topic is a bug or an account problem were read as billing.

Expected label (rows) against the model's label (columns), from the example. Real output.

The confusion table makes the pattern visible at a glance: everything off the diagonal sits in the billing column. Then read the failing tickets themselves, looking for a common thread. distinctive_words helps by listing words that occur in failures more than in passes (payment, cannot, invoice). The thread here: a money word appears, and the money rule is tried first.

Name the cluster before you fix it

Write the hypothesis as a sentence: "bug and account tickets that mention money words are classified as billing". A fix that does not address a stated hypothesis is a guess.

Change one thing at a time

One-factor-at-a-time

Compare each candidate change alone against the same base prompt on the same cases. If you change three things and the score moves, you do not know which one mattered, or whether one helped and another hurt.

rows = single_changes(BASE, CHANGES, CASES, complete)

Four candidate changes, each tried alone. Real output of the example (imitation model).

Change	Pass rate	Fixed	Broke
base	80%
"be careful with tickets that mention several topics"	80%	0	0
`Priority: bug > billing`	90%	3	0
`Priority: account > billing`	83%	3	2
`Priority: account > bug > billing`	93%	6	2

What to read from it:

A vague warning did nothing. Telling a model to "be careful" rarely changes behaviour, and here the imitation ignores it entirely. Prefer concrete rules and examples (module 2).
The cleanest change fixes three and breaks none.
The highest score still broke two cases that used to pass: t07 ("I was charged after I changed my login email", which is truly billing) and t13 ("The dashboard freezes after login", truly a bug). A pass rate alone would have hidden this. Always look at the regressed list, then decide whether the trade is acceptable or whether a better rule is needed.