Bisect and noise: Debugging prompts

Bisect a regression

When a whole new prompt version regresses, find the guilty part. bisect_fields copies each differing field of the new prompt onto the old one separately:

bisect_fields(old, new, CASES, complete)
# [{'field': 'context',     'regressed': ['t17','t18','t19','t20','t23']},
#  {'field': 'constraints', 'regressed': []}]

Here the new version both dropped the account label from the context and added a harmless rule. The bisect says: the context change broke five account tickets; the extra rule is innocent. This is the same idea as git bisect, applied to the fields of a prompt, and it pairs with the diff and comparison from module 5.

Noise: is the failure real?

A real model can give different answers to the same prompt. Before chasing a failure, run the case several times.

fracs = repeat_check(spec, CASES, noisy_model, k=10)   # per-case pass fraction
classify_flakiness(fracs)    # always_fail, flaky, always_pass

Always fails: a real, repeatable problem. Debug it.
Always passes: fine.
Flaky: passes sometimes. Look at what makes the output vary (temperature, ambiguous instructions) before changing the prompt.

In the imitation with a 15% random slip, ten runs found 2 always-failing cases, 20 flaky cases and only 8 clean ones. A single run would have shown a random set of failures and sent you after the wrong things.

Do not over-fit the case set

If you keep tuning until every case passes, you may be memorising your cases rather than improving the prompt. Keep a second, held-out set that you look at rarely, and add new cases from real traffic (module 5).

Interview angle

"A prompt change made things worse. How do you debug it?" Describe the loop: run the full case set, compare old and new case by case, cluster the failures and read them in bulk, state a hypothesis, change one thing, measure fixed and regressed cases, bisect the edit by field if needed, repeat noisy cases before trusting a result, and add a regression case for the fix.

Check yourself

A prompt change makes 3 of 20 test cases fail and fixes 2 others. You re-run and a different 4 fail. What does that tell you, and what do you do next? (The failures move between runs, so part of it is noise. Run each case several times, compare pass rates rather than single runs, and only then judge the change.)

For the curious Why one change at a time

If you change the wording and the example together and the result improves, you do not know which change helped, and you may carry the one that hurt. The cost of testing one change at a time is small next to the cost of an unexplained regression.

What you will use daily at work

Practice	What it looks like
A saved set of failing cases	Every bug report becomes a test case before you touch the prompt.
Repeat runs	Run each case several times and compare rates, not single results.
Version history	Prompts kept in version control, so a regression can be bisected to one change.

Draft: confirm where your team stores prompts and which tool runs the test cases.