Part 1 of 5 · 1 min read
Overview
In this module you will
- Turn a list of failures into a picture: a confusion table and a few named clusters.
- Form a hypothesis from the clusters, test one change at a time, and read the result including regressions.
- Find which part of a prompt edit caused a regression by trying the parts one by one.
- Tell a flaky failure from a consistent one, and know how many runs to trust.
When a prompt misbehaves, the tempting move is to read one bad output, rewrite a sentence, and run it again. That fixes one case and quietly breaks others, and you never learn why. The method in this module replaces guessing with a short loop: collect failures, find the pattern, change one thing, measure everything.
About the examples
The model is the same imitation as modules 1 and 5, extended with a Priority: line that sets the order in which its keyword rules are tried, so there is something to fix. The failures here are caused by a rule order, which is why a one-line change can fix them. Real failures are messier and rarely respond to a single rule. Use the method; do not expect the same numbers. No real model was called.