Part 1 of 5 · 1 min read
Overview
In this module you will
- Build a draft, critique, revise loop and put a stopping rule on it.
- Tell apart a second look by the same model and a second look from outside (a test, a validator, a calculator).
- Predict when self-critique spoils a correct answer, and measure it.
- Write checks whose messages tell the writer what to fix.
- Keep lessons from one failure for the next job, in the style of Reflexion.
Agents make slips: a wrong format, a missed requirement, a wrong number. An obvious fix is "ask the model to look again". Sometimes that works very well, and sometimes it adds cost and quietly makes answers worse. The difference is what the second look can see. A critique from something that has new information (a test result, a validator, the real order record) is reliable. A critique from the model re-reading its own text has no new information, so it is only as good as the model's ability to notice and to not invent problems.
About the examples
Everything here is an offline imitation. The "writer" is a program that makes six kinds of slip at fixed random rates and revises according to what it is told. The "critic" is a program with a chosen chance of finding a real problem and a chosen chance of inventing one. These rates are my assumptions, not measurements of any real model, so the numbers show the shape of the trade, not what your model will do. The checks, however, are real code run on real text. Measure your own model on your own task before believing any figure. No real model or service was called.