CSE Roadmap

Core tier · about 5 hours · 6 parts

Part 3 of 6 · 2 min read

Reasoning prompts

Ask for steps in a format code can read, then check the steps that code can understand.

Chain of thought

Asking the model to write its reasoning steps before the final answer ("work step by step, then give Answer:"). It tends to help on problems with several steps, and it costs more output tokens and time.

For code to use the reply, you need a contract for its shape, just as in Level 2, module 3:

Think step by step. Write each step on its own line starting with "Step N:".
End with a line "Answer: <value>".

Then three small functions do the work:

parse_answer(reply)        # last "Answer:" line, or None
steps_of(reply)            # the list of steps
verify_arithmetic(steps)   # equations in the steps that are WRONG

parse_answer reads the last Answer: so a step that happens to mention the word does not confuse it. verify_arithmetic is the real benefit of structured steps: code can check the parts it understands. In the demo, 7 * 12 = 85 is caught at once, which you cannot do with a bare number.

Rules the checker follows, each with tests:

  • Usual precedence. 2 + 3 * 4 = 14 is right and = 20 is flagged. Brackets work.
  • Rounded answers are allowed to be rounded. 10 / 3 = 3.33 is right because 3.33 is 10/3 to two decimals. 3.4 is flagged.
  • What it cannot read, it skips. A line such as "so x = 3" is not judged either way.

Visible reasoning is not proof

The steps a model writes may sound convincing and still be wrong, and a correct answer can come with sloppy steps. Verify what you can with code, and treat the rest as evidence, not a guarantee. Some provider models do their reasoning in a separate setting instead of in your prompt, and adding "think step by step" to those can be unnecessary or counterproductive. Read your provider's documentation first.

For the curious Two bugs the first checker had

The first verify_arithmetic looked for exactly a op b = c, one operator. It reported correct lines as wrong in three common cases: 2 + 3 + 4 = 9 (it found 3 + 4 = 9 inside it), 10 / 3 = 3.33 (the rounding), and 2 + 3 * 4 = 14. A checker that cries wolf is worse than none, because you stop trusting it or you throw away good answers. The exercise that asked for "left to right" chains would also have taught the wrong rule, since normal arithmetic uses precedence. The checker now evaluates the whole expression safely (numbers and operators only, never eval), and seven correct and five wrong lines are tested.

Check yourself

A model writes "Step 2: 25 / 4 = 6.3". Is that flagged? (No. 25/4 is 6.25, which rounds to 6.3 at one decimal. A written value is only wrong if it differs from the rounded real value.)