Voting over samples: Examples and reasoning prompts

Ask several times and take the most common answer. The maths, and why real gains are smaller.

Self-consistency (voting)

Ask the same question several times with some randomness (Level 1, module 4), then take the most common answer.

If each sample is right with probability p and the samples are independent, the chance that more than half are right is a sum you can compute exactly:

expected_vote_accuracy(0.7, 3)   # 0.784
expected_vote_accuracy(0.7, 9)   # 0.901

Voting when each sample is 70% right. Grey is the exact lower bound; green is a simulation that takes the most common answer.

Samples	Exact (more than half right)	Simulated (most common answer)
1	0.700	0.700
3	0.784	0.779
5	0.837	0.887
9	0.901	0.975

The simulation is higher for larger n because the most common answer can win without being a majority: the imitation's four kinds of wrong answer are spread out, so they split their votes. In real models the wrong answers are often the same wrong answer, which makes the exact column the more honest one, and even that is optimistic. Two cautions:

Real samples from one model are not independent. The same misunderstanding repeats, so real gains are smaller than this idealised picture (exercise Hard 2 shows it).
Every extra sample multiplies cost and time.

Voting also does nothing when p is below one half: more samples then make a wrong majority more likely.

Use voting where mistakes are expensive and answers are short.

What you will use daily at work

Practice	What it looks like
Example banks	Examples live in a file with their labels, are reviewed like code, and are checked for leakage against the test set.
Answer contracts	A fixed `Answer:` line or JSON field lets the code read a reasoning reply.
Cheap checks on reasoning	Arithmetic, units and format are verified by code before the answer is used.
Voting only on critical steps	Extra samples are spent where an error is costly, not on every call.

Draft: confirm how your chosen model handles reasoning (in the prompt or as a model setting) and what it costs, before relying on this.

Interview angle

"When would you use few-shot examples, and when chain of thought?" Examples for format, boundaries and custom labels. Reasoning for multi-step problems where you can check steps. Say you measure both on a held-out set, because they cost tokens, and mention that voting trades cost for reliability under an independence assumption that only partly holds.