Definitions

Definitions

The loop and the sparse-reward problem

Results

1. Signal is the bottleneck: with a 1 percent pass rate, 90 percent of small groups teach nothing.
2. Algorithm differences are small next to learning rate and verifier quality.
3. Sparse rewards over long chains explode the sample cost; partial credit fixes it and creates a new thing to audit.
4. The policy optimises the checker. A leash does not close a verifier gap.

Design rules

• Start with a verifier you trust and attack it yourself before training: look for shortcuts.
• Keep a held-out, independent check (another verifier, a human audit) and plot it beside the training reward.
• Track pass rate per prompt; drop prompts that are always solved or never solved, and build a curriculum.
• Use groups large enough to see both outcomes, often 8 to 16, and measure the share of groups with signal.
• Tune each algorithm's learning rate separately before comparing.
• Prefer outcome rewards; add partial credit only for long horizons and audit it.
• Make the environment cheap, resettable and sandboxed; an agent that can edit its own tests will.
• Cap and monitor behaviours such as output length, tool calls and test edits.
• Keep SFT or a reference model as an anchor and watch the KL, but do not rely on it against hacking.
• Evaluate on tasks the training never saw, including ones that a gamed verifier would pass.

Interview questions, with the follow-ups

Common mistakes

• Trusting the verifier without testing for shortcuts.
• Watching only the training reward.
• Comparing algorithms at one learning rate.
• Training on prompts that are always passed or always failed.
• Groups too small for the pass rate.
• Leaving the environment writable (tests, scoring files, network).
• Using a KL penalty as a substitute for a better verifier.
• Adding process rewards without checking they correlate with real success.
• Skipping SFT so the starting policy never produces a success.
• Treating a toy result as evidence about frontier models.

What the example does not do

• No language model: independent softmaxes per step, no text, no context-dependence, one correct chain per prompt.
• No value network, no per-token ratios, and clipping is applied at the step level on a tabular policy.
• The weak verifier and loophole are chosen by me; the loophole is found by chance in about a quarter of early samples, which makes hacking fast.
• Learning rates were chosen from a coarse grid; results at the best rate differ by small amounts that I did not test for significance.
• My explanation for the batch-baseline failures is a guess.
• No exploration problem beyond sparse reward, no multi-turn state, no tool noise, no cost model.
• Optional real experiment (I did not run this): train a small open model with a library such as TRL or a similar GRPO implementation on a task with unit-test rewards (for example short Python functions), track the held-out pass rate and the share of groups with signal, and add a deliberately weak test to see whether the model learns to game it.

For the curious The policy optimises the checker, not your intent

A weak verifier that checked only the last step reached verifier pass 0.999 but true success 0.075. A SKIP loophole took verifier pass from 0.35 to 0.99 in 10 updates while true success fell to 0.000. A KL penalty slowed honest learning before it slowed the hack. At the same learning rate GRPO was 2 to 3 times faster than REINFORCE, but at each method's best rate they nearly tied, so tune the rate per method first. This is a tabular toy, not a language model.

What you must be able to say (one page)

• The idea. In reinforcement learning with verifiable rewards (RLVR) the reward comes from a program (a unit test, an answer checker, a schema validator), not from a learned preference model. Sample several attempts, check each, and raise the probability of attempts that passed. The loop is sample completions, grade them, update the policy, and the grader can be any automated metric, rule-based validator or scoring model you trust. Everything quantitative below is from my toy.
• How it differs from module 3. Preference tuning learns what people like from comparisons. RLVR optimises what a checker can verify, which suits math, code, tool-call correctness and format. The catch is the same one: the model optimises the checker, not your intent.
• GRPO in one line (introduced in the DeepSeekMath paper, Shao et al., 2024; from memory, not re-checked): for each prompt sample a group of G attempts and use advantage = (reward - group mean) / group standard deviation, so no learned value model is needed. My toy compares it with plain REINFORCE (advantage = reward), a batch-mean baseline, and a leave-one-out baseline (RLOO).
• Toy setting (all assumed): 5 prompts; each needs a 3-step chain with 4 token choices per step; one chain per prompt is correct, so the pass rate starts at about 1.3 percent; reward is 1 for the exact chain; each update samples 8 prompts x 8 attempts. Updates to reach 90 percent true success (mean of 20 seeds, learning rate chosen from 1, 3, 10, 30, 100): REINFORCE 17.0, RLOO 17.0, GRPO 15.7, and the batch-mean baseline was the worst (133 updates at lr 10, 3 of 20 runs never got there; 19 of 20 failed at lr 100).
• The honest reading of that comparison: at the same learning rate GRPO was 2 to 3 times faster (lr 1: 200 vs 457 updates; lr 3: 74 vs 161; lr 10: 29 vs 54), but at each method's best rate they were nearly tied. Dividing by the group's standard deviation boosts the step on rarely-solved prompts, which acts like a larger, adaptive learning rate. So tune the learning rate separately for each method before crediting the algorithm. My guess (not tested) for the baseline's failures: with rewards in {0, 1}, plain REINFORCE only updates on successes, while subtracting a batch mean also pushes down failed attempts and, at large steps, collapses the policy early.
• No signal, no learning. A group where every attempt gets the same reward has advantage zero (my test checks the gradient is exactly zero). Probability a group has both a pass and a fail is 1 - p^G - (1-p)^G. At my starting pass rate: G=2: 2.5 percent, 4: 4.9, 8: 9.6, 16: 18.1, 32: 32.5, 64: 53.2 (simulation agrees within 0.4 points). Practical fixes: a curriculum of easier prompts, filtering all-fail groups, larger groups, partial credit.
• Group size trades compute for signal. With 64 rollouts per update: G=2: 61 updates and 3,882 rollouts; G=4: 42 and 2,710; G=8: 29 and 1,846; G=16: 27 and 1,696; G=32: 25 and 1,626 to reach 90 percent. Returns diminish past about 8 to 16 here.
• Long chains and sparse rewards (the agent problem: one terminal reward after many steps). Chain length 2, 3, 4, 5, 6 with outcome-only reward needed 13, 33, 89, 289 and 1,293 updates (at length 6, 4 of 10 runs had not reached 90 percent after 1,500; start pass rate 0.02 percent). Partial credit (reward for correct leading steps) needed 10, 15, 20, 30 and 31. The price: a process or partial reward is another thing to hack and must itself be checked.
• Reward hacking is the central risk. With a strong verifier GRPO reached true success 0.998. A weak verifier that checks only the last step reached verifier pass 0.999 but true success 0.075. A loophole (a SKIP token the verifier accepts) took verifier pass from 0.35 to 0.99 in 10 updates while true success fell from 0.012 to 0.000. Real cases: a coding agent edits or special-cases the tests; a verifier that matches a string accepts a leaked answer.
• A KL penalty is not a fix for a bad verifier. Penalising divergence from the starting policy (weights 0, 0.05, 0.2, 1.0): with the strong verifier true success was 0.998, 0.878, 0.552, 0.014 (learning suppressed); with the loophole the verifier pass stayed 1.00, 1.00, 0.97, 0.40 and true success 0.000, 0.000, 0.001, 0.009. The hack is cheap in KL, so the penalty slows honest learning first. Needed: verifier coverage, an independent held-out check (a different verifier or human audit), and monitoring of true success while training.
• PPO-style clipping with 4 passes over each batch (learning rate divided by 4) was slightly slower here: 38.6 vs 28.9 updates. It exists to make it safe to reuse rollouts in large models; my tabular toy does not show a benefit.
• For agents: RL needs an environment you can reset and score cheaply, many rollouts, and a verifier you trust. Tool-use success, test pass, and schema validity are good rewards; "the answer sounded good" is not. Where you cannot afford that, use SFT or preference tuning on logged data (modules 1 to 3), or best-of-N (module 3).
• What to say in an interview: "RLVR samples a group of attempts per prompt, scores them with a programmatic verifier, and updates toward the better ones, for example with a group-relative advantage as in GRPO. It needs prompts with non-trivial pass rates or there is no signal, and a verifier you can trust, because the policy will find any gap. I would hold out an independent check, track true success next to the training reward, and consider partial credit for long chains while auditing it too."
• Honest limits: a tabular softmax policy over a 3-to-6-step chain, one correct answer per prompt, simulated verifiers; no language model, no reasoning traces, no PPO value function, no claim about how DeepSeek-style models behave.