Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-coding.
Done when: you can explain each mode's merged and truly-correct counts.
Easy 2: Add a task (30 minutes)
Add a seventh bug with a visible test, a held-out test and two candidates.
Done when: the gated loop handles it and the results change as you predicted.
Hint
make the visible test pass for a wrong patch to see the held-out test matter.
Easy 3: Read a gate finding (30 minutes)
Print pr_summary for the gaming patch.
Done when: you can explain each finding in one sentence.
Medium
Medium 1: Evade the gate (3 hours)
Write three gaming patches the gate misses and three legitimate patches it wrongly blocks.
Done when: you list each and propose one improvement that does not add false positives.
Hint
dictionaries, any() over data and renamed helpers are classic gate evasions.
Medium 2: Feedback in the loop (3 hours)
Change the proposer so the test output chooses the next candidate (a simple rule-based stand-in).
Done when: you compare attempts with the fixed order.
Medium 3: Sandbox it (4 hours)
Run the test step inside a container or a restricted subprocess with no network and a memory limit.
Done when: a test that tries to open a socket or allocate too much memory fails safely.
Hint
containers with --network none and memory limits are the easiest start.
Hard
Hard 1: Real model on your repo (8 hours)
Run a real model through the harness on five bugs in a repository you own. (I did not run this, so your numbers are the ground truth.)
Done when: you report merged, truly correct, attempts, cost and any gaming you saw.
Hint
record the model's raw patches so you can replay them offline.
Hard 2: Multi-file change (8 hours)
Extend the workspace to atomic multi-file edits and a rollback on failure. (I did not run this.)
Done when: a failing edit leaves the repository unchanged and a test proves it.
Hard 3: Review bot (8 hours)
Build a second agent that reviews the diff and the gate findings and compare it with human review. (I did not run this.)
Done when: you report agreement on twenty merged changes and the cases it missed.