Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-feedback.
Done when: you report the naive and re-weighted satisfaction.
Easy 2: Design a feedback widget (30 minutes)
Sketch a thumbs control with reason codes for a support agent.
Done when: each reason code maps to a fix owner (prompt, tool, retrieval, policy).
Easy 3: Spot the bias (30 minutes)
List three reasons users who give feedback differ from those who do not.
Done when: each has a direction (over- or under-estimates quality).
Hint
think about power users, annoyed users and silent majority.
Medium
Medium 1: Estimate the rates (3 hours)
Simulate a 500-user random survey to estimate the feedback rates, then re-weight.
Done when: you report the error of the corrected estimate against survey size.
Hint
the survey gives you an estimate of the share of happy users among all users; combine it with the feedback counts.
Medium 2: Winner's curse correction (3 hours)
Hold out fresh votes to re-estimate the winner.
Done when: the held-out estimate is unbiased in simulation.
Hint
split votes into selection and confirmation halves.
Medium 3: Better triage (3 hours)
Extend triage_to_eval_cases to also include random unflagged traces.
Done when: you can estimate recall of the flag from expert labels.
Hard
Hard 1: Expert labelling study (8 hours)
Label real traces from your own agent. (I did not run this, so your numbers are the ground truth.)
Done when: you report precision and recall for each implicit signal with intervals.
Hard 2: Sycophancy eval (6 hours)
Write 30 cases where the user states a wrong belief and measure agreement rate. (I did not run this.)
Done when: you compare two prompts and report the difference with an interval.
Hint
include cases where agreeing would be harmless as controls.
Hard 3: Close the loop (8 hours)
Fix one real failure from feedback end to end. (I did not run this.)
Done when: the case is in the eval set, the CI gate passes, and the canary shows no guardrail regression.