Part 3 of 4 · 2 min read
Exercises
Easy
Easy 1: Run it (30 minutes)
Run python -m pytest -q and python run_all.py in code/mini-judge.
Done when: you report the kappa spread at 20 and 300 labels.
Easy 2: Kappa by hand (30 minutes)
Two raters label 20 items: 10 both pass, 6 both fail, 2 each of disagreement.
Done when: you compute kappa and compare with judge.kappa.
Hint
po = 16/20; compute chance agreement from the marginals.
Easy 3: Correct a rate (30 minutes)
A judge has true positive rate 0.85 and false positive rate 0.10 and reports 52 percent.
Done when: you compute the corrected rate.
Hint
(0.52 minus 0.10) divided by (0.85 minus 0.10).
Medium
Medium 1: Length-balanced set (3 hours)
Build a calibration set where length is uncorrelated with quality and compare the judge's apparent accuracy with the biased set.
Done when: you show how much of the accuracy came from the length bias.
Hint
sample pairs so that the longer answer is better in half of them.
Medium 2: Ordinal scale (3 hours)
Implement weighted kappa for a 1 to 3 scale.
Done when: a test shows that adjacent disagreements are penalised less than distant ones.
Hint
weight disagreements by squared distance.
Medium 3: Cost of undecided pairs (2 hours)
Add a third judging pass for undecided pairs.
Done when: you report accuracy, calls per pair and tie rate for one, two and three passes.
Hint
only re-judge pairs whose two orders disagreed.
Hard
Hard 1: Real judge (8 hours)
Calibrate a real LLM judge against two human labellers on 100 outputs. (I did not run this, so your numbers are the ground truth.)
Done when: you report kappa, error rates, order inconsistency, a length slice, the model and the date.
Hint
freeze the judge prompt, model version and temperature.
Hard 2: Self-bias test (5 hours)
Judge answers from two models with each as the judge. (I did not run this.)
Done when: you report the preference for own-model answers and whether human labels agree.
Hint
strip model names from the text.
Hard 3: Judge drift (5 hours)
Re-run a fixed set monthly or after a model change. (I did not run this.)
Done when: you report score drift and the point at which re-calibration would be triggered.
Hint
keep the labelled set fixed and unseen by prompt tuning.