Cohen's kappa: observed agreement corrected for the agreement expected by chance; 1 is perfect, 0 is chance.
True positive rate: the share of truly good items the judge passes.
False positive rate: the share of truly bad items the judge passes.
Self-bias, verbosity bias, position bias
Self-bias: favouring answers from the same model.
Verbosity bias: favouring longer answers.
Position bias: favouring the first (or last) option shown.
Prevalence correction (Rogan-Gladen)
An estimator that turns an observed pass rate into an estimate of the true rate: (observed minus false positive rate) divided by (true positive rate minus false positive rate).
Calibrating a judge
Write a rubric, label a set, run the judge, compare, then fix or correct.
Results
Measured kappa for a judge with true positive rate 0.9 and false positive rate 0.15, prevalence 0.6 (400 repetitions each):
Labelled cases
Median kappa
5th percentile
95th percentile
20
0.73
0.44
1.00
50
0.75
0.58
0.88
100
0.75
0.61
0.85
300
0.75
0.68
0.81
1,000
0.75
0.71
0.79
Pairwise judging (4,000 pairs; the judge sees the better answer 70 percent of the time and otherwise guesses):
Judge
Accuracy on decided pairs
Undecided
No bias
85%
0%
30% position bias
75%
0%
30% position bias, both orders, keep consistent
92%
42%
30% length bias, longer answer is better
90%
0%
30% length bias, longer answer is worse
61%
0%
Estimating a pass rate (true 30 percent; judge reports 41 percent raw; mean absolute error in points over 300 repetitions, with a balanced labelled set to measure the judge's rates):
Labelled cases
Raw judge rate
Corrected
20
11
12
50
11
7
100
11
5
500
11
2
Pairwise accuracy under position and length bias, and the error of a raw and a corrected judge pass rate.
Human ceiling: human-human kappa 0.67; judge against truth 0.73; judge against a noisy human 0.60.
A judge validated only on average agreement can still be biased in one direction. Slice the calibration set (long against short, own-model against other).
Swapping order trades coverage for accuracy; use it where you can afford to drop or re-judge the undecided pairs.
The calibration set must be big enough (tens of cases give an unusable kappa; correction with 20 labels was no better than raw).
Use the corrected rate for reporting, and the raw judge for ranking versions, if the bias is stable across versions (test that).
Design rules
Write the rubric with a scale, definitions and one example per level; ask for the score and a short reason; use a low temperature.
Prefer binary or small scales (pass or fail, 1 to 3), which are easier to calibrate than 1 to 10.
Calibrate on at least 100 human-labelled cases, balanced across outcomes and sliced by length and source.
Measure human-human agreement first, and treat it as the ceiling.
Use a different model family as judge where you can, to reduce self-bias, and strip model names from inputs.
Randomise or swap order in pairwise comparisons; control for length (instruct, or compare within length bands).
Use code where a check is exact (schema, arithmetic, state) and the judge only for fuzzy criteria.
Re-calibrate when the judge model, the rubric or the task distribution changes; pin the judge version.
Keep a human spot-check on a rolling sample.
Interview questions, with the follow-ups
What are the failure modes of LLM-as-judge?
"Self-bias, favouring its own model's answers; verbosity bias, favouring long answers; position bias in pairwise comparisons; plus rubric ambiguity and drift when the judge model changes. I mitigate by swapping order and keeping consistent verdicts, controlling length, using a different model family, and calibrating against human labels with kappa and error rates." Follow-up: "How many labels do you need?" At least about 100: in my simulation the measured kappa ranged from 0.44 to 1.0 at 20 labels and 0.61 to 0.85 at 100.
How do you know a judge is good enough?
"I compare it with human labels on a held-out set, report kappa and the true and false positive rates, slice by length and source, and compare with human-human agreement, which is the ceiling. In my example the judge agreed with a noisy human at 0.60 and with the truth at 0.73, so a low judge-human kappa did not mean a bad judge." Follow-up: "What kappa is acceptable?" It depends on the decision; I tie it to how much a wrong gate costs, and I would rather report the error rates than a single number.
Your judge says 41 percent pass. Is that the quality of the agent?
"Not necessarily. If the judge has a false positive rate of 20 percent and a true positive rate of 90 percent, the true rate is (0.41 minus 0.2) over 0.7, which is 0.30. I can estimate those rates from a labelled set and correct, but the correction is only as good as the labelled set: with 20 labels it was no better than the raw rate." Follow-up: "Can you use it to compare two prompts?" Yes if the bias is stable, which I check by calibrating each version or on a shared labelled set.
Pairwise or pointwise judging?
"Pairwise is easier for a model and humans to be consistent on, but it has position bias and scales as pairs; pointwise scales linearly and gives absolute scores but is harder to calibrate. I use pairwise to compare versions, with swapping, and pointwise with a binary rubric for gating." Follow-up: "What about ties?" Count them as no preference and report the share; a high tie rate after swapping means the judge cannot tell the difference.
Common mistakes
No human labels at all, so the judge is unvalidated.
A calibration set of 20 cases.
Calibrating on easy cases, or cases where length tracks quality.
Judging with the same model that produced the answers, without checking self-bias.
A rubric with undefined terms ("good", "helpful").
A 1 to 10 scale with no anchors.
Changing the judge model silently, so trends break.
Reporting the raw judge rate as the quality.
Using an LLM where exact code works.
What the example does not do
No language model: the judge is a parametric coin, so it cannot show rubric effects or reasoning errors.
Biases are independent additions; real biases interact with content.
Humans make independent random errors; real annotators disagree systematically on hard cases.
A fixed prevalence; the correction would shift with a different mix.
Binary labels only. Ordinal scales need weighted kappa.
Optional real experiment (I did not run this): label 100 real outputs with two humans, run a judge in both orders, and compute kappa, true and false positive rates, the swap-inconsistency rate, and the correlation between length and judge score.
Check yourself
Your judge agrees with a human at kappa 0.60 and the humans agree with each other at 0.67. Is the judge bad? (Not necessarily. Human agreement is the ceiling, and the judge is close to it. In the simulation the judge agreed with the truth at 0.73.)
For the curious A judge with a position bias needs swapped order, at a cost
A judge with a 30 percent position bias fell from 85 to 75 percent accuracy. Asking twice with the order swapped and keeping only consistent verdicts raised accuracy on decided pairs to 92 percent, but 42 percent of pairs were undecided. Also, kappa from 20 labels ranged from 0.44 to 1.0 for a judge whose true kappa was about 0.75, so use at least 100 labels. The judge here is a probability model, not an LLM.
What you will use daily at work
Practice
What it looks like
Write a rubric with an example per level
Prefer pass or fail, or a 1 to 3 scale.
Measure human agreement first
It is the most you can ask of the judge.
Swap the order in pairwise tests
Keep only verdicts that survive the swap.
Pin the judge version
Re-calibrate when the judge, rubric or task mix changes.
Draft: confirm which judge models your provider offers and how long a version stays available.
What you must be able to say (one page)
An LLM judge is a measuring instrument with error rates you must measure. Evaluation quality depends on judge calibration. Calibrate against human labels: Cohen's kappa (agreement beyond chance), plus the judge's true positive and false positive rates.
Known biases:self-bias (a model favours its own answers), verbosity bias (longer answers are preferred; studies have found judges preferring a ~100-word answer with factual errors over a ~50-word correct one), and first-position bias in pairwise comparison. Self-evaluation is still useful as a sanity check.
Position bias has a standard mitigation: ask twice with the order swapped and keep only consistent verdicts. In my simulation a judge with a 30 percent position bias fell from 85 to 75 percent accuracy; with swapping, accuracy on decided pairs rose to 92 percent, but 42 percent of pairs were undecided and dropped.
Length bias cuts both ways: with a 30 percent length bias the judge looked better (90 percent) when longer answers happened to be better, and worse (61 percent) when they were worse. A judge validated on a set where length correlates with quality will not reveal the bias.
Agreement is capped by human agreement. Humans with 8 percent label noise agreed with each other at kappa 0.67; the simulated judge agreed with the truth at 0.73 but with a noisy human at only 0.60. Do not demand judge-human agreement above human-human agreement.
Kappa needs enough labels. For a judge with true kappa about 0.75, the 5th to 95th percentile of measured kappa was 0.44 to 1.0 at 20 labels, 0.58 to 0.88 at 50, 0.61 to 0.85 at 100, 0.68 to 0.81 at 300.
A judge's raw pass rate is biased, and a labelled set can correct it. With a true pass rate of 30 percent, a judge with 90 percent true-positive and 20 percent false-positive rates reported about 41 percent; the error was 11 points raw, and after correcting with measured rates it was 12 points at 20 labels (worse than raw), 7 at 50, 5 at 100 and 2 at 500.
Honest limits: the judge is a probability model, not an LLM; the biases and rates are assumptions.