Evaluating LLM Judges
To validate an LLM judge before trusting its verdicts:
- Pick a judging format (pairwise comparison by default; add reference-guided grading for math/code).
- Run the judge twice on each pairwise case with answer order swapped — measure position-bias consistency.
- Sample 300+ cases, collect human labels, compute agreement rate between judge and human, and compare it to human-human agreement rate.
- Only deploy the judge at scale if agreement ≈ human-human agreement (target: >80%).
Prompt template (pairwise, no reference):
"[System] Please act as an impartial judge and evaluate the quality of the
responses provided by two AI assistants to the user question displayed below.
... Avoid any position biases and ensure that the order in which the
responses were presented does not influence your decision. Do not favor
certain names of the assistants. Be as objective as possible..."
[Question] {question}
[Assistant A's Answer] {answer_a}
[Assistant B's Answer] {answer_b}
Progress:
- [ ] Step 1: Define evaluation goal (capability check vs. preference/alignment check)
- [ ] Step 2: Choose judge format (pairwise / single-answer / reference-guided)
- [ ] Step 3: Build question set covering diverse categories (writing, reasoning, math, coding, knowledge, roleplay, extraction)
- [ ] Step 4: Draft judge prompt with explicit anti-bias instructions
- [ ] Step 5: Run bias audits (position, verbosity, self-enhancement)
- [ ] Step 6: Apply mitigations (swap-and-average, few-shot examples, reference answers for math/code, chain-of-thought before verdict)
- [ ] Step 7: Collect human ground-truth labels on a sample
- [ ] Step 8: Compute judge-human agreement and compare to human-human agreement
- [ ] Step 9: Decide deployment: use as-is, restrict to certain categories, or fall back to human eval
Step 1 — Define the goal. Core-knowledge benchmarks (MMLU-style) measure capability but don't predict human preference. If the target question is "will humans like this response," use preference-based judging, not accuracy-based benchmarks.
Step 2 — Choose format.
- Pairwise comparison: best when you need relative ranking between two models; scales quadratically with number of models — avoid for >5-way comparisons.
- Single-answer grading (e.g., score 1-10): needed when there's no natural competitor to compare against, or you need an absolute quality bar; less stable than pairwise, more sensitive to judge-model drift.
- Reference-guided grading: mandatory for math/code/logic questions — first have the judge (or a strong model) independently solve the problem, then grade against that reference. Without a reference, judges frequently rubber-stamp wrong answers that "look" confident.
Step 3 — Build diverse question set. Cover multi-turn dialogue, not just single-turn. A second-turn instruction (e.g., "now rewrite starting every sentence with letter A") stresses instruction-following in a way single-turn benchmarks miss. Aim for categories spanning: writing, roleplay, extraction, reasoning, math, coding, STEM knowledge, humanities knowledge.
Step 4 — Draft prompt. Explicitly instruct the judge to: ignore response order, ignore assistant names/labels, ignore length as a proxy for quality, and explain its reasoning before giving a verdict (encourages more careful judgments and gives explainability).
Step 5 — Audit for bias.
- Position bias: run every pairwise case twice with A/B swapped. Compute consistency % (same winner both times). If consistency is low (<70%), the raw verdicts aren't trustworthy without correction.
- Verbosity bias: construct pairs where one answer is padded/longer but not more correct; check if the judge favors length.
- Self-enhancement bias: check if a model-family judge (e.g., GPT-4) systematically favors outputs from its own family over equally good competitor outputs.
- Limited reasoning: for math/logic, check judge accuracy on problems with known correct answers — judges without reference solutions perform poorly here.
Step 6 — Mitigate.
- Position bias → run both orderings and only count a "win" when consistent across both; treat inconsistent pairs as ties, or average the two verdicts.
- Verbosity bias → add explicit instruction against favoring length; spot-check with same-content-different-length pairs.
- Reasoning limitation → always supply reference answers/solutions for math, code, and fact-based questions before asking for a verdict.
- Use the strongest available model as judge (larger/newer models show measurably better position consistency).
Step 7-8 — Validate against humans. Collect independent human judgments on the same cases (ideally from multiple annotators to also compute human-human agreement as the ceiling). Compute agreement rate = fraction of cases where judge verdict matches (majority) human verdict. A judge is only as good as its agreement relative to the human-human baseline, not in absolute terms.
Step 9 — Deploy with guardrails. Even a validated judge should be spot-checked periodically, especially after switching judge model versions, changing prompt wording, or expanding to new question categories not covered in validation.
Example 1: Input: Need to compare two 13B chat models on open-ended multi-turn questions; no existing benchmark distinguishes them well. Output: Build an 80-question multi-turn set spanning 8 categories; use GPT-4 pairwise judging with swap-and-average to cancel position bias; validate on ~300 human-labeled pairs; report win-rate plus agreement-with-human percentage alongside it.
Example 2: Input: LLM judge is asked to grade a math derivation and gives a high score to a confidently-worded but incorrect final answer. Output: Diagnosis = limited reasoning ability / lack of reference. Fix = reference-guided grading — supply or have the judge independently derive the correct solution first, then compare the candidate answer against that reference before scoring.
Example 3: Input: Judge shows 90% preference for whichever answer is shown first across near-identical answer pairs. Output: Severe position bias. Do not trust raw single-order verdicts. Re-run with swapped order; only count consistent outcomes as decisive, mark inconsistent ones as ties; consider switching to a judge model with better documented position consistency (e.g., GPT-4 over GPT-3.5/Claude-v1 in comparable setups).
- Always test position bias before trusting any pairwise judge; it is the single most common failure mode.
- Prefer pairwise comparison for ranking; reserve single-answer grading for cases with no natural comparator.
- Supply reference solutions for any question with an objectively checkable answer (math, code, facts).
- Ask for explanation-before-verdict — improves both judge accuracy and human trust/debuggability.
- Report agreement-with-human alongside any LLM-judge result, not the judge's verdict alone.
- Combine capability benchmarks (MMLU-style) with preference benchmarks (LLM-judge based) — they measure different things and neither alone is sufficient.
- Use multi-turn, open-ended questions to stress-test instruction-following, not just single-turn factual recall.
- Trusting a single pairwise ordering without swap-testing — position bias can flip verdicts entirely.
- Using an LLM judge on math/code without a reference solution — judges are unreliable independent solvers.
- Assuming a chatbot that wins on human preference will also score well on MMLU/HELM-style benchmarks, or vice versa — these measure different things and can diverge sharply.
- Treating judge-human disagreement as automatic judge failure — first check the human-human agreement ceiling; some questions have genuinely low inter-annotator agreement.
- Ignoring verbosity bias — longer, padded answers often get over-scored unless explicitly guarded against in the prompt.
- Using a weak or outdated judge model and expecting the same reliability as stronger frontier models — position consistency and reasoning ability vary significantly by judge model.