Skip to content
Call wiki

How do you test an LLM grader against human ground truth?

Last updated 2026-08-17 · Confidence: documented — Anthropic’s eval docs and engineering guide plus published LLM-judge and reliability studies; the held-out-set discipline follows a practitioner guide (reported).

Treat the LLM grader as one more rater joining your team: it grades a human-labeled sample without ever seeing the answers, a separate script scores its output against those labels, and it must clear the same agreement threshold you demand of human graders — measured on cases its prompt was never tuned on.

Blind grader, separate scorer

Section titled “Blind grader, separate scorer”

The grader sees the rubric and the submission, never the ground-truth label — otherwise agreement measures leakage, not judgment. A deterministic script then compares its outputs to the labels; Anthropic’s eval guide says to constrain the output (“Output only the number and nothing else”) so scoring stays automatic, and to use a different model as evaluator than generator. Its agent-evals guide adds: calibrate judges “closely… with human experts”, grade each rubric dimension with an isolated judge, and give the model an “Unknown” escape hatch against guessing.

Agreement metrics: reuse the human bar

Section titled “Agreement metrics: reuse the human bar”

Teams that already calibrate human graders with an intraclass correlation coefficient can run the identical test on the LLM. The standard interpretation guideline: 0.75–0.90 is good reliability, above 0.90 excellent. For binary pass/fail grades, prefer precision and recall over raw percent agreement — imbalanced classes inflate raw agreement (judge-building guide). The ceiling is real: GPT-4-class judges reached over 80% agreement with humans — the human-human level — while showing position, verbosity, and self-enhancement biases.

Hold out cases the prompt never saw

Section titled “Hold out cases the prompt never saw”

Iterate the grader prompt against one labeled subset, but report agreement only on untouched cases — “calculate the error on unseen data only” (judge guide). Patching the prompt failure-by-failure overfits it to those failures. Two Anthropic closers: “prioritize volume over quality” — many auto-graded cases beat few hand-graded ones — and never take scores at face value until someone reads the actual transcripts.

Sources

Section titled “Sources”