LLM Judge Variance Audit

What one run showed

On identical inputs, the judges' pass rates spanned roughly 54.6%, from the strictest judge calling almost nothing accurate to the most permissive passing more than half. Read it as raw output, not a settled result: the "No verdict" counts below are mostly judges getting cut off by untuned token limits, so these pass/fail rates sit on incomplete data.

Per-judge pass rate

Consensus breakdown

How to read this

1. This measures disagreement, not accuracy. Nothing here establishes which judge is correct, only that, on identical inputs under one rubric, they diverge. Consensus is a proxy for truth, not truth: this run carries no human-labeled ground-truth verdicts.
2. Absolute pass rates are rubric-dependent. The adversarial framing (including a simulated $500 false-pass penalty) was the single biggest lever on judge behavior. A different rubric shifts the numbers; the relative ordering between judges is the steadier signal.
3. Small-model collapse is its own effect. The smallest judges returned FAIL on nearly every response under this rubric, a fact about small-model behavior under strict framing, not a verdict on the responses. The larger local model (GPT-OSS 20B) behaved much more like the frontier API judges than like the small local ones.
4. Refusal handling is upstream. Refusal filtering happened in data prep before this audit ran. The shipped dataset contains only the 125 rows where the target model produced a response. Measuring refusal precision and recall on the source corpus would be a natural extension.

Methodology, in brief

Each judge sees the same adversarial scoring rubric and returns PASS or FAIL with a short justification. The rubric uses a simulated financial incentive structure (bonus for catching subtle errors, penalty for false-pass) and explicitly scopes judges to grade only the model output, not the prompt. Consensus is computed by simple majority across the judges that returned a verdict for a given row.