Why the judge matters.
When fifteen frontier language models graded the same 22,252 responses, their average scores diverged by 2.5 points on a 10-point scale. The model you pick to judge your model determines who wins.
The one-judge problem.
Every time an AI lab publishes a benchmark result, someone chose a judge. Usually it's GPT-4. Sometimes it's Claude. Occasionally it's a panel. The choice is rarely foregrounded. It should be.
Over the past two months we ran what we call a blind peer matrix: ten frontier models simultaneously generating answers to the same question, then blindly scoring each other's responses across five dimensions. Self-judgments excluded. Identity masked. Order randomized. We ran 286 of these across nine category pools and 198 questions we wrote from scratch.
The result is 22,252 valid judgments — to our knowledge, the largest publicly available multi-judge LLM evaluation dataset with full provenance. The chart below shows a finding that fell out almost immediately, before we got to the more interesting analysis: how much grades depend on who's grading.
Figure 01
Ask GPT-5.4, you get a 7.2. Ask Mistral Small, you get a 9.7.
- Stricter than average
- Near median
- More lenient than average
What this means.
Think about what that 2.5-point spread does to any benchmark. Two models with genuinely identical output quality could be ranked two places apart if evaluated by different judges. A model that wins against GPT-5.4 as judge might lose against Mistral Small. In neither case would the comparison reveal anything about the models being tested — only about the grader.
This is not a problem that goes away by averaging more responses through the same judge. It goes away by using many judges simultaneously and measuring where they agree, where they don't, and why. Which is what the Multivac does.
Two smaller findings we think matter. First: OpenAI models as judges exhibit much higher variance than Anthropic or Google models. GPT-5.4 has a score standard deviation of 2.22 across its judgments; Granite 4.0 Micro compresses all scores into a 0.49-point band. Second: judge strictness does not predict judge usefulness. Some of the stricter judges produced the most stable rankings; some of the most lenient produced the noisiest.
Findings
Three takeaways.
-
Report your judge.
Any leaderboard number without a judge citation is an incomplete claim. The judge is half the experiment.
-
Use more than one.
Panel-style evaluation isn't a nice-to-have. A single-judge result on modern frontier models is closer to an opinion than a measurement.
-
Strictness is not skill.
Picking the harshest grader doesn't make your benchmark more rigorous. It just picks the harshest grader.
Read the full analysis
The full paper — the blind peer matrix.
Twenty-two thousand judgments. Fifty-five models. A corrected within-response analysis showing that the apparent same-vendor bias in Mistral and Google is an artifact of judge leniency and response quality — not favoritism — while a same-vendor scoring premium survives the controls for only Anthropic (robust) and MiniMax (significant but judge-dependent under leave-one-judge-out); Qwen does not survive two-way clustering. That premium is an upper bound on favoritism, not a measurement of it — the estimator cannot separate preferential scoring from a sibling judge parsing same-distribution output more accurately. Full dataset open-sourced under MIT.
"The last question was asked for the first time, half in jest, on May 21, 2061..."