The evaluation theatre problem in LLM systems
When the model can write its own tests, passing tests stops being evidence of correctness.
There's a failure mode in LLM evaluation that I've started calling evaluation theatre: the appearance of rigorous measurement without the substance of it. It's surprisingly easy to build a benchmarking system that produces numbers nobody should trust.
The problem has a specific shape. You want to evaluate whether your LLM system produces accurate outputs, so you write test cases. But test cases require expected outputs, and expected outputs are often generated by — the model. You've now built a system where the evaluator and the evaluated are the same entity, or share significant capability overlap. The model can optimise its outputs toward the test distribution in ways that look like improvement but aren't.
This is most visible in LLM-as-judge evaluations. If you use GPT-4 to evaluate GPT-4 outputs, you're measuring something closer to stylistic consistency than correctness. The judge assigns high scores to outputs that look like what GPT-4 would produce, which is tautological. The scores are high. The system looks excellent. Production tells a different story.
The deeper issue is that evaluation requires a ground truth that's independent of the system being evaluated. For narrow tasks — information extraction, structured output generation, code that runs or doesn't — this is achievable. You can write test cases with ground truth derived from authoritative sources external to the model. For open-ended generation tasks, independent ground truth is genuinely hard. The honest answer is that we don't have good general-purpose evaluation methods for tasks where correctness is inherently subjective.
My working approach: separate the evaluation question from the generation question. If you're evaluating factuality, use a factuality evaluator that operates on specific, verifiable claims extracted from the output — not on the output holistically. If you're evaluating format adherence, write a parser that checks structural constraints programmatically, not a prompt that asks the model whether the format looks right. Move as much of the evaluation as possible into a regime where the answer is not model-generated.
The hardest cases are the ones where you can't escape the model in the loop. There, I'd argue for using multiple judges from different model families, treating their agreement as a confidence signal rather than their individual scores as a correctness signal, and reporting the disagreement rate as part of your evaluation output. A system where judges consistently disagree is telling you something about the task, not just the model.
Good evaluation is mostly about what you refuse to measure. The metrics you remove because they're not actually measuring what you care about are as important as the ones you keep.