LLM Evaluation Harness
Framework for evaluating language model outputs across factuality, format adherence, and task-specific rubrics. Designed to be extended as models change, not replaced.
Model evaluation cycle reduced from 3 days of manual review to 4 hours of automated runs across 500 test cases. Caught three separate factuality regressions before they reached production.
Every model update required 3 days of manual review with no reproducible baseline.
Evaluating whether a new model version was better or worse meant running sample queries, reading outputs, and making a judgement call. There was no quantified baseline. Two engineers could evaluate the same model swap and reach different conclusions. When a model update caused a regression in a specific task type, there was no systematic way to catch it — it had to surface through user feedback, which was always too late.
Evaluators as first-class, independently versioned objects with a standard interface.
Built a harness where every evaluator implements a common interface: it receives an input, an output, and optional context, and returns a typed score with a reason. Evaluators are composable — you can stack a factuality evaluator and a format evaluator and get both scores in a single run. The evaluator registry is separate from the test suite, so adding a new evaluation dimension does not require touching existing test cases.
Evaluator setup takes longer, but the score means the same thing across model versions.
The interface contract means new evaluators require more careful design upfront. This disciplined teams but occasionally slowed contributors who wanted a quick one-off check. The payoff is that "factuality score 0.87" means the same thing in March as it does in November, enabling real trending over time. That comparability is what makes the system useful — without it, you are generating numbers, not evidence.
- Test suite — JSONL with (input, expected_output, context)
- Input loader — batched, provider-agnostic
- Model runner — pluggable provider (Anthropic / OpenAI / local)
- Evaluator registry — factuality / format / task-specific
- Score aggregator — per-evaluator + combined
- Baseline comparator — diff against frozen reference run
- Report generator — HTML dashboard + JSON artifact