What is an LLM evaluation framework? A vendor-neutral guide

What is an LLM evaluation framework?

An LLM evaluation framework is the infrastructure that answers one question continuously: is this AI system actually working, by our definition of working? Not "does the demo impress people" — but does version B beat version A on the cases that matter, and would we know within an hour if a model swap made things worse?

Concretely, it's four layers:

1. Dataset — a versioned collection of real inputs (and, where possible, reference outputs) drawn from your actual workflow. Not synthetic toy prompts: the tickets, documents, and queries your system will really face, including the ugly ones.
2. Scorers — functions that grade an output. These range from deterministic checks to semantic similarity to LLM-as-judge rubrics to human review queues.
3. Harness — the runner that pushes every dataset case through your system (prompt + model + retrieval + tools, the whole pipeline) and collects scores reproducibly. Same inputs, same code path as production.
4. Gates and dashboards — the part that makes it a framework instead of a report: CI gates that block a release when scores regress, and a dashboard a non-engineer can read.

If any layer is missing, you don't have evaluation — you have vibes with extra steps. The dataset without a harness is a spreadsheet; scorers without gates are trivia; a dashboard without real cases is theater.

Why bother? Because ungated AI systems fail silently

LLM systems degrade in ways traditional software doesn't: a provider silently updates a model, a prompt tweak fixes one case and breaks nine others, a retrieval index drifts stale. Without a framework, each of these is discovered by a user — or worse, by nobody.

This is also the mechanism behind the most-quoted failure number in the industry: MIT's finding that 95% of enterprise AI pilots deliver no measurable P&L impact. A pilot with no evaluation baseline can't demonstrate improvement, can't be tuned against a target, and can't survive a budget review. Systems that learn require a definition of better; the framework is that definition, made executable.

What metrics actually matter?

The honest answer: the ones tied to your workflow's definition of failure. Metric taxonomies are a menu, not a checklist:

• Deterministic checks — format validity, JSON schema conformance, required facts present, forbidden content absent, length bounds. Cheap, fast, zero ambiguity. Most teams under-use these; they catch a surprising share of real regressions.
• Reference-based metrics — exact match, semantic similarity to a known-good answer. Great where ground truth exists (extraction, classification, structured drafting).
• Task-family metrics — for RAG: faithfulness (is the answer grounded in retrieved context?), answer relevance, context precision/recall. For agents: task completion, tool-call accuracy, trajectory efficiency (did it take 4 steps or 40?).
• LLM-as-judge — a model scoring outputs against a rubric. The only scalable option for fuzzy qualities (tone, helpfulness, reasoning quality). Powerful, and biased — next section.
• Human review — the calibration anchor. You never escape it entirely; you economize it, spending human labels where automated scorers are least trustworthy.

A useful discipline: for each metric, write down the decision it drives. "Faithfulness < 0.9 blocks release." A metric that drives no decision is decoration.

LLM-as-judge: powerful, and here's how it lies to you

Judge models made evaluation affordable, and every platform now leans on them. Use them — with their failure modes in view:

• Length and confidence bias — judges systematically prefer longer, more assertive answers, independent of correctness.
• Self-preference — judges score outputs from their own model family higher; grading GPT outputs with a GPT judge inflates scores.
• Position bias — in pairwise comparisons, the first-presented answer wins more than it should. Randomize order or use both orders.
• Judge drift — upgrade the judge model and your entire score history silently re-bases. Pin the judge version; treat judge upgrades like schema migrations.

Treat an LLM judge like an untrusted classifier you happen to employ: measure its agreement with ~100 human-labeled examples before you believe anything it says, and re-calibrate whenever you change it.

The tool landscape, honestly

Here's the part most guides won't give you, because most guides are written by one of these companies. Every tool below is legitimate; every one also has an incentive shaping its advice. Declared bias is the closest thing to neutrality this market offers:

Tool	What it is	Who's behind it / what they'd like you to buy
DeepEval	Open-source pytest-style eval library, strong metric catalog	Confident AI — hosted eval platform
promptfoo	Open-source CLI/CI evals + red-teaming, config-driven	Promptfoo Inc. — enterprise/security tier
RAGAS	Open-source RAG-specific metrics (faithfulness, context quality)	Exploding Gradients — hosted app layer
LangSmith	Tracing + datasets + evals, deeply integrated with LangChain	LangChain — platform subscription; happiest if you're all-in on their stack
Braintrust	Polished eval + logging platform, good team UX	Venture-backed platform — the platform is the product
Evidently	Open-source ML/LLM monitoring + eval library	Evidently AI — cloud offering
Arize Phoenix	Open-source tracing/eval, OpenTelemetry-friendly	Arize — enterprise observability platform
W&B Weave	Eval + tracing inside the Weights & Biases ecosystem	W&B/CoreWeave — seat expansion in their suite
MLflow	Open-source experiment tracking with LLM eval modules	Databricks — the lakehouse around it
lm-evaluation-harness	EleutherAI's academic benchmark runner (MMLU etc.)	Nonprofit — but it benchmarks models, not your application; often mistaken for an app-eval tool
OpenAI Evals	Open-source eval registry and framework	OpenAI — an ecosystem gravity well, and quiet since the API-based eval products arrived

Selection heuristic: RAG system → start with RAGAS metrics inside DeepEval or promptfoo. Agent system → trajectory evals in Phoenix or DeepEval. Team of 5+ needing shared visibility → trial Braintrust or LangSmith, but export your datasets weekly. Benchmarking foundation models themselves → lm-evaluation-harness, and know that's a different job.

Build or buy?

Split the question by layer. Your dataset and failure definitions must be built — they encode your domain, your customers, your risk tolerance. No vendor sells them, and they're 70% of the value. The harness is thin glue: a few hundred lines or an open-source library; never a reason to buy. Platforms earn their money on team UI, tracing, and production monitoring — buy one when multiple people need to look at the same scores daily, and keep your dataset and scorers in portable formats so the platform stays replaceable.

The anti-pattern we see most: teams buy a platform first, believing it is the framework, then never build the dataset. That's buying a gym membership and calling it exercise.

A minimum viable eval, in one afternoon

1. Collect 50 real cases from the workflow — logs, tickets, documents. Include the ugly 20%: edge cases, hostile inputs, ambiguous ones.
2. Write the failure definition in one sentence per case type ("a good answer cites the contract clause and never invents a number").
3. Implement 3 scorers: one deterministic (schema/must-contain), one task metric, one judge rubric with 10 human-labeled calibration examples.
4. Script the harness: run all 50 through the real pipeline, emit a scores table, commit it.
5. Gate it: CI fails if the aggregate drops. Congratulations — you now measure more rigorously than most of the enterprise AI market.

Evaluating agents is the same discipline, harder

Agents add trajectory to the problem: not just was the final answer right, but did it call the right tools with the right arguments, recover from errors, and finish in a sane number of steps. Score both levels — outcome metrics for the business, trajectory metrics for debugging. Every step of an agent's run should be logged as a first-class event you can replay; if your agent framework can't produce that trace, you'll be evaluating a black box. (This is why we build agent runtimes event-sourced — it's the only way the audit trail exists.)

The bottom line

The framework question is never really about tools. It's about whether "better" is defined, executable, and enforced in your shipping process. Four layers — dataset, scorers, harness, gates. Build the first, glue the middle, buy the last only when the team needs it.