Evals Are the New Test Cases: How to QA an AI Model

The QA engineer opened the test file, stared at the assertion, and realized it was checking the wrong thing entirely.

The assertion read: assert response.status_code == 200. The feature was an AI-powered email summarizer. Status 200 meant the endpoint responded. It said nothing about whether the summary was accurate, whether it omitted the most important point in the email, or whether it hallucinated a deadline that was not in the original message. Every test in the suite passed. The feature shipped with a subtle but consistent quality problem that support tickets started surfacing two weeks later.

This is the failure mode that traditional QA misses when the system under test includes a language model. Functional tests check that the infrastructure works. They check status codes, response times, and schema compliance. They do not check whether the output is good, because “good” for a language model output is a judgment that cannot be encoded in a standard assertion.

Evaluation-driven development (eval-driven development, or EDD) is the discipline that fills that gap. It borrows the structure of test-driven development: define quality criteria before you build, run them automatically, and use failures to guide improvement. The difference is in what you test and how you measure it. This post walks through what evals are, the four types that matter, how to build an eval set that actually catches quality regressions, and how to run evals in CI so they work the same way your test suite does.

What Evals Are and What They Are Not

The word “eval” gets used loosely, so a precise definition first.

An eval is a measurement of how well a model or model-powered feature performs on a defined task across a representative set of inputs. It produces a score or a pass rate, not a binary pass/fail verdict for a single request. You run an eval across an eval set, which is a collection of inputs that represents the range of real-world usage, and you compare the result to a threshold or to a previous result to detect regressions.

Evals are not integration tests. An integration test checks that the system components connect correctly and that data flows between them as designed. An eval checks whether the model’s output quality is acceptable. You need both, and they serve different purposes at different stages of the CI pipeline.

Evals are not A/B tests. An A/B test measures the effect of a change on user behavior over time, in production, with real users. An eval measures quality offline, on a fixed dataset, without any user exposure. Evals are faster and cheaper than A/B tests, which makes them useful as a gate before any production exposure rather than as an alternative to it.

Evals are not benchmarks in the academic sense. Industry benchmarks like MMLU or HumanEval measure model capability on standardized tasks. Your evals measure your specific feature’s quality on your specific use cases. They are product-specific, and they should be: a model that scores well on a coding benchmark but poorly on your customer support summarization task is the wrong model for your product.

The critical property of a useful eval set is that it is representative. If your email summarizer handles three categories of emails, sales outreach, support requests, and internal announcements, your eval set should include examples from all three categories, weighted roughly by their frequency in production. An eval set built entirely from easy examples will show great results right up until a real user sends a hard one.

The Four Types of Evals

Four categories of evals cover most of what matters for a production AI feature. Build at least one from each category before you ship.

The first is exact-match evals. These apply to the subset of model outputs that have a single correct answer: classification labels, structured data extraction, named entity recognition, code that either compiles or does not. For these tasks, you can collect input-output pairs with ground-truth labels and measure the model’s accuracy directly. An email that asks for a refund should be classified as category: refund_request. A code snippet that should extract the date from a string should return datetime(2026, 3, 9). Exact-match evals are the closest LLM equivalent to traditional unit tests: fast, cheap, and unambiguous when they fail.

The second is model-graded evals. For outputs where multiple correct answers exist, you use a separate model as a judge. You send the input, the model’s output, and a rubric to a judge model, and it returns a score or a verdict. A summary eval might ask the judge: “Given this email and this summary, rate the summary on accuracy (does it represent the email correctly?), completeness (does it cover the main point?), and conciseness (is it appropriately brief?).” The judge model returns structured scores you can aggregate across the eval set.

The third is human-graded evals. Some quality dimensions are too nuanced for a model judge to score reliably, or the stakes are high enough that you want human calibration rather than model calibration. Human-graded evals use a structured rubric and a small set of human raters to score a sample of outputs. They are slower and more expensive, so you run them less frequently, but they are the calibration anchor that tells you whether your model-graded evals are measuring the right things.

The fourth is behavioral evals. These test properties of the model’s behavior rather than the quality of individual outputs: does it refuse harmful requests, does it stay within the system prompt’s scope, does it behave consistently under paraphrase attacks (same question asked different ways), does it handle long inputs without degrading? Behavioral evals often look like adversarial test cases, and they are: you are testing the failure modes, not the happy path.

Building an Eval Set

The eval set is the most important artifact in eval-driven development, and it is the one most teams under-invest in. A model’s measured quality is only as good as the eval set you measure it against.

Start by collecting real inputs from production if the feature is already live, or from user research if you are building something new. Real inputs have distributional validity that synthetic inputs do not: they represent the actual variety in how users phrase requests, the edge cases that show up in the wild, and the failure modes that emerge from real behavior rather than constructed scenarios.

Aim for at least 100 inputs in the initial eval set, with a minimum of 10 from each distinct category or user persona you expect the feature to serve. 100 is not a magic number; it is the minimum that gives you stable aggregate statistics. With 10 inputs, a single failure changes your pass rate by 10 percentage points. With 100, a single failure changes it by 1, which is a more reliable signal.

Label each input. For exact-match evals, the label is the ground-truth output. For quality evals, the label is a human rating of the reference output the model should aspire to. For behavioral evals, the label is the expected behavior class (should refuse, should comply, should escalate). The labeling work is the most time-consuming part of building an eval set, and it is also the most valuable: it forces you to define quality concretely rather than gesturing at it.

Keep the eval set stable across model versions and prompt changes. The stability is what makes results comparable. If you add 20 new examples between run A and run B, you cannot compare the pass rates because they are measuring different things. Extend the eval set in batches, snapshot the extended set, and treat it as a new baseline.

One anti-pattern to avoid: building the eval set after you know how the model performs. Selection bias is subtle and destructive. If you collect the inputs that the model currently handles well, your eval will show high quality regardless of how the model actually performs on harder cases. Collect the eval set before running the model, or collect it from production where the model has no influence on what inputs arrive.

Running Evals in CI

An eval that only runs when someone remembers to run it is not a quality gate. It is a dashboard that shows optimistic numbers until it does not. Running evals in CI is what makes them a real part of your quality process.

The CI integration looks like a test job that runs after the functional test suite. It takes the current model configuration (prompt version, model version, temperature settings) as inputs, runs the eval set through the feature endpoint, collects the model-graded and exact-match scores, and compares them to the thresholds you defined.

A passing eval run does not mean the model is perfect. It means the model’s quality on your eval set is above the minimum bar you defined as acceptable. Set thresholds based on your quality baseline: if the current model passes 91% of exact-match evals and 87% of model-graded evals, set your gate at 88% and 84%, giving yourself a margin for natural variance while still catching meaningful regressions.

The CI job should fail loudly on two conditions: a drop in pass rate below the threshold, and a change in the error distribution (different inputs failing than last time, which can indicate a regression even if the total pass rate is stable). Both signals are worth surfacing to the team before a deploy.

One practical consideration: model-graded evals cost money to run at scale. You are calling a judge model for every example in the eval set on every CI run. At 100 examples and a few cents per model call, the cost is small. At 1,000 examples and three judge calls per example for majority voting, it adds up. Manage this by running exact-match evals on every commit (fast and free), model-graded evals on merges to main (moderate cost), and full human-calibrated eval runs on model or prompt version changes (expensive but infrequent).

Regression Strategy for Model Updates

Model providers update weights, release new versions, and occasionally deprecate old ones. Each update is a potential regression, and without evals you have no systematic way to detect it until a user reports a problem.

The regression strategy is straightforward: run the full eval set before and after any model version change and compare the results dimension by dimension, not just in aggregate. An update that improves accuracy by 3 percentage points but reduces safety behavior compliance by 8 points is not an upgrade. The aggregate might look neutral or positive while a critical dimension got worse.

When a new model version is available, run a shadow eval: send the same inputs to both the current model and the new version in parallel, score both, and produce a side-by-side comparison report. This tells you not just whether the new version is better in aggregate but which inputs got better, which got worse, and whether the distribution of failures shifted in a way that matters for your use case.

One thing that catches teams off guard: prompt changes can cause regressions as severe as model changes, and they happen more frequently. Every prompt change should trigger an eval run. A phrasing change that seems minor can shift the model’s behavior on a subset of inputs in ways that are not visible from reading the prompt. Evals catch what intuition misses.

The Eval-Driven Development Loop

Eval-driven development, like test-driven development, works as a loop: define quality criteria, measure against them, iterate until they pass, and maintain them as the system evolves.

Before you build a feature: write the eval set. Define the input examples, write the quality rubric, collect human ratings for the reference outputs. This step forces you to define what “good” means before you are tempted to define it as “whatever the current model produces.”

During development: run evals frequently as you iterate on the prompt, the retrieval strategy, or the model configuration. Evals give you a signal about whether a change helped or hurt across the full input distribution, which is more reliable than eyeballing a few examples.

Before you deploy: run the full eval suite, including the behavioral evals for safety and scope. Set the quality gate. Do not deploy if it does not pass. Treat a failing eval the same way you treat a failing integration test: fix the root cause before the deploy continues.

After you deploy: collect production samples and add them to the eval set periodically. Production inputs will always be more diverse than your initial collection, and the eval set should grow to represent that diversity. A quarterly review of the eval set to add new examples and remove outdated ones keeps the measurement aligned with what the feature actually faces.

Quality for Systems That Do Not Have a Right Answer

The discomfort that makes QA engineers hesitant about LLM features is real. You spent your career in a domain where correctness is binary: a test either passes or fails, a requirement is either met or not, a bug either exists or it does not. LLM features put you in a domain where correctness is a distribution, where the same input produces multiple valid outputs, and where “good” is a rubric rather than a specification.

Evals do not make that discomfort disappear. They make it manageable by replacing “I think this is probably good” with “83% of the time this meets our defined quality criteria, and here are the 17% of cases where it falls short.” That is not certainty, but it is accountability, and accountability is what quality engineering has always been about.

The assertion library that works for REST APIs still works for the deterministic parts of AI features. For everything else, evals give you the same rigor it took a decade to develop for functional testing: a defined standard, a reproducible measurement, and a gate that fails loudly when quality drops. The standard is harder to write, the measurement is probabilistic, and the gate allows for variance rather than demanding exact match. But the discipline is the same.

Build the eval set before you ship. Run it in CI. Treat a failing eval as a failing test. Your AI features will ship with fewer quality surprises, and you will have the evidence to explain what “working correctly” means for a system that does not return the same answer twice.