Eval-Driven Development: LLM Quality Gates in Your CI Pipeline

The model upgrade looked clean in testing. The team ran the feature manually, checked a few outputs, agreed they looked good, and merged. Two days after deploy, the support team started getting tickets about the AI assistant giving answers that were technically accurate but oddly cold and transactional, a shift in tone that users found off-putting. The root cause: the new model version had different defaults around formality, and none of the team’s checks had measured tone at all.

Eval-driven development is the practice of treating language model quality as something you measure continuously and gate on, the same way you gate on unit tests and linting. The goal is to make the model upgrade scenario above impossible, not by catching every possible quality change, but by having defined what quality means before you shipped, measuring it on every relevant change, and failing the build when it drops below the bar you set.

This post covers the architecture of an eval gate in CI, the specific gating decisions that require care, how to manage the cost of running evals in a pipeline, and the rollout strategy that keeps a new eval suite from blocking deploys before it is calibrated.

Why CI Is the Right Place for Evals

The alternative to CI evals is production monitoring with rollbacks. You deploy, watch the quality metrics, and roll back if they drop. This approach has the advantage of using real traffic and real user behavior to measure quality, and it has the disadvantage of exposing real users to quality regressions before you catch them.

For most LLM features, a quality regression that reaches production is not a simple rollback scenario. Users who received bad outputs may have acted on them. Support tickets have been filed. If the feature produced advice, recommendations, or decisions, the downstream effects of poor-quality outputs may have already propagated before the rollback. The window between deploy and rollback is not zero-consequence.

Eval gates in CI move the detection before the deploy. You measure quality on a fixed eval set before any code or configuration change reaches production, and you block the deploy if quality falls below your threshold. The eval set is smaller than production traffic and does not have real user distribution, which means it will not catch every regression. But it will catch the regressions that your eval set covers, which is the coverage you designed deliberately rather than leaving to chance.

The two approaches are complementary rather than competing. CI eval gates catch known failure modes before deploy. Production monitoring catches distribution shift and novel failures after deploy. Both are necessary for a mature LLM quality process. CI gates are the prerequisite.

The Eval Gate Architecture

An eval gate in CI is a job that runs on a defined trigger, exercises the LLM feature against a fixed dataset, scores the outputs, and passes or fails the build based on the scores.

The trigger determines which changes invoke the gate. At minimum, run the eval gate on: any change to the model version or model configuration, any change to the system prompt, any change to the retrieval configuration for a RAG feature, and any change to the prompt templates used at inference time. These are the changes that directly affect output quality. Application code changes that do not touch the model interaction layer can be excluded from the eval gate, keeping the gate fast on most PRs.

The eval dataset is a fixed collection of inputs with associated expected properties. For exact-match tasks (classification, extraction), each input has a ground-truth label. For quality tasks (summarization, generation), each input has a human-rated reference output and a quality rubric. For behavioral tasks (safety, scope), each input has an expected behavior class (should comply, should refuse, should ask for clarification). The dataset should not change between runs on the same code version; it is the fixed measuring stick.

The scoring step runs each input through the feature and scores the output using the method appropriate for the task type: exact-match comparison for classification, LLM-as-judge with majority voting for quality dimensions, behavioral assertion for safety and scope. Each method returns a score or a pass/fail verdict per input.

The gate decision aggregates the per-input scores into pass-rate metrics for each dimension, compares them to the defined thresholds, and returns a build result. A build passes if all dimension pass rates are at or above their thresholds. A build fails if any dimension falls below its threshold, and the failure message reports which dimension failed and by how much.

Setting Thresholds That Mean Something

A threshold is only meaningful relative to a baseline. Before you set a threshold, run the eval gate against the current production version of the feature and record the pass rates for each dimension. These are your baseline scores.

Set thresholds below the baseline by a margin that accounts for natural score variance and acceptable quality trade-offs. A feature that scores 91% on factual accuracy at baseline should not have a threshold of 90% if the eval set has enough variance that the score moves by plus or minus 2% between runs on the same code. A threshold that triggers false failures on unchanged code teaches the team to ignore the gate.

A practical formula: set the threshold at the baseline minus two times the standard deviation of scores across five runs on the same code version. This gives you a threshold that a genuine regression will cross and random variance will not. If the baseline is 91% and the standard deviation is 1.5%, the threshold is 91% minus 3% = 88%. A change that drops the score to 87% fails. Natural run-to-run variance that produces 89% passes.

Different dimensions should have different thresholds. A safety dimension (does the model refuse harmful requests?) may have a zero-tolerance threshold: any safety failure blocks the deploy, regardless of pass rate. A style dimension (does the model’s tone match the brand voice?) may have a threshold of 70%, accepting some variance because tone is harder to measure precisely. Match the threshold to the cost of a regression in that dimension.

The Cost Problem and How to Manage It

Running LLM-as-judge evals against a 200-example dataset, with three judge calls per example per dimension, at five dimensions, means 3,000 judge model calls per CI run. At current pricing, this might cost five to ten dollars per run. On a team that merges ten PRs a day, that is fifty to one hundred dollars per day just for the eval gate.

This is real and worth managing, but it is not prohibitive in the context of the cost of a quality regression in production. The question is not whether to run evals but how to structure the cadence to match the cost to the value.

A tiered cadence manages this efficiently. Run fast, cheap evals on every commit to main: exact-match tests and a small behavioral test suite (25 to 50 examples, no judge model calls). These run in under a minute and cost fractions of a cent. Run the full quality eval suite (all dimensions, full dataset, majority-voting judge calls) on merges to main and on any change that touches model configuration or prompts. Run the full suite plus a broader distribution of examples on model version upgrades.

This means most PRs trigger only the fast tier. The expensive full evaluation runs only when a change actually affects the feature’s quality profile. The cost on a ten-PR-per-day team with this tiering might be five to fifteen dollars per day rather than fifty to one hundred.

Caching inputs through the feature’s retrieval layer (for RAG features) also reduces cost. If the retrieval step is deterministic for a fixed input, cache the retrieved context once and re-use it across eval runs. The eval then only calls the generation model, not the retrieval pipeline, on each run.

Rolling Out a New Eval Suite Without Blocking Deploys

A new eval suite should not block deploys until it is calibrated. The rollout process is: shadow mode first, then warn, then block.

In shadow mode, the eval gate runs and reports scores but does not affect the build outcome. Every CI run gets an eval score in the build artifacts, visible to the team but not blocking. Run in shadow mode for two to four weeks, across enough code changes to see the score distribution. This tells you whether your thresholds are set correctly, whether the gate triggers false failures on benign changes, and whether the eval set covers the failure modes you care about.

In warn mode, the gate fails the build with a warning rather than a blocking failure. The team can override the warning and proceed with the deploy, but the override requires an explicit action and is logged. Warn mode builds team trust in the gate while catching genuine regressions before the gate becomes a blocker.

In block mode, the gate blocks deploys below the threshold with no override. Block mode is appropriate once the calibration period has shown the gate is reliable: false failure rate is below 5%, every genuine regression the team caught was caught by the gate, and the threshold is set correctly relative to the quality baseline.

Do not skip the shadow mode period. The temptation to go directly to block mode when a new eval suite looks clean in development is strong, and the failure mode is a gate that blocks legitimate deploys because the eval set covers a slice of the input distribution that changes don’t affect, or because the threshold was set too tightly against a baseline that had some lucky high runs. Shadow mode reveals these problems cheaply.

Regression Diagnosis When the Gate Fails

A failing eval gate should tell you which examples caused the failure and why, not just that the pass rate dropped below threshold.

The CI job output should include: the pass rate for each dimension, the delta from the previous run’s pass rate for each dimension, the specific inputs that failed (or that changed from pass to fail since the last run), and the judge’s verdict or the expected vs. actual label for each failure.

This output lets the team diagnose a failure in minutes rather than hours. A prompt change that caused a tone regression will show up as a cluster of failures on a specific dimension, with the judge’s comments indicating the tone shift. A model version upgrade that improved accuracy but hurt conciseness will show two dimensions moving in opposite directions, making the trade-off visible before the deploy.

The diagnosis output is also useful for deciding whether a failure is a blocker or an acceptable trade-off. If a model upgrade drops the style score from 88% to 84% (below the 86% threshold) but improves the accuracy score from 91% to 95%, the team may decide the accuracy gain is worth adjusting the style threshold. That is a deliberate decision made with visible data, not a regression that slipped through.

The Loop That Keeps Quality Moving

The eval gate is a quality floor, not a ceiling. Setting a threshold and meeting it does not prevent the team from raising the bar over time.

After each major model or prompt improvement, record the new baseline scores and raise the thresholds accordingly. A team that ships a prompt improvement that brings accuracy from 91% to 95% should raise the accuracy threshold from 88% to 92%, locking in the gain so a future change cannot quietly regress it back to 91% without triggering the gate.

This threshold-raising loop, done consistently over time, produces a ratchet effect: quality can go up and is hard to bring down, because the gate blocks regression at whatever the current best-achieved level is. Teams that manage the eval gate this way see their quality baselines improve steadily over months, rather than oscillating as prompt changes and model updates move the quality up and down without accountability.

The eval set should grow alongside the quality bar. Every time the team encounters a quality failure in production that was not caught by the eval gate, that failure becomes a new example in the eval set. The gate’s coverage grows with the team’s experience of what fails in the real world. A gate that started with 100 representative examples and has grown to 300 examples across two years of production operation, with every real failure represented, is a substantially better quality insurance policy than the gate you started with.

The model upgrade scenario at the start of this post, the tone shift that no one caught, would have been caught by a gate that included tone as a measured dimension with a threshold. The calibration work to add tone measurement would have taken a day. The production fallout, the support tickets, the rollback, the user impact, cost more than that. Eval-driven development is, in the end, an argument about where you want to pay the cost of quality: before the ship or after it.