Testing the Untestable: QA Strategies for LLM-Powered Features

The product manager opened a ticket with a simple description: “The AI summary should be accurate and concise.” You read it, looked at your assertion library, and realized you had no idea what to assert.

Your test suite knows how to check that a status code is 200, that a JSON field is non-null, and that a value matches an expected string exactly. LLM features do not give you an expected string. They give you a response that varies every time the endpoint is called, that changes when the model is updated, and that can be correct in multiple different ways simultaneously. The entire mental model that makes functional testing work — call a function, compare the output to a known value — breaks down the moment the output is generated rather than computed.

This is not a niche problem. Every product team that ships an AI-powered feature runs into it, and most teams respond by either skipping tests entirely (“we’ll do human review”) or writing assertions that are so loose they do not catch real regressions (“assert the response is not empty”). Neither is a quality strategy.

A practical framework for LLM-powered features tests properties of the output rather than the output itself. It borrows techniques from evaluation-driven development, uses models to evaluate models, and treats non-determinism as a design constraint to work around rather than a reason not to test. This post walks through the framework from first principles.

What Makes LLM Features Different

Before the framework, it helps to be precise about what is actually different, because the difference shapes every testing decision you make downstream.

Traditional software features are deterministic. Given the same input, they produce the same output. That property is what makes automated testing tractable: you call the function with known inputs, you compare the output to the expected value, and a mismatch is a defect. The test is a specification in executable form.

LLM features are stochastic. The same prompt sent to the same model at different times will produce different outputs, because the model samples from a probability distribution at each token position. Temperature settings modulate how much variation there is, but at any temperature above zero, variation exists. You cannot write a test that says “the summary should be exactly this sentence” because tomorrow it will be a different sentence, and the day after that another one, and all three may be equally correct.

The output also changes when the model changes. Model providers update weights, adjust behavior, and release new versions continuously. A response that was acceptable under model version A may be subtly different under model version B, even with identical inputs. Testing against exact strings means every model update breaks every test, which is the opposite of useful.

There is a third difference that matters for test design: LLM outputs can be correct in multiple ways. A summary of a three-paragraph article can be accurate whether it is two sentences or four, whether it uses passive or active voice, whether it lists the main points in order or groups them by theme. “Correct” is not a single point in the output space; it is a region. Any test strategy that defines a single correct output and checks against it is testing the wrong thing.

The implication is that LLM testing is fundamentally about properties rather than values. You do not check what the response is. You check whether the response has the qualities that matter: is it relevant to the input, is it within the expected length range, does it contain the required information, does it avoid producing content that is harmful or off-topic? Those properties are testable even when the exact wording is not.

The Four Properties Worth Testing

Across LLM-powered features, four categories of properties cover most of what matters. Start here before building custom checks.

The first is relevance. Does the output address the input? A summary of a product return policy should contain information about return policy, not about shipping rates or account management. A code explanation should reference the actual code. This seems obvious until you see a production LLM feature that occasionally produces a response that is fluent and grammatically correct and completely unrelated to what the user asked. Testing relevance catches that class of failure.

The second is safety and content policy. Does the output stay within the boundaries you defined? This covers the obvious cases like profanity and harmful content, but for enterprise products it also covers things like not revealing internal system prompts, not making claims about competitors, and not generating content outside the product’s scope. These constraints can be tested with targeted adversarial inputs rather than general-purpose runs.

The third is format compliance. LLM outputs that get parsed downstream must conform to a schema: JSON that parses, markdown that renders, bullet lists with the right count, responses under a character limit. Format compliance is deterministic: either the output parses or it does not. This is the easiest category to assert on with traditional tests, and it is the one most often broken when prompt changes are deployed carelessly.

The fourth is information coverage. Does the output contain the key facts it should contain? A summary of a meeting transcript should mention the decision that was made. A product description should include the product name. These are testable through substring matching, keyword checks, or entity extraction, all of which are deterministic even though the surrounding prose is not.

Snapshot Testing for Semantic Stability

The closest LLM equivalent to traditional snapshot testing is semantic snapshot testing: instead of checking that the output matches a stored string, you check that the meaning of the output has not drifted significantly from a reference response.

The practical version looks like this. Run the feature with a fixed set of test inputs and store the responses. On each subsequent test run, generate new responses for the same inputs and compute a similarity score between the new and stored responses using an embedding model or an LLM judge. If the similarity drops below a threshold — meaning the outputs have diverged substantially in meaning — flag it for human review.

This approach catches the regression where a prompt change or model update causes the feature to start summarizing differently, emphasizing different points, or drifting toward a different tone. It does not catch cases where all the responses are equally good but different. That is by design: semantic snapshots track drift, not correctness.

Choosing the similarity threshold is calibration work. Start by running the same inputs through the model ten times and computing pairwise similarity scores across the natural variation. That gives you a baseline for how much the model varies even without any changes. Your drift threshold should sit below the natural variation floor; anything above it means the model changed, not just sampled differently.

Reset the snapshots explicitly when you deploy a deliberate change to the prompt or model version. Snapshots that are never reset accumulate drift silently, which defeats the purpose.

LLM-as-Judge: Using a Model to Evaluate a Model

The most powerful technique in LLM testing is using a separate model to grade the output of the model under test. This lets you evaluate properties like relevance and quality that are too nuanced for keyword matching but too subjective for deterministic assertions.

The pattern: you send the LLM’s output, plus the original input and any relevant context, to a judge model with a structured evaluation prompt. The judge returns a score or a PASS/FAIL verdict with a reason. You run this judgment in your CI pipeline the same way you run any other assertion.

A judge prompt for summarization quality might look like this:

You are a quality evaluator. Given the following article and summary, 
rate the summary on three criteria:
1. Accuracy: does it correctly represent the article content? (PASS/FAIL)
2. Completeness: does it cover the main points? (PASS/FAIL)  
3. Conciseness: is it appropriately brief without omitting key information? (PASS/FAIL)

Article: {article}
Summary: {summary}

Return only valid JSON: {"accuracy": "PASS"|"FAIL", "completeness": "PASS"|"FAIL", 
"conciseness": "PASS"|"FAIL", "verdict": "PASS"|"FAIL"}

The judge model should be separate from the model under test. The same model that produced a flawed summary may grade that summary as acceptable because it carries the same biases in both directions. Use a different model family, or at minimum a different model tier from the same provider.

LLM-as-judge introduces its own non-determinism: the judge may score the same output differently on different runs. Address this with majority voting: run the judge three times per output and accept the majority verdict. For critical quality gates, run five judges from different model families and require at least four to agree.

One calibration exercise worth doing before relying on LLM-as-judge in CI: collect 50 to 100 outputs that a human has already rated, run the judge over them, and measure agreement. If the judge agrees with human raters more than 80% of the time, it is useful in CI. Below that, it adds noise rather than signal.

Boundary Testing: Inputs That Expose Failure Modes

LLM features fail in predictable ways, and many of those failures are easier to surface with targeted adversarial inputs than with general-purpose runs.

The empty or minimal input case. Send the feature a one-word input, an empty string, or a single punctuation character. Models that are not properly guarded will either refuse gracefully, hallucinate content, or produce an error. All three outcomes are testable: you check the response type and whether the feature handles the edge case according to your product’s defined behavior.

The off-topic input case. Send the feature an input that is clearly outside its intended domain: a coding assistant sent a recipe, a summarization feature sent an empty document, a customer service bot asked about stock prices. These inputs reveal whether the model stays in scope or drifts into territory you did not design for.

The adversarial input case. Prompt injection, attempts to elicit the system prompt, requests to ignore previous instructions, jailbreak patterns that were published in security research. These should be part of your test suite for any feature that accepts user-supplied text. The expected behavior is a refusal or a graceful deflection, and you can assert on that.

The long-input case. Send inputs significantly longer than your expected maximum and verify the feature degrades gracefully: truncating intelligently, summarizing the portion it can handle, or returning a clear error. Features that accept long inputs without bounds can produce incoherent outputs or hit context limits mid-generation.

These boundary cases have something in common: the expected behavior can often be specified as a property (refuses, stays in scope, handles gracefully) even when the exact output cannot. That makes them testable with the same framework as the general case.

Regression Strategy for Non-Deterministic Output

When a model update or prompt change ships, you need a way to tell whether quality improved, degraded, or stayed the same. Traditional regression testing compares outputs to stored baselines and fails on any difference. That approach collapses immediately for LLM features.

A regression strategy for non-deterministic output works in three steps. First, build an evaluation set: a fixed collection of inputs that represents the range of real-world usage, with a human-rated quality label for each stored response. This set is your ground truth, and it should be built before you deploy any changes, not after you discover a regression.

Second, define a quality gate: the minimum pass rate across the evaluation set for the LLM-as-judge score. If the feature currently passes the judge on 94% of eval inputs, your quality gate might be 90%, giving you a four-point buffer for natural variation while still catching meaningful regressions.

Third, run the evaluation set on every significant change: prompt update, model version bump, context window change, system instruction edit. A run that drops below the quality gate blocks the deploy and triggers a human review of the specific inputs where quality degraded.

This approach means you are comparing quality distributions rather than individual strings. A change that causes 10% of outputs to get worse is a regression even if no two outputs are identical between runs. A change that causes 98% of outputs to score as high or higher is an improvement even if every output is worded differently. The distribution is the thing to track.

Putting It Together in CI

A working CI pipeline for an LLM-powered feature runs in three layers, each one catching a different category of failure.

The first layer runs on every commit and covers everything deterministic: does the endpoint return a valid response, does the output parse according to the expected schema, does it fall within the character limits, does it pass the content policy keyword check? These are fast, cheap, and exact. Any failure here is a clear defect with a clear owner.

The second layer runs on merges to the main branch and covers property-based checks: LLM-as-judge scoring for relevance and quality across the evaluation set, semantic drift checks against stored snapshots, boundary case coverage. These are slower and cost model API calls, so they run less frequently than the first layer.

The third layer runs on a nightly schedule and covers the full evaluation set with multiple judge runs and majority voting to reduce non-determinism in the scores themselves. This layer produces the quality distribution report that tells you whether the feature is holding, improving, or degrading over time without any obvious code change.

The separation of concerns is what makes this tractable. Fast deterministic checks run often and fail loudly. Probabilistic quality checks run on a longer cadence and surface trends rather than individual failures. Human review happens when a trend warrants it, not for every CI run.

The Framework in One Sentence

Test properties of the output, not the output itself. Check that the response is relevant, safe, correctly formatted, and informationally complete, using a combination of deterministic assertions, LLM-as-judge scoring, semantic snapshots, and adversarial boundary cases. Run the deterministic layer on every commit and the probabilistic layer on every merge. Track the quality distribution over time, not individual response strings.

The assertion library that worked for REST APIs still works for the parts of LLM features that are deterministic. For the parts that are not, the framework above is how you bring the same rigor to a fundamentally different problem. The goal is not certainty. It is a quality signal you can trust enough to deploy with confidence, catch regressions before production does, and know when a model update made things better or worse.

That is achievable for LLM features. It just requires a different set of tools than the ones you used for your last JSON endpoint.