Testing RAG Systems: When "Correct-Sounding" Isn't Correct

The support engineer read the AI-generated answer twice. It was clear, it was well-structured, and it cited a return policy that did not exist. The customer had asked about returning a product after 45 days. The RAG system confidently explained the 30-day return window and an exception process for premium subscribers — a process the company had discontinued eight months before the system was built.

The retrieval had pulled the wrong document. The generation had wrapped it in fluent, confident prose. The output looked exactly like a correct answer, and it was entirely wrong.

This is the failure mode that makes RAG systems hard to test with traditional QA tools. A plain LLM hallucinates with a certain frequency that you can measure and manage. A RAG system can fail in a different way: it retrieves plausibly related but factually incorrect content and presents it as grounded truth. The answer sounds correct because it sounds authoritative. The wrongness is in the retrieved context, not in the generation, and a standard functional test that checks whether the endpoint returns a response will not catch it.

Testing a RAG system well requires splitting the pipeline in two and evaluating each half with the right metrics. For retrieval: are you pulling the right documents? For generation: is the answer faithful to what was retrieved? RAG evaluation in 2026 has a mature metric set for both halves. This post walks through each one and explains how to build a testing pipeline that scores groundedness instead of guessing at it.

Why RAG Fails Differently

Understanding the failure modes before reaching for the metrics saves you from measuring the wrong thing.

A RAG pipeline has two distinct stages. The retrieval stage takes the user’s query, embeds it, and searches a vector store or document index for the chunks most semantically similar to the query. The generation stage takes the retrieved chunks plus the original query, wraps them in a prompt, and asks the language model to produce an answer grounded in the retrieved content.

Each stage has its own failure modes, and they compound when they stack.

Retrieval can fail by missing the relevant document entirely. The query embedding does not match the document embedding closely enough, the chunk size is too large and the relevant sentence is diluted, or the relevant document was never indexed. If the right document is not retrieved, no amount of generation quality can produce a correct answer. You are asking the model to write from evidence that was not provided.

Retrieval can also fail by pulling the wrong document. The system retrieves something topically adjacent but factually different: the old return policy instead of the current one, a regional pricing document instead of the global one, a draft that was never published alongside the final version. The model receives context and generates an answer faithfully grounded in that context. The context was wrong.

Generation can fail independently of retrieval. Even with correct documents retrieved, the model may ignore parts of the context, synthesize across chunks in ways that introduce errors, or add information from its training data that was not in the retrieved content. This last failure, adding facts not present in the retrieved chunks, is what “faithfulness” measures.

The failure modes do not have a single signal. A confident, well-written answer that looks grounded is not evidence that the answer is correct. The metrics exist to surface what fluent prose hides.

Context Precision: Are the Retrieved Chunks Relevant?

Context precision measures the proportion of retrieved chunks that are actually relevant to the question. If your retriever pulls five chunks for a query and three of them are about adjacent topics rather than the specific question asked, your context precision is 0.6.

Low context precision is a problem because irrelevant chunks dilute the context window. The model receives a mix of relevant and irrelevant information and must figure out which parts to use. Some models are better at this than others, but no model consistently ignores irrelevant context perfectly. Irrelevant chunks also push the cost up: you are paying for tokens that do not contribute to answer quality.

Measuring context precision requires knowing which chunks are relevant to each question in your eval set. For a human-labeled eval set, a subject-matter expert labels each retrieved chunk as relevant or not for each query. For automated measurement, you use an LLM judge to rate chunk relevance: “Given this question and this retrieved chunk, is the chunk relevant to answering the question? Respond RELEVANT or NOT_RELEVANT with one sentence of reasoning.”

Run this rating across your full eval set and average the relevant proportion. A precision score below 70% typically indicates a retrieval configuration problem: chunk size may be too large, the embedding model may not suit your domain, or the similarity threshold is too permissive. Adjust one variable at a time and re-measure to isolate the cause.

Context Recall: Are the Right Chunks Being Retrieved?

Context recall measures whether the retrieval step is pulling all the chunks needed to answer a question correctly. A question may require information from three distinct passages, and if the retriever returns only one of them, recall is low even if that one chunk is highly relevant.

Low recall produces incomplete answers. The model generates based on whatever it retrieved, and if part of the answer was not in the retrieved context, that part either comes from the model’s training data (unreliable) or is omitted (incomplete). For factual, domain-specific queries where completeness matters, low recall is as damaging as wrong retrieval.

Measuring recall requires a ground-truth answer and a set of evidence passages that the ground-truth answer depends on. For each question in the eval set, you identify the passages in your corpus that a correct answer must reference. Recall is the proportion of those necessary passages that your retriever actually returns.

Building this ground-truth annotation is labor-intensive, which is why most teams measure context recall on a sampled subset of the eval set rather than the full set. A sample of 50 to 100 questions with annotated evidence passages gives you a reliable recall estimate without requiring full annotation of every query in the eval set.

Context precision and recall together give you a complete picture of retrieval quality: precision tells you whether what you retrieved is relevant, recall tells you whether you retrieved everything necessary. High precision with low recall means the retriever is accurate but incomplete. High recall with low precision means it is broad but unfocused. You want both above 80% for a production RAG system.

Faithfulness: Does the Answer Stay in the Retrieved Context?

Faithfulness is the generation-side metric that matters most. It measures whether every claim in the generated answer can be traced back to something in the retrieved context, with no additions from the model’s training data.

A faithful answer does not mean a correct answer. If your retriever pulls the wrong document and the model generates an answer faithful to that wrong document, the answer is unfaithful to the truth but technically faithful to the context it was given. Faithfulness is a retrieval-independent measure: it tells you whether the model is inventing or amplifying, not whether the retrieved content was right.

Measuring faithfulness: decompose the generated answer into individual claims, then check each claim against the retrieved chunks. A claim is faithful if it is directly supported by at least one retrieved chunk. A claim is unfaithful if it introduces a fact not present in any chunk, no matter how plausible that fact might be.

The automated version of this uses an LLM judge with a structured prompt: “Given the following retrieved context and the following answer, list every claim in the answer. For each claim, state whether it is fully supported by the context or introduces information not present in the context.” Aggregate the proportion of supported claims across your eval set.

A faithfulness score below 85% is a significant concern in production. It means the model is regularly adding to or contradicting its retrieved context. Common causes: the retrieved chunks do not contain enough information and the model is filling gaps from training data, the system prompt does not strongly enough instruct the model to ground its answer in the context, or the model is being asked questions that require synthesizing across many chunks in ways that introduce errors.

Answer Relevancy and Correctness

Faithfulness tells you whether the model stayed within the retrieved context. Answer relevancy tells you whether the answer actually addresses what the user asked. A model can generate a faithful, grounded response that is about a related but different topic, and relevancy catches that.

Answer relevancy is measured by asking a judge model: “Given this question and this answer, does the answer directly address the question that was asked? Rate on a scale of 1 to 5.” You can also measure it more mechanically by embedding both the question and the answer and computing their cosine similarity — a proxy for semantic alignment.

Correctness sits above relevancy and faithfulness on the difficulty scale. It asks: is the answer actually right? To measure correctness, you need a ground-truth answer for each question in the eval set. The judge compares the generated answer to the ground truth and scores the factual alignment. Correctness requires the most human investment to measure because ground-truth answers must be human-authored or human-validated. It is the metric with the most direct product impact and the most labor cost to track.

In practice, teams typically combine metrics: measure context precision and recall for retrieval quality, measure faithfulness for generation grounding, measure answer relevancy for utility, and measure correctness on a sampled subset where ground-truth answers exist. The combination gives you a quality profile rather than a single number, and each dimension has a distinct remediation path when it is low.

Building an Eval Pipeline for RAG

A RAG eval pipeline runs automatically on every significant change: chunk size adjustment, embedding model swap, retrieval threshold change, prompt update, or model version bump. Here is the structure that works at scale.

Start with an eval dataset of 100 to 200 question-answer pairs covering the main categories of queries your system handles. For each pair, annotate the ground-truth answer and, where feasible, the expected evidence passages. This is the investment that pays back across every future eval run.

For each eval run, retrieve context for every question using the current retrieval configuration, then generate answers using the current prompt and model. Score the run across all four metrics using an LLM judge with majority voting (three judge calls per metric per example, take the majority verdict to reduce judge variance).

Store results with a timestamp, retrieval configuration hash, and model version, so you can compare across runs. A results table that shows context precision, context recall, faithfulness, and answer relevancy for each run, alongside the configuration that produced them, gives you the audit trail to understand whether a change helped or hurt and which dimension changed.

Set per-metric thresholds based on your quality baseline and treat a metric drop below threshold as a blocking failure. A 5% drop in faithfulness is a regression worth investigating before deploying. A 3% drop in context precision alongside a 4% gain in recall might be acceptable if the net quality trend is neutral, but it should be a deliberate decision rather than an unnoticed drift.

The Eval Framework Landscape

Several frameworks handle RAG evaluation at different parts of the pipeline, and knowing which does what saves you from building what already exists.

Ragas specializes in offline RAG evaluation and implements context precision, context recall, faithfulness, and answer relevancy as first-class metrics. You provide the question, the retrieved contexts, the generated answer, and optionally the ground truth, and Ragas scores all metrics in one pass. It is the standard choice for building your eval dataset and running quality assessments outside the live pipeline.

DeepEval integrates RAG metrics into a CI-compatible assertion library. You write eval assertions in Python, specify thresholds, and run them as part of your test suite. A merge that drops faithfulness below 0.85 fails the CI job the same way a failing unit test would. DeepEval is where Ragas-style metrics become build gates rather than reports.

TruLens focuses on production monitoring. It instruments your RAG pipeline, records every retrieval and generation call with the associated metrics, and surfaces trends over time in a dashboard. Where Ragas and DeepEval catch regressions in controlled eval runs, TruLens catches them in live production traffic. The combination — offline eval for development, CI gate for deploys, production monitoring for drift — covers the full lifecycle.

None of the three frameworks gives you the ground-truth annotation that makes correctness measurement possible. That investment stays with your team, and it is the one that separates RAG systems where quality is tracked from RAG systems where quality is assumed.

Scoring Groundedness, Not Vibes

The return policy hallucination that opened this post is not unusual. It is the natural output of a system where the retrieval step is not tested, the generation step is not constrained to its retrieved context, and quality is evaluated by reading a few answers and deciding they sound right.

Grounding-based evaluation replaces that judgment with measurement. Context precision and recall tell you whether the retrieval step is finding the right evidence. Faithfulness tells you whether the generation step is staying within it. Answer relevancy tells you whether the model addressed the actual question. Correctness, measured on the sampled subset where you have ground truth, tells you whether the answers are right.

Each metric has a remediation path when it is low. Low retrieval precision: tighten the similarity threshold or reduce chunk size. Low recall: increase retrieval count or improve the embedding model’s domain alignment. Low faithfulness: strengthen the grounding instruction in the system prompt or add a post-generation faithfulness filter. Low relevancy: review whether the query preprocessing is distorting the user’s intent before it reaches the retriever.

A RAG system that scores well across all four dimensions does not produce correct answers by luck. It produces them by design, with a pipeline configured to retrieve the right evidence and a model instructed to stay within it. That is what separates a RAG system you can trust from one that reads correctly until it does not.