RAG and retrieval evaluation

A RAG system answers questions in two steps. The retrieval step fetches the documents that fit a question from your store, such as the product documentation. The answer step gives question and documents to a model, which writes the answer from them. The user sees only the answer. If it is wrong, the cause can sit in either step. And the repair looks completely different in each case.

The support assistant from the worked example is such a system. For ticket T-48213 the retrieval step fetched the section pricing#plans. The answer step turned it into 39 € for a customer on the Pro plan. The right figure was 45 €. From the outside you cannot see what happened. Did the retrieval step deliver the wrong section? Or did the answer step misread the right one? In the first case nobody needs to touch the prompt. In the second case nobody needs to touch the search index.

So the first rule of RAG evaluation is: measure the two steps separately, then the whole. This chapter turns that into three measurements, each with the file you need for it and what you see afterwards.

Measurement 1: The retrieval step

What you do. The question to the retrieval step is not "was the answer good?". It is: "were the documents the answer needs among the ones fetched?" For that you need, for every question, a list of the documents that really answer it. Those are the labels of this measurement. Then you run the retrieval step alone over the questions and compare what it fetches with the list.

The file. One line per question in evals/relevance.jsonl. relevant are the sections a person marked as needed:

{"query_id":"q-001","query":"Was kostet ein zusätzlicher Platz im Monat?","relevant":["pricing#plans","pricing#seats"]}
{"query_id":"q-002","query":"Gilt der Nachtzuschlag ab 21:45?","relevant":["rules#night-surcharge"]}
{"query_id":"q-003","query":"Wie exportiere ich den Dienstplan nach Excel?","relevant":["export#excel","export#formats","export#permissions"]}

Where these labels come from: take answers your team considers good. For every statement in them, find the section it came from. Those sections are relevant. Fifty such lines are an incomplete but real label set. It is the most valuable labelling work a RAG team can do.

The arithmetic. The number is called Recall@k: how many of the needed sections are among the first k fetched? Question q-003 needs three sections. The retrieval step delivers five, among them two of the three needed. Recall@5 is 2 of 3, so 0.67. The answer step never got to see the third section. No prompt and no better model can bring it back, because the information is not in the room. That is why this number comes before every other.

# evals/recall.py
import json
import pathlib
import sys

from app.search import retrieve          # the product's retrieval step, no model call

k = int(sys.argv[1])                     # e.g. 5
rows = [json.loads(line) for line in pathlib.Path("evals/relevance.jsonl").read_text().splitlines()]

total = 0.0
for row in rows:
    fetched = [doc["ref"] for doc in retrieve(row["query"])[:k]]
    hits = sum(1 for ref in row["relevant"] if ref in fetched)
    total += hits / len(row["relevant"])
    if hits < len(row["relevant"]):
        print(f"{row['query_id']}  {hits}/{len(row['relevant'])}  missing: {set(row['relevant']) - set(fetched)}")

print(f"\nRecall@{k} = {total / len(rows):.2f}  over {len(rows)} questions")

What you see. One line per question where something is missing, and the mean:

q-003  2/3  missing: {'export#permissions'}
q-017  0/1  missing: {'billing#invoices-pdf'}

Recall@5 = 0.84  over 50 questions

The lines with missing are the work list for the retrieval step: sections that exist but are not found for their question.

Two more numbers from the same file, both decades old and well understood. Precision@k asks the reverse: how many of the five delivered sections were relevant? That matters, because irrelevant sections do not only cost tokens. They make the answer worse. MRR and nDCG also account for position: did the relevant section come first or fifth? That matters because a model reads a long context unevenly.

Measurement 2: The answer step, with retrieval held fixed

What you do. Now the other step. This is the measurement teams skip most often. "Held fixed" means: the retrieval step does not run at all. You give the answer step a set of documents you chose yourself and know to be the right ones. Then you grade the answer. If it is wrong, the retrieval step cannot be to blame, because it did not run. You are measuring the answer step alone.

The file. That is exactly the case from An eval in five files. There retrieved sits ready in the case file, and run.py hands those sections to the model without searching:

{
  "id": "case-001",
  "ticket": "Was kostet ein zusätzlicher Platz im Monat?",
  "customer": { "plan": "pro" },
  "retrieved": [
    { "ref": "pricing#plans", "text": "Starter: 39 € per seat and month. Pro: 45 € per seat and month." },
    { "ref": "pricing#seats", "text": "Additional seats are billed pro rata from the day they are activated." }
  ],
  "checks": ["citations_resolve", "no_unsourced_number", "mentions_expected"],
  "expect": { "mentions": ["\\b45\\s?€"] }
}

The answer step gets the right two sections here. If it still says 39 €, the failure is in it. Grading is done with the checks from the five files and the judge from the worked example.

What you see. The pass rate of the answer step under perfect retrieval. It is the ceiling for the whole system: the whole cannot get better than this number, however good retrieval becomes.

Why this measurement can be defended when somebody wants to cut it: your overall score drops three points. In the same week one person changed the prompt and another swapped the embedding model. With measurements 1 and 2 you know which of the two changes it was. Without them you guess, and two teams argue.

Measurement 3: The whole

What you do. Only now does the real pipeline run, retrieval step and answer step one after the other, over the real eval set. What is graded is whether the task got done. The cases look like the ones in the worked example: retrieved is not in the file; the runner runs retrieval, against the pinned documentation.

What you see. The number you report. The two measurements before explain this number to you when it moves: if measurement 1 drops, it was retrieval. If measurement 2 drops, it was the answer step.

In production the cheap checks that need no reference answer come on top: every citation points at a section that was actually fetched, every number in the answer appears in the fetched text. Online evaluation shows how these checks run on real traffic.

The trap if you skip measurement 1

Most teams measure the answer step with one popular number: Faithfulness. A judge splits the answer into individual statements. It checks each statement against the fetched sections: is that in there? The score is the number of supported statements divided by all statements.

Read that definition again and notice what is missing. Faithfulness never asks whether the fetched sections were the right ones. It only measures whether the answer step stuck to what it was given.

An example. A system has a faithfulness of 0.91, an excellent value. At the same time users report that about every sixth answer leaves out something important. Both observations are true. The answer step behaves impeccably: every statement is supported, nothing is invented. But the retrieval step, on questions that need two sections, misses the second one. Recall sits at about 0.62. The answer step faithfully summarises half the answer.

Faithfulness cannot see this failure by construction. That is exactly what measurement 1 protects against.

(This pattern is widely described. The specific numbers above are illustrative and not from a published case study.)

Common mistake: Reporting faithfulness as the RAG quality metric. It covers only the answer step. On its own it rewards a system that retrieves badly and hedges carefully. Recall is the metric that catches the failure users actually notice. And recall needs labelled sections. That is exactly why teams skip it.

Whose fault was it?

If the whole system gives a wrong answer, measurements 1 and 2 tell you which step to suspect. That also works per answer. Split the answer into its statements. Check each. For every wrong statement ask: was the section that would have made it right fetched at all?

Take an answer:

"Your Plus membership includes free returns within 30 days, and refunds are processed in 3 business days."

That is three statements, not one answer. Check each against the sections the system actually fetched:

The third column is the whole trick. The two lower statements are both wrong. A single quality score would lump them together. But they are different failures with different owners. Working on the prompt does not help the third row. Working on retrieval does not help the second row.

The question "does this section really support this statement?" has a technical name: entailment check. It appears in the next finding.

Finding: RAGChecker (NeurIPS 2024) decomposes answers into claims and uses entailment checking to attribute each error to the retriever or the generator. Its claim-level metrics were reported to correlate with human preference at up to 62% Pearson, above BLEU, ROUGE, BERTScore, TruLens, RAGAS and ARES in the same comparison. (Pearson correlation measures how closely two rankings move together, on a scale from 0, no relationship, to 1, perfect agreement. 62% against human preference is a lot for this kind of metric.)

Source: RAGChecker, NeurIPS 2024

Finding: ARES takes a different approach: synthetic training data plus lightweight judge models, calibrated with prediction-powered inference. That is a statistical procedure which corrects the estimates of cheap classifiers using a small human-labelled sample. ARES is the most principled published answer to "how do I trust a cheap judge on unlabelled data?", and it deserves more attention than it gets.

Source: ARES, 2023

Checking citations

If your product shows sources, the correctness of the citations is probably the number your users really care about. It splits into three checks. Two of them you already know from the five files:

1. Resolvable: the cited section exists and was among the fetched ones. That is citations_resolve. Pure code, no judge.
2. Supporting: the cited section contains a statement that supports the claim. That is the judge's question from the worked example. Needs a judge or a human.
3. Complete: every statement that needs support has a citation. Needs the split into statements.

Check 1 runs for free on 100% of production traffic. Many teams discover a notable rate of unresolvable citations on the first run. That is embarrassing and at the same time the cheapest possible win.

The names you will meet in tools

RAGAS supplied most of the vocabulary eval tools use today. The names are catchier than the definitions. So here is what each metric asks and which measurement in this chapter it belongs to:

By 2026 RAGAS had expanded well beyond RAG: agentic workflows, text-to-SQL and multimodal metrics such as Multimodal Faithfulness and Noise Sensitivity. The reference-free philosophy travelled with it. Reference-free means: the metrics need no labelled sections.

That is convenient and worth a warning. The two metrics without labels are exactly the two that cannot see the retrieval step. A stack that is reference-free throughout has optimised for easy setup over diagnostic power. The trap above shows what that trade-off looks like in production.