Evaluate a RAG pipeline

A RAG system looks like a single call: a question goes in, an answer comes out. Inside, it is a short pipeline. The model decides whether to look something up (routing), a retriever pulls in context (retrieval), and the model writes an answer from that context (generation). A wrong answer can come from any of the three, and the fix is different in each case. If you score only the final answer, you cannot tell them apart, so every failure turns into a guessing game.

The pipeline, and how each step fails

question
  → route      does this even need a lookup?
  → retrieve   pull the context that should answer it
  → generate   write the answer from that context
answer

Each step has its own failure mode:

• Routing looks things up when it should not (slow, noisy) or skips a lookup it needed (answers from memory, gets it wrong).
• Retrieval returns the wrong documents, or the right topic but not the specific passage that answers the question.
• Generation ignores good context and answers from the model’s own memory, or invents claims the context does not support (hallucination).

Here is why measuring only the final answer is not enough. The answer is wrong. Did the retriever miss the document, or did the model ignore a document that was right there? Those are opposite problems with opposite fixes, and the final answer looks identical either way. Score each step and the failure points at its own cause.

What to evaluate at each step

Make each step its own unit so you can grade it. The natural way is to trace the pipeline, so routing, retrieval, and generation are each a separate span, then attach an evaluator to each one. Then, per step:

Common evaluator names for these are routing accuracy and unnecessary retrieval (routing), context relevance and context completeness (retrieval), and groundedness and context utilization (generation). Each is usually an LLM judge scoring that span’s input and output against a short rubric.

Build the datasets

Two levels, for two jobs:

• End-to-end (the user query as input, the final answer as output). This is your regression check: fast, cheap, and it tells you whether the pipeline got worse.
• Per step (one dataset per span: the query and the routing decision; the query and the retrieved context; the query-plus-context and the answer). This is your diagnosis: it tells you which step got worse.

Sample both from real traffic. The end-to-end set catches the regression; the per-step sets tell you where it happened.

Run it and read it

Run an experiment per dataset, with that step’s evaluators attached, and read the aggregate pass rate. The workflow is: watch the end-to-end score for regressions, and when it drops, go to the step-level scores to find the broken link. A drop in retrieval relevance sends you to the retriever; a drop in groundedness with healthy retrieval sends you to the generation prompt.

For the full implementation, tracing the pipeline into spans and wiring up the evaluators across Chroma, Milvus, LlamaIndex, or LangChain, see the cookbook: Trace and evaluate a RAG pipeline.

What this gives you

• A wrong answer points at its cause. Retriever problem or prompt problem, you can tell which, because you scored them separately.
• Every score maps to a lever. Relevance sends you to embeddings and chunking; groundedness sends you to the prompt. No more changing things at random.
• Regression plus diagnosis. The end-to-end score guards against getting worse; the per-step scores tell you what to fix.

Next steps

• Keep these evaluators running on live traffic, so retrieval drift and new failure modes get caught in production, not by a user. That is the online evals page.
• Once a step is measured, iterate its prompt with the single-call loop from earlier in this guide.