RAG explained: what it is and how to evaluate its quality

RAG (Retrieval-Augmented Generation) is an architecture that connects a language model to an external knowledge base: instead of answering only with what it “remembers” from its training, the system first searches for relevant documents and then generates the answer drawing on them. It’s the most used technique for having an assistant answer about a company’s specific information —its policies, its catalog, its documentation— without retraining the model.

RAG is also the best-known defense against hallucinations, because it anchors answers in real sources. But it has a catch: a RAG system can fail in two very different ways, and if you don’t know which of the two is happening, you’ll fix what isn’t broken. Understanding how to evaluate it starts with understanding that it’s not one component, but two.

The anatomy: two components that fail differently

Every RAG system has two parts, and it’s best to think of them separately:^[1]

1. The retriever. It receives the user’s question and searches, in the knowledge base, for the most relevant document fragments. Its job is to bring in the right evidence.
2. The generator. It takes those fragments and the question, and writes the final answer. Its job is to use the evidence it received well.

The key to evaluating RAG is this: a bad answer can come from either of the two, and they are fixed in opposite ways. If the retriever brought in the wrong document, it doesn’t matter how good the generator is: it will produce a fluent answer based on irrelevant information. If the retriever brought in the right document but the generator ignored it or distorted it, the problem is on the other side. An end-to-end quality score tells you that something failed, not which of the two.^[2] That’s why RAG evaluation measures each component separately.

How the retriever is evaluated

Here the question is: did it bring in the right evidence? The standard metrics come from the world of information retrieval:^[3]

• Context precision: of the fragments it brought in, what proportion is actually relevant? Low precision means it’s introducing noise, which dilutes the answer and makes the generator spend attention on irrelevant information.
• Context recall: of all the relevant information that existed, how much did it manage to retrieve? Low recall means evidence it needed slipped through.

There are finer metrics (MRR, nDCG, which also look at whether the correct documents ended up well ranked), but context precision and recall are the starting point.^[3:1]

How the generator is evaluated

Here the question changes: given the context it received, did it use it well? The two central metrics:^[4]

• Faithfulness: is the answer consistent with the retrieved context, or does it add things that weren’t there? This is the RAG metric: it measures whether the generator stuck to the evidence or “invented” beyond it. An answer can be correct in the real world but unfaithful to the context —and that, in a system that must answer only from approved sources, is also a problem.
• Answer relevancy: does the answer actually address the question that was asked, without wandering off?

Some frameworks combine these four metrics —context precision, context recall, faithfulness and answer relevancy— into a single score that summarizes the health of the pipeline.^[5]

Why the old metrics aren’t enough

If you come from the classic NLP world, you might think of using metrics such as BLEU or ROUGE, which compare the generated answer against a reference answer. They’re not enough: they were designed for pure generation tasks and measure word overlap, not whether the answer is grounded in the retrieved context nor whether the retriever did its job well.^[6] Evaluating RAG requires metrics built for its hybrid architecture —and, in large part, AI judges (LLM-as-a-judge) that can assess faithfulness and relevance with the nuance a string comparison doesn’t capture.^[7]

A detail that matters in production

Two practical warnings. The first: RAG evaluation is not a one-time event. In production it has to be evaluated continuously —because the knowledge base changes, documents get updated and behavior can drift— through monitoring and, when possible, A/B testing.^[3:2] The second: for enterprises, RAG errors are not cosmetic. In a system that answers about policies, contracts or regulations, a retrieval or faithfulness failure can mean a compliance failure, reputational damage or legal exposure.^[3:3] A chatbot being wrong about the weather is not the same as being wrong about a contract clause.

How ArtificialQA solves it

Measuring all this —separating retriever from generator, over representative cases and continuously— is what distinguishes knowing that your RAG fails from knowing why. ArtificialQA connects to your system —by URL or API, without writing code— and evaluates it on the dimensions that matter: adherence to context and to the documents (the faithfulness that is the heart of RAG), relevance, accuracy and hallucinations. And because it calibrates its own AI judges, the faithfulness and relevance scores —which depend on a judge— withstand scrutiny instead of being an opaque figure.

For a team that built an assistant over its own documentation, that means being able to answer the question that really matters: when the system gets it wrong, is it that it didn’t find the information, or that it found it and used it wrong? Knowing the difference is half the fix.

Frequently asked questions

What is RAG in artificial intelligence? It is an architecture (Retrieval-Augmented Generation) that connects a language model to an external knowledge base: the system searches for relevant documents and generates the answer drawing on them, instead of answering only from its training.

Why is RAG used? So that an assistant can answer about specific information (a company’s policies, catalogs, documentation) without retraining the model, and to reduce hallucinations by anchoring answers in real sources.

What are the metrics to evaluate a RAG system? On the retriever side: context precision and context recall (and ranking metrics such as MRR or nDCG). On the generator side: faithfulness (fidelity to the context) and answer relevancy. Faithfulness is the central RAG metric.

Why must the retriever and the generator be evaluated separately? Because a bad answer can come from either of the two and they are fixed in opposite ways: if the retriever brought in the wrong document, the problem is retrieval; if it brought it in well but the generator ignored it, the problem is generation. An end-to-end score doesn’t distinguish which one failed.

Are BLEU or ROUGE useful for evaluating RAG? They’re not enough. They were designed for pure generation and measure word overlap, not fidelity to the retrieved context nor the quality of the retrieval. RAG needs its own metrics, often based on AI judges.