How to Properly Evaluate a RAG System: the 9 Essential Metrics

A RAG (Retrieval-Augmented Generation) system has two components that can fail independently. If the answer is poor, you need to know which part is to blame. This article summarizes the 9 essential metrics, explains them visually and shows how to implement them in practice — including how we already use them in a real system of ours.

Why you need metrics on TWO axes

The two components of a RAG:

1. Retrieval — which documents/chunks it brings from the knowledge base.
2. Generation — what answer the model produces from the retrieved documents.

That is why the metrics split into two families + a human one:

Diagram 1 — Where each metric applies in the pipeline

How to read the diagram: the RETRIEVAL metrics evaluate what comes out of the retriever (before the LLM); the GENERATION metrics evaluate the final answer; Human Evaluation is the human verdict over everything.

Family A — RETRIEVAL metrics

1. Context Precision — how much of what you retrieved is relevant

Precision = relevant chunks retrieved / total chunks retrieved

• Intuition: low noise. If you retrieve 5 chunks and only 2 are relevant → precision 0.4.
• Higher = better. Low precision = the retriever brings junk that distracts the LLM.
• Fix when it’s low: add a reranker, lower k, filter on metadata.

2. Context Recall — you retrieved EVERYTHING that was relevant

Recall = relevant chunks retrieved / all relevant chunks

• Intuition: coverage. If there were 3 relevant chunks and you retrieved only 2 → recall 0.67.
• Higher = better. Low recall = the retriever MISSES information → incomplete answer.
• Fix when it’s low: increase k, improve embeddings/chunking, hybrid search (semantic + keyword).

Precision and Recall are in tension: raise k → recall ↑ but precision ↓. The reranker reconciles them (retrieve a lot, then filter to the top).

3. Context Relevance — how relevant the documents are to the question

Checks, qualitatively, whether the retrieved documents actually relate to the query. Higher = better. Implementation: LLM-as-judge gives a relevance score per (query, chunk).

7. Hit Rate (Hit@k) — at least one hit in the top-k

Hit@k = 1 if a relevant doc is in the top-k, otherwise 0.

• Intuition: binary per query. The average over the whole set = % of questions for which the retriever hit at least one good doc in the first k.
• Higher = better. A quick “is the retriever working?” metric.

8. Mean Reciprocal Rank (MRR) — how HIGH the first relevant doc appears

MRR = (1/N) · Σ (1 / rankᵢ) — where rankᵢ = the position of the first relevant document for query i, and N = the total number of queries.

• Intuition: if the first relevant doc is at position 1 → 1.0; at position 2 → 0.5; at position 5 → 0.2.
• Higher = better. Unlike Hit@k (just yes/no), MRR penalizes poor ranking.
• Fix when it’s low: reranker, better embeddings, HNSW tuning.

Family B — GENERATION metrics

4. Faithfulness — the answer is supported by the context (no hallucinations)

Checks whether every claim in the answer is covered by the retrieved context. “No hallucinations.” Higher = better.

• The most important anti-hallucination metric. Low faithfulness = the model is making things up.
• Implementation: LLM-as-judge (compare answer vs context) OR grounding via embeddings (cosine answer↔source span).

5. Answer Relevance — the answer addresses the user’s question

An answer can be 100% faithful to the context yet off topic. It measures how well it answers what the user actually asked. Higher = better. Fix when it’s low: improve the generation prompt, reformulate the query.

6. Answer Correctness — factual correctness against the reference truth

Compares the answer with a ground truth / reference. Higher = better. Requires a “gold” set of correct answers. Implementation: LLM-as-judge or semantic matching vs the reference.

Family C — Human Evaluation

9. Human Evaluation — human judgment

Human judgment of usefulness, accuracy and grounding, on a rubric:

• Correctness (factually correct)
• Helpfulness (useful to the user)
• Grounding (anchored in sources)
• Completeness (complete)

It remains the gold standard for ambiguous cases; it is done on a sample, not on everything.

Diagram 2 — Taxonomy of the 9 metrics

How to implement them in practice

Tools: the ragas library (Python) covers Context Precision/Recall, Faithfulness, Answer Relevance, Answer Correctness (LLM-as-judge) out of the box. The pure ranking metrics (Hit@k, MRR) you compute with simple code over the retriever’s results.

How WE do it (real implementation)

Our RAG pipeline automates exactly 2 of the 9 metrics as a quality gate, at scale:

• Retrieval: vector DB (Qdrant) + embeddings (BGE-M3) + reranker → on a gold set we measure Hit@k, MRR, Context Precision/Recall. The reranker is precisely the fix for low Context Precision + MRR.
• Faithfulness (metric 4): for each question/answer pair, we compute cosine similarity ≥ 0.85 between the answer and the source span → automatic grounding = “No hallucinations”.
• Answer Correctness (metric 6): an LLM-as-judge (score ≥ 9 out of 10) verifies factual correctness against the source; the final certification is given by a stronger judge.

In practice, the “platinum dataset” is built by passing each example through automated Faithfulness + Answer Correctness — two of the metrics above, applied to hundreds of thousands of pairs.

Rough thresholds + what to do when a metric is low

Key takeaways

• You evaluate RAG on retrieval (do you bring the right thing?) and generation (do you answer correctly from what you brought?).
• Recall = you don’t miss; Precision = you don’t bring junk; MRR/Hit@k = good ranking.
• Faithfulness = no hallucinations (the most important); Answer Relevance/Correctness = on-topic and correct answer.
• Human Evaluation = the final verdict on a sample.
• In production, you automate Faithfulness + Correctness with LLM-as-judge / grounding and run them on thousands of examples.