Citation Grounding in RAG: Architecture and Three Traps for Legal and Audit
Every legal counsel who has watched a model invent an article number knows the feeling. The answer sounds authoritative, the paragraph flows, and the citation points to a regulation that does not exist.
Citation Grounding in RAG: Architecture and Three Traps for Legal and Audit
Every legal counsel who has watched a model invent an article number knows the feeling. The answer sounds authoritative, the paragraph flows, and the citation points to a regulation that does not exist. Citation grounding is what stops that failure — and it is harder to build correctly than most engineering teams assume, especially when the downstream reader is an auditor or a judge.
The architecture that actually holds up
A grounded RAG pipeline is not a retriever with a footnote generator bolted on. It is a system that keeps a verifiable chain from the source paragraph to the rendered claim, and refuses to answer when the chain breaks.
The non-negotiable pieces are stable document IDs (never regenerated on re-index), span-level offsets that survive reformatting, and a post-generation check that rejects any sentence lacking a mapped citation. The European Commission’s guidance for AI in justice systems explicitly asks for traceability from output back to source material, which is what this loop provides. Teams building for GDPR obligations should also read ENISA’s note on AI cybersecurity and transparency before deciding retention windows for the audit trail.
Three traps we keep seeing
Trap one — citing the chunk, not the claim. A model quoting “Regulation (EU) 2016/679, Article 6(1)(f)” while the retrieved chunk actually covered Article 6(1)(a) is technically cited and materially wrong. Fix it with per-sentence alignment scoring, not per-answer.
Trap two — silent re-chunking. When someone re-embeds the corpus with a new chunk size, old citation offsets point to different text. Freeze the chunk boundaries or version them; the EUR-Lex document versioning convention is a useful mental model for tracking drift over time.
Trap three — trusting a single retriever on legal negations. “Nu se aplică” and “se aplică” sit inches apart in the same paragraph. Dense retrieval frequently ranks the wrong half first. A hybrid setup with BM25 fallback and a rerank step catches this, as NIST’s TREC Legal Track documented years before RAG existed.
grounding:
min_coverage: 0.80
refuse_below: 0.65
span_check: strict
retention_days: 730
reranker: cross-encoder-legal-ro
chunk_version: 2026-03-frozenWhere we sit
At CAI Technology we treat citation grounding as the boundary between a demo and a production system for regulated work. Our RAG security practice leans hard on refusal-when-uncertain rather than confident guessing, and our agent routing work reuses the same trace format so a lawyer can follow a multi-hop answer end to end. The audit trail is not a nice-to-have added at the end; it is the first table you design, before you pick an embedding model. If you are scoping a legal or audit build, start from the refusal policy and work backwards to retrieval — then talk to us about the parts that break at production scale.