RAG is not a truth machine

Retrieval-augmented generation appears to solve a familiar problem elegantly. Instead of relying only on parametric model knowledge, the system searches an external collection and provides relevant material as context. Sources can be updated, internal documents included and answers linked to evidence.

This is a major improvement, but it is not a truth mechanism. A RAG system can retrieve the wrong source, split the right source badly, miss a valid passage, rank obsolete content higher, expose unauthorised material or distort evidence during generation. Quality is created along an evidence chain, not by one model call.

RAG moves the problem

Without retrieval, the central questions concern what the model learned and how reliably it can use that knowledge. RAG adds more: does the corpus contain the required information? Is it valid, accessible and correctly described? Was the query interpreted properly? Did the retriever find the relevant passage? Was it ranked high enough, included in the context window and used faithfully?

RAG reduces some risks while adding a complex system surface. The foundational NeurIPS work combined parametric memory with retrievable non-parametric memory and reported more specific and factual language than a parametric-only comparison. That result does not imply that every modern pipeline is reliable. Corpus, retriever, ranking, prompt, model and evaluation differ substantially.

A useful model separates six controls: ingest knowledge, constrain candidates, retrieve evidence, rank evidence, assemble context and generate an answer. Each can preserve or destroy quality.

Similarity is neither truth nor authority

Vector search roughly identifies representations close to a query. It does not automatically establish which source is true, current, approved or authorised for the requester. A detailed obsolete draft may be semantically closer than a brief current decision. A widely repeated claim may be easier to retrieve than its correction.

Retrieval therefore needs an admissible candidate space. Tenant, permission, protection class, validity period, status, language, region and permitted purpose are filtered before or enforceably within search. Security rules cannot live only in the prompt. The model should never receive content it is not authorised to process.

Hybrid search can then combine lexical and semantic signals. Exact identifiers, names, clauses and product codes often benefit from keyword retrieval; paraphrases benefit from dense representations. A reranker can score candidates more precisely for the query. These tools improve selection but do not replace authority, status or validity logic.

Chunking determines what can be seen

Documents are divided because entire files rarely fit efficient indexing and generation. Chunks that are too small lose definitions, exceptions or temporal scope. Chunks that are too large dilute relevant signal and consume context. Fixed character windows can separate a heading from its paragraph, a table header from values or a rule from its exception.

Chunking should respect structure: headings, paragraphs, lists, tables, pages, speakers and domain units. Every chunk inherits document ID, position, version, status, language, permission and provenance. Overlap can preserve context, but creates duplicates and must not inflate retrieval measurements.

Cross-section relations are especially difficult. A definition may appear at the beginning, an exception three pages later and the effective date in an annex. No single chunk contains the answer. Parent-child retrieval, neighbourhood expansion or multi-stage retrieval can help. Such cases must exist in the evaluation set.

Query transformation can help and distort

Systems rewrite user questions, generate subquestions, expand synonyms or create hypothetical answers as retrieval anchors. These techniques can increase recall. They can also shift meaning, remove constraints or turn an open question into an unsupported assumption.

Preserve the original query. Record every rewrite as a derived object and compare it in tests. Critical constraints — time, person, product, jurisdiction — must survive transformation. For complex questions, multiple searches followed by evidence synthesis are often safer than treating one "optimised" query as authoritative.

Conversation history can also interfere. A pronoun may need context, but previous assumptions should not silently control a new search. Query rewriting is a controlled interpretation step, not an invisible convenience filter.

Retrieved evidence is not yet a faithful answer

Even with the right passage in context, the model may ignore, miscombine or embellish it. Fluent text can be stronger than its source: "may" becomes "must", an example becomes a universal rule, correlation becomes causation.

An evidence contract narrows the task. Material claims point to concrete passages. Contradictions remain visible. The system distinguishes direct evidence, permitted inference and unsupported interpretation. If evidence is insufficient, it lowers the claim, asks a question or abstains.

Abstention is a quality capability. A system that always answers optimises appearance rather than reliability. Thresholds follow risk: cautious hypotheses may support ideation; consequential decisions require stronger evidence and often human review.

Citation is a separate verification task

A citation can exist and still be wrong. The passage may mention a topic without supporting the assigned claim. An answer may cite one sentence correctly and leave other material claims unsupported. Three questions therefore require separate evaluation:

1 · Citation correctness: does the passage entail or support the associated claim?

2 · Citation completeness: are all material verifiable claims supported?

3 · Source quality: is the source appropriate for purpose, time and scope?