RAG is not a chatbot with attached PDFs

A document chat may look like retrieval-augmented generation: PDFs are added, a question is asked and a fluent answer appears. Yet the implementation may only provide temporary file context. It may lack a governed knowledge corpus, explicit search strategy, stable source references, separate quality measurements and an update process. The tool can still be useful; that alone does not make it a defensible RAG architecture.

RAG connects parametric model knowledge with an external, replaceable knowledge source. The foundational research describes a generator combined with non-parametric memory. In practice this is not one feature but a pipeline. Quality depends on document preparation, indexing, retrieval, context selection, answer logic and evidence—not only on the language model.

Four stages instead of a file attachment

An understandable RAG pipeline has four core stages. Retrieval finds candidates for a query. Context Assembly orders, filters and limits the passages. Synthesis produces an answer using that context. Source Binding links claims to verifiable locations.

Every stage can fail independently. Retrieval can miss the decisive passage. Context assembly can cut off an exception. Generation can add a plausible invention. Citation logic can display a topically related source that does not support the actual claim. A polished interface hides these differences; professional engineering exposes them.

The knowledge corpus exists before the first question

Before chat comes the ingestion pipeline. Documents are acquired, checked for readability, parsed, structured, segmented, enriched with metadata and indexed. Version, validity, owner, language, permission and provenance must survive. A scan without OCR, a table without column relations or a manual without version status does not become reliable merely by receiving an embedding.

At query time, the complete archive is not copied into the context window. The retriever creates a bounded candidate set using lexical, semantic, hybrid or multi-stage search. Results may then be reranked, filtered by permission and arranged within a context budget. Only then does answer generation begin.

A vector index is a search component, not a knowledge guarantee

Embeddings represent content numerically; similarity search finds nearby vectors. This is powerful, but proximity does not establish professional relevance, currency or authority. Dense Passage Retrieval demonstrates retrieval with dense representations. FAISS demonstrates efficient similarity search over large vector collections. Neither decides which source is binding in a particular organisation.

Metadata and policy therefore remain essential. A result can be semantically close while expired, unapproved or restricted for the requesting role. Retrieval must enforce those conditions instead of leaving them to a language model after the fact.

RAG changes knowledge without fully retraining a model

One advantage is that the external knowledge corpus can be replaced. Documents may be added, blocked or re-indexed without fully retraining the base model. This can make updating and provenance easier, but it does not ensure immediate propagation. Caches, replicas, old indexes and delayed pipelines also require control.

RAG is therefore useful when answers should depend on a defined, changing corpus. A simpler design may be better for tasks without knowledge retrieval or for a very small set of stable rules. Architecture begins with the problem, not with a desire for a vector database.

RAG moves the problem

Without retrieval, the central questions concern what the model learned and how reliably it can use that knowledge. RAG adds more: does the corpus contain the required information? Is it valid, accessible and correctly described? Was the query interpreted properly? Did the retriever find the relevant passage? Was it ranked high enough, included in the context window and used faithfully?

RAG reduces some risks while adding a complex system surface. The foundational NeurIPS work combined parametric memory with retrievable non-parametric memory and reported more specific and factual language than a parametric-only comparison. That result does not imply that every modern pipeline is reliable. Corpus, retriever, ranking, prompt, model and evaluation differ substantially.

A useful model separates six controls: ingest knowledge, constrain candidates, retrieve evidence, rank evidence, assemble context and generate an answer. Each can preserve or destroy quality.

Similarity is neither truth nor authority

Vector search roughly identifies representations close to a query. It does not automatically establish which source is true, current, approved or authorised for the requester. A detailed obsolete draft may be semantically closer than a brief current decision. A widely repeated claim may be easier to retrieve than its correction.

Retrieval therefore needs an admissible candidate space. Tenant, permission, protection class, validity period, status, language, region and permitted purpose are filtered before or enforceably within search. Security rules cannot live only in the prompt. The model should never receive content it is not authorised to process.

Hybrid search can then combine lexical and semantic signals. Exact identifiers, names, clauses and product codes often benefit from keyword retrieval; paraphrases benefit from dense representations. A reranker can score candidates more precisely for the query. These tools improve selection but do not replace authority, status or validity logic.

Query transformation can help and distort

Systems rewrite user questions, generate subquestions, expand synonyms or create hypothetical answers as retrieval anchors. These techniques can increase recall. They can also shift meaning, remove constraints or turn an open question into an unsupported assumption.

Preserve the original query. Record every rewrite as a derived object and compare it in tests. Critical constraints—time, person, product, jurisdiction—must survive transformation. For complex questions, multiple searches followed by evidence synthesis are often safer than treating one „optimised" query as authoritative.

Conversation history can also interfere. A pronoun may need context, but previous assumptions should not silently control a new search. Query rewriting is a controlled interpretation step, not an invisible convenience filter.

Retrieved evidence is not yet a faithful answer

Even with the right passage in context, the model may ignore, miscombine or embellish it. Fluent text can be stronger than its source: „may" becomes „must", an example becomes a universal rule, correlation becomes causation.

An evidence contract narrows the task. Material claims point to concrete passages. Contradictions remain visible. The system distinguishes direct evidence, permitted inference and unsupported interpretation. If evidence is insufficient, it lowers the claim, asks a question or abstains.

Abstention is a quality capability. A system that always answers optimises appearance rather than reliability. Thresholds follow risk: cautious hypotheses may support ideation; consequential decisions require stronger evidence and often human review.

Citation is a separate verification task

A citation can exist and still be wrong. The passage may mention a topic without supporting the assigned claim. An answer may cite one sentence correctly and leave other material claims unsupported. Three questions therefore require separate evaluation:

1 · Citation correctness: does the passage entail or support the associated claim?

2 · Citation completeness: are all material verifiable claims supported?

3 · Source quality: is the source appropriate for purpose, time and scope?

Citations should identify the used version and, where possible, a stable passage. A home page or entire PDF makes verification harder. The system must not present generated prose as a quotation. Direct quotation, paraphrase and synthesis remain distinguishable.

Provenance connects answer, run, retrieved chunks, index version, extraction and original. This allows error localisation and impact analysis after source withdrawal.

Evaluate retrieval and generation separately

A single judgement that the „answer was good" does not locate failure. Evaluation begins with the corpus: does it cover the questions and contain valid versions? Retrieval is then tested: does relevant evidence appear in the top-k, at which position, and are prohibited or obsolete sources excluded?

Generation evaluation asks whether the answer is supported by supplied context, professionally correct, appropriately uncertain, citation-complete and capable of abstaining when the question is unanswerable.

A defensible test set contains ordinary questions, rare terms, ambiguity, multi-hop cases, old versions, contradictions, hard negatives, missing answers and attacks. It measures critical error classes and affected groups, not only averages.

Retrieval metrics aid diagnosis. Recall@k asks whether relevant evidence appears among the first k results. Precision measures the relevant share. MRR weights the rank of the first relevant result; nDCG accounts for graded relevance and ordering. None replaces professional relevance judgements. BEIR also demonstrates that retrieval architectures vary across domains and that robust baselines matter.

Worksheet: Decompose a RAG answer

Choose at least ten documents and create eight test questions.

1. Qualify the corpus: label valid, obsolete, restricted, contradictory and insufficient sources.

2. Compare chunking: create two segmentation strategies and test definitions, exceptions, tables and multi-hop questions.

3. Test retrieval: compare lexical, semantic and hybrid retrieval with a reranker.

4. Enforce permissions: ensure restricted content never reaches model context, even with perfect similarity.

5. Apply an evidence contract: map each material claim to a passage and label inference or uncertainty.

6. Test abstention: add two unanswerable questions and define the expected response.

7. Simulate an attack: place an indirect instruction in a test document and inspect tool and response behaviour.

8. Locate failures: assign every error to corpus, chunking, query, retrieval, ranking, context, generation or citation.

Reflection: Which failure looked like hallucination but originated in retrieval? For which question was abstention the highest-quality decision?

All materials to download — the topic overview and the worksheet:

Scope: RAG is an architecture family, not a single quality promise. Retrieval can make answers more current, specific and evidentially supported, but it does not guarantee truth, completeness, legality or security. Metrics such as Recall@k, Precision, MRR and nDCG require interpretation against the dataset, relevance judgements and use case. This article presents a technical quality model, not a guarantee for a provider or deployment.