A RAG system needs more than a vector database

Many RAG projects begin by embedding documents, storing vectors and retrieving by similarity. Once a demo answers a few questions, the architecture appears complete. Production reveals that the vector index is only one search component.

Exact product codes are missed, obsolete policies outrank approved versions, semantically close passages come from the wrong department and confidential material enters the candidate list before access rules apply. A larger vector database solves none of this. A retrieval stack must combine meaning, exact strings, metadata, permissions and quality control.

An index answers a limited question

An embedding model maps queries and segments into numerical space. A similarity function ranks proximity. This helps differently phrased texts meet around a topic, but numerical proximity does not establish professional validity.

Without separate fields, an index does not know approval status, effective date, owner, region, language or confidentiality. It cannot tell whether a hit is a definition, example, exception or incidental mention. Those properties must survive outside the vector as structured signals.

Distance function, embedding version and normalisation affect ranking. Representations from different models are not silently interchangeable. Every index build needs versioned configuration and traceable re-indexing.

Lexical search remains essential

Dense retrieval is strong at paraphrases and topical proximity. Full-text search is strong at exact terms, rare names, identifiers, error codes, part numbers and quotations. A query for AB-17.4, a standard or a precise error message should not depend on whether its embedding resembles a general description.

Methods such as BM25 weight term frequency, rarity and document length. They are not obsolete; they answer a different relevance question. Exact identifiers may trigger lexical boosting or deterministic lookup.

Synonyms, language variants and controlled spellings can help, but automatic expansion may distort meaning. „Bank" may denote a financial institution, a seat or a technical component.

Hybrid retrieval combines evidence signals

Hybrid retrieval merges lexical and semantic candidates. Their scores use different scales and should not simply be added. Reciprocal Rank Fusion combines positions from several result lists without requiring comparable scores. Other systems use trained or manually weighted fusion.

Fusion does not guarantee quality. It must be evaluated on real queries. Product-code corpora may need stronger lexical weight; archives with varied language may benefit more from semantic search. Query classes are often better than one global weighting.

The candidate pool should be broader than the final context. Retrieval first optimises recall; reranking then improves precision and order.

Metadata is search logic

Metadata filters are not decoration for citations. They constrain the valid search space. Common fields include document type, status, version, effective period, organisational unit, product, region, language, data class and owner.

Filters may run before or after vector search, changing recall and latency. Approximate nearest-neighbour implementations may evaluate filters after a bounded index scan. The pgvector documentation notes that too few results can remain and describes iterative scans, partial indexes and partitioning among possible strategies.

top_k=10 does not guarantee ten permitted hits. If status or permission filters remove nine candidates, retrieval must continue under control or the context becomes empty or weak.

Permissions apply before generation

Access control cannot live only in the interface. An unauthorised chunk should not enter the candidate pool, logs, caches or model context. Retrieval needs requester identity and role together with document- or segment-level rules.

Permissions are lost when chunks lack provenance. Every unit needs a stable source link and access-control information. Group and approval changes must propagate reliably.

The principle is security trimming before generation. A model cannot forget confidential material after receiving it as context.

Query processing determines what is searched

Questions may be short, ambiguous or conversation-dependent. Query processing can normalise spelling, resolve abbreviations, detect language, concretise time references or rewrite a follow-up as a standalone request.

Transformation helps but can add assumptions. Original and transformed queries should be recorded; critical ambiguity requires clarification. Query routing may select full-text search, a structured database, knowledge graph, API or archive. A current account balance should not come from a static vector index if an authoritative transaction system exists.

Reranking examines candidates in context

A bi-encoder represents query and passage separately for speed. Compression can lose fine relations. Rerankers assess a smaller set more accurately, often by examining query and passage together.

Cross-encoders are accurate but compute-intensive. Late-interaction approaches such as ColBERT retain token-level representations and move part of the interaction into search. The right method depends on latency, corpus size and quality target.

Reranking cannot replace professional rules. An obsolete source does not become authoritative because a model ranks it highly. Hard filters apply first; relevance models then order permitted candidates.

Context assembly is more than top-k

Sending the first five hits unchanged to generation is rarely optimal. Hits may duplicate one another, share a parent, contain conflicting versions or lack neighbours. Context assembly deduplicates, expands relations, preserves source diversity and orders evidence within a budget.

A good packet contains the smallest sufficient evidence rather than maximum text. It may need a definition, rule, exception and validity statement. Source IDs and positions survive so claims can be bound later.

When sources conflict, the system should not silently choose one. It can show both, compare validity or escalate. Retrieval is also conflict detection.

Operations need controlled index generations

Production contains source documents, parse outputs, chunks, embeddings, indexes, caches and replicas. When a document is withdrawn, teams must know when it disappears from every stage. Otherwise a current service answers from an old cache.

An index generation connects corpus snapshot, parser version, chunking rules, embedding model, metadata schema and build time. Deployments switch generations under control. Rollback and deletion are tested.

Observability records query, route, filters, candidates, scores, reranking, selected context and latency while respecting privacy. Without traces, search, filtering and generation failures cannot be separated.

Evaluation needs more than answer scoring

Fluent output can rest on poor retrieval. Recall@k checks whether evidence enters the pool. Precision@k measures relevant hits. MRR and nDCG account for position and graded relevance.

Systems also need permission, filter and freshness tests, latency budgets and cost measurement. A Golden Query Set includes identifiers, paraphrases, ambiguity, version conflicts, empty results and unauthorised material.

Ablation tests compare vector only, lexical only, hybrid without a reranker and hybrid with one. The result shows whether complexity adds measurable value. Architecture should be justified, not maximal.

Candidate records make decisions inspectable

Every retrieval run should produce a compact Candidate Record: original query, route, filters, candidate IDs, rank changes, exclusion reasons and the final context selection. This is not a copy of every confidential passage. Stable references and minimised signals are often enough to reconstruct why one source entered the answer while another did not.

The record also captures controlled empty results. A system should be able to say that no permitted, current and sufficiently relevant evidence was found. Filling the gap with a weak neighbour merely hides retrieval failure. Teams can define a failure budget for empty results, stale hits, permission violations and latency breaches, then investigate recurring patterns by query class.

Method: ROUTE → FILTER → RETRIEVE → FUSE → RERANK → ASSEMBLE → VERIFY

ROUTE selects the source. FILTER limits results to permitted, valid material. RETRIEVE creates lexical and semantic candidates. FUSE combines lists. RERANK orders precisely. ASSEMBLE builds evidence. VERIFY tests relevance, grounding, permissions and freshness.

The vector database remains important. It is only overestimated when mistaken for the entire retrieval system.

Exercise: design a Retrieval Stack Canvas

1. Select ten real questions with identifiers, paraphrases and version references.

2. Assign authoritative sources and routes.

3. Define metadata and permission filters.

4. Specify lexical and semantic candidate sets.

5. Describe fusion, reranking and context assembly.

6. Define recall, ranking, freshness and security tests.

7. Compare the stack against vector-only search.

Reflection: Which question needs exact search rather than semantic similarity? Which result is relevant but prohibited by status or permission?

All materials to download — the topic overview and the worksheet:

Scope: Architecture depends on corpus, query types, permissions, quality targets and operations. No single method guarantees professional correctness. Editorial review date: 17 July 2026.