From documents to usable project knowledge

Storage keeps files. Usable project knowledge preserves meaning, relations, validity and decisions.

A full folder looks like knowledge. At first, however, it contains only files: PDFs, presentations, spreadsheets, chat logs, images and transcripts. Storage becomes a knowledge space only when people and systems can determine why an item exists, where it came from, what it relates to and which decision followed from it.

This distinction is fundamental to AI applications. Full-text search finds words. Semantic search finds passages with similar meaning. Both may still return an outdated, unapproved or decontextualised statement. Findability is only one property. Usability also requires structure, provenance, validity, permissions and testable relations.

Four states of the same information

Information passes through at least four professionally distinct states:

1 · Raw material: audio, a transcript, email, scan, spreadsheet export or conversation note. It contains signals but often also repetition, recognition errors and unresolved speaker references.

2 · Document: the material has a stable title, format, source and timestamp. It can be stored, but is not automatically relevant to a decision.

3 · Context: selected excerpts are provided for a specific task. Selection and segmentation change what becomes visible.

4 · Project knowledge: statements are connected to provenance, relations, status, validity and an accountable role. Decisions remain distinguishable from assumptions and open questions.

Many AI projects jump directly from the first or second state to the third: files are uploaded, split and indexed. The professional transformation is missing. A language model may then summarise a passage persuasively without knowing that it describes a rejected proposal, an unconfirmed remark or a superseded version.

Extraction is not understanding

The first processing stage makes content technically accessible. Text can be extracted from digital PDFs, while scans require optical character recognition. Tables, columns, footnotes, headings and reading order must be preserved or reconstructed. A plausible stream of text does not prove that document structure was captured correctly.

Quality therefore starts with testable extraction attributes: page reference, detected document type, language, OCR confidence, table count and warnings for unreadable regions. For critical sources, extracted text should link back to the visual source page. An error can then be traced from an answer to a passage and finally to the scan.

A useful principle is to keep the original immutable, version the extraction and assign a separate identifier to every derivative. A corrected OCR result should not silently overwrite the previous version. It records its source, transformation process and time.

Segmentation determines what remains retrievable

Long documents are usually divided into smaller units for semantic retrieval. Very large chunks include distracting material; very small chunks lose definitions, qualifications and reasoning. There is no universally correct chunk size. It depends on document type, expected questions, language, embedding model and available context window.

A structure-aware strategy is stronger than blind character counts. Headings, paragraphs, table rows, decisions and speaker changes provide natural boundaries. Overlap can preserve context, but also creates duplicates and must not make one claim appear to be several independent pieces of evidence.

Each segment needs a stable link to its document:

chunk_id: decision-2026-014#section-03
source_id: decision-2026-014
source_version: 2.1
page_or_timecode: "04:12-05:08"
section: "Decision and rationale"
status: approved
valid_from: 2026-06-01
access_class: internal

These fields are not decoration. They enable filtering, citation, access control and error analysis. A text segment without stable provenance has limited value for a defensible answer.

Relations carry more meaning than folders

Folders usually represent one storage location. Project knowledge is multidimensional. One document may relate to a customer, decision, work package, risk assumption and open question at the same time. Relations expose what a path cannot express.

A minimal knowledge structure does not require a knowledge graph. Markdown files with machine-readable headers, persistent identifiers, internal links and a maintained index are often sufficient for smaller collections. What matters is the distinction between relation types: "supports", "contradicts", "supersedes", "applies to", "derived from" and "requires approval" do not mean the same thing.

For larger collections, graph structures and vector search can be combined. Vectors support similarity; graph edges support explicit relations. Neither decides automatically whether a source is valid. Status and permissions must act as binding filters before retrieval, rather than merely appearing as suggestions in a prompt.

From search result to evidence package

A reliable retrieval system does not simply return the most similar text. It builds an evidence package for a concrete question:

1. Determine user intent, task and permitted knowledge space.

2. Apply permissions and protection classes before searching.

3. Find candidates lexically, semantically or through relations.

4. Rerank by relevance, validity, source quality and diversity.

5. Expose contradictions, superseded versions and missing evidence.

6. Pass only the smallest sufficient package to the model.

7. Log the answer, sources used and retrieval configuration.

The original RAG work combined a generative model with an external non-parametric memory. That does not make every RAG answer true. Retrieval may return irrelevant passages, generation may combine evidence incorrectly, and the knowledge base itself may contain errors. Retrieval and answer quality therefore require separate evaluation.

Quality is measured at several layers

End-to-end scoring alone reveals too little. The pipeline needs distinct checks. Extraction tests assess reading order, tables and character errors. Metadata tests assess required fields, controlled values and references. Retrieval tests determine whether relevant sources appear near the top and whether blocked material remains excluded. Answer tests measure grounding, completeness, uncertainty and citation correctness.

A useful test set contains real tasks, difficult negative cases and known contradictions. It also includes questions the system must not answer. "Insufficient evidence" is the correct result when the knowledge space provides no defensible basis.

Knowledge needs a lifecycle

Project knowledge changes. Decisions are revised, sources age, roles change and protection classes are updated. Every knowledge base therefore needs a cycle of capture, extraction, classification, linking, review, approval, use, update and archive.

AI can propose metadata, recognise entities, group similar content, flag contradictions and create summaries. A proposal is not an approval. Humans secure professional meaning, authority and exceptions; automation makes discovery scalable and repeatable.

The decisive architectural idea is simple: the goal is neither the largest collection nor the highest number of links. The goal is to give a person or agent the smallest reliable, permitted and traceable context for a concrete task.

Worksheet: From raw material to knowledge

Choose a transcript, conversation note or longer document and turn it into a small, testable knowledge unit.

1. Secure the source. Assign a persistent identifier and record origin, date, format, language and protection class. Do not alter the original.

2. Check extraction. Mark at least two places where structure, speaker identity or table content could be misread.

3. Create knowledge units. Produce a summary, three semantic segments and one decision record. Every artefact links back to the source.

4. Model relations. Use at least two distinct relation types, such as "supports", "contradicts" or "supersedes".

5. Test retrieval. Write five questions: three answerable, one ambiguous and one prohibited or unsupported. Record results, evidence and correct refusal behaviour.

6. Separate approval. Mark what was generated automatically, reviewed by a person and formally approved.

All materials to download — the topic overview and the worksheet:

Scope: A vector index is neither proof of truth nor automatically a records-management system. The architecture described here combines technical retrieval with provenance, validity, permissions and human accountability. Concrete obligations depend on the use context and legal classification.

Sources and professional scope