Chunking, parsing and the lost connections

Retrieval systems rarely operate on complete documents. They divide content into smaller units, create representations and later search for relevant segments. The procedure sounds mechanical: extract text, cut at a fixed length, create embeddings and finish. Yet this is where many failures emerge that are later blamed on the language model.

When retrieval returns a table row without its header, an exception without its rule or a caption without its figure, the result is semantically damaged. It may contain similar words while no longer expressing the same claim. Chunking is not a storage optimisation. It determines which knowledge units the system can see, compare and cite.

Parsing comes before chunking

A PDF is initially an arrangement of characters and graphic elements on pages. Visual order does not automatically equal machine-readable reading order. Columns, headers, footers, page numbers, footnotes, callouts, tables and scans can be scrambled by simple extraction.

Parsing reconstructs document structure from layout. It identifies heading levels, paragraphs, lists, tables, figures, captions, footnotes and page references. Scans also require optical character recognition. Only after reconstruction can a system decide defensibly what belongs together.

Parsers can fail: a footer becomes repeated prose, two columns interleave, hyphens are resolved incorrectly or a table collapses into unrelated words. A parser should therefore emit structured elements with type, position, reading order and confidence—not only plain text. Docling describes such an approach with layout analysis and table-structure recognition. The tool is replaceable; the architectural principle is not.

Documents contain several orders at once

A document has linear order, hierarchical organisation and reference relations. A paragraph follows another, belongs beneath a heading and may point to a table three pages later. Plain text preserves only the linear sequence.

Reliable retrieval also needs a path: document → chapter → section → element. Neighbours, cross-references and source version belong with it. „The following exceptions apply" is meaningless without its heading and predecessor. A path such as Security policy → Access control → External accounts → Exceptions exposes the scope.

Fixed windows are a baseline, not a strategy

The simplest segmentation cuts at a fixed number of tokens or characters. It is fast and reproducible but ignores sentence, paragraph and section boundaries. The cut can fall precisely between a rule and its condition.

Small chunks improve topical precision while losing definitions, qualifications and rationale. Large chunks preserve more context but mix subjects, consume the context budget and dilute the relevant passage. There is no universally correct size. A glossary entry, contract clause, table row and method chapter require different segmentation profiles.

Overlap repeats part of one segment in the next. It helps with short arguments but creates duplicates and does not restore structure. Repeating one hundred tokens does not guarantee that a table row receives its column header. A footnote remains ambiguous without its reference anchor. Overlap is a local safety margin, not a substitute for explicit relations.

Tables are small data models

A table cell derives meaning from row and column headers, units, grouping and often a footnote. A segment containing „12 | 18 | 24" cannot reveal whether those values are months, percentages or quantities. A useful table segment includes title, relevant headers, units, row label, values, footnotes and source location.

Large tables may be divided into row groups, but every child links to a parent representing the full table. A structured data representation may be better than text for trend questions. Likewise, caption, figure ID and explanatory paragraph remain linked even when stored separately.

Cross-references create hidden dependencies

Phrases such as „as described above", „these values" or „it does not apply to" point elsewhere. A segment can be grammatically complete while its referent has vanished. A context packet may therefore contain a small primary chunk, section path, short parent summary and selectively resolved references.

Parent-child models separate search from answer granularity. Small children are easy to retrieve; a hit triggers controlled loading of a larger parent or defined neighbourhood. This is often more robust than making every chunk large from the outset.

Semantic and hierarchical methods

Semantic chunking places boundaries where the topic appears to change. It may outperform rigid windows in prose but does not always recognise formal relations. A robust strategy combines hard structural boundaries with soft semantic signals.

Late Chunking first processes a broader context and only then produces representations for subregions. Chunk embeddings can consequently carry information from surrounding text. This is promising but not a universal repair: broken reading order, table structure or missing versions remain broken.

RAPTOR recursively summarises and organises content in a tree so retrieval can operate at several abstraction levels. Detail segments and higher-level syntheses may be indexed together, but they remain different evidence types. A summary is not a primary source; final claims should return to underlying passages.

A large context window does not remove the problem. Research on the „Lost in the Middle" effect reports that models can use relevant information less effectively depending on its position in a long input. More context is not automatically better context. Selection, ordering and emphasis remain engineering tasks.

Extraction is not understanding

The first processing stage makes content technically accessible. Text can be extracted from digital PDFs, while scans require optical character recognition. Tables, columns, footnotes, headings and reading order must be preserved or reconstructed. A plausible stream of text does not prove that document structure was captured correctly.

Quality therefore starts with testable extraction attributes: page reference, detected document type, language, OCR confidence, table count and warnings for unreadable regions. For critical sources, extracted text should link back to the visual source page. An error can then be traced from an answer to a passage and finally to the scan.

A useful principle is to keep the original immutable, version the extraction and assign a separate identifier to every derivative. A corrected OCR result should not silently overwrite the previous version. It records its source, transformation process and time.

Segmentation determines what remains retrievable

Long documents are usually divided into smaller units for semantic retrieval. Very large chunks include distracting material; very small chunks lose definitions, qualifications and reasoning. There is no universally correct chunk size. It depends on document type, expected questions, language, embedding model and available context window.

A structure-aware strategy is stronger than blind character counts. Headings, paragraphs, table rows, decisions and speaker changes provide natural boundaries. Overlap can preserve context, but also creates duplicates and must not make one claim appear to be several independent pieces of evidence.

Each segment needs a stable link to its document:

chunk_id: decision-2026-014#section-03
source_id: decision-2026-014
source_version: 2.1
page_or_timecode: "04:12-05:08"
section: "Decision and rationale"
status: approved
valid_from: 2026-06-01
access_class: internal

These fields are not decoration. They enable filtering, citation, access control and error analysis. A text segment without stable provenance has limited value for a defensible answer.

Relations carry more meaning than folders

Folders usually represent one storage location. Project knowledge is multidimensional. One document may relate to a customer, decision, work package, risk assumption and open question at the same time. Relations expose what a path cannot express.

A minimal knowledge structure does not require a knowledge graph. Markdown files with machine-readable headers, persistent identifiers, internal links and a maintained index are often sufficient for smaller collections. What matters is the distinction between relation types: „supports", „contradicts", „supersedes", „applies to", „derived from" and „requires approval" do not mean the same thing.

For larger collections, graph structures and vector search can be combined. Vectors support similarity; graph edges support explicit relations. Neither decides automatically whether a source is valid. Status and permissions must act as binding filters before retrieval, rather than merely appearing as suggestions in a prompt.

Quality is measured at several layers

End-to-end scoring alone reveals too little. The pipeline needs distinct checks. Extraction tests assess reading order, tables and character errors. Metadata tests assess required fields, controlled values and references. Retrieval tests determine whether relevant sources appear near the top and whether blocked material remains excluded. Answer tests measure grounding, completeness, uncertainty and citation correctness.

A useful test set contains real tasks, difficult negative cases and known contradictions. It also includes questions the system must not answer. „Insufficient evidence" is the correct result when the knowledge space provides no defensible basis.

Knowledge needs a lifecycle

Project knowledge changes. Decisions are revised, sources age, roles change and protection classes are updated. Every knowledge base therefore needs a cycle of capture, extraction, classification, linking, review, approval, use, update and archive.

AI can propose metadata, recognise entities, group similar content, flag contradictions and create summaries. A proposal is not an approval. Humans secure professional meaning, authority and exceptions; automation makes discovery scalable and repeatable.

The decisive architectural idea is simple: the goal is neither the largest collection nor the highest number of links. The goal is to give a person or agent the smallest reliable, permitted and traceable context for a concrete task.

Method: PARSE → STRUCTURE → SEGMENT → LINK → RETRIEVE → REASSEMBLE → TEST

PARSE reconstructs reading order and elements. STRUCTURE models hierarchy and types. SEGMENT creates purpose-specific units. LINK preserves parents, neighbours and references. RETRIEVE finds small candidates. REASSEMBLE adds necessary context under control. TEST exercises boundaries, citations and versions.

The rule is: as small as precise retrieval requires, as rich in relations as correct interpretation demands.

Exercise: build a Chunk Integrity Map

1. Choose three pages with a heading, table or cross-reference.

2. Mark reading order and element types.

3. Draw and justify chunk boundaries.

4. Record parents, neighbours, references and metadata.

5. Write one table and one exception question.

6. Test whether a hit needs controlled expansion.

7. Document one parsing, boundary and reassembly failure.

Reflection: Which information loses its meaning without a heading or neighbouring paragraph? Which relation should be stored explicitly instead of relying on overlap?

All materials to download — the topic overview and the worksheet:

Scope: Optimal chunking depends on corpus, model and task. These approaches are architectural components, not universal performance guarantees. Editorial review date: 17 July 2026.