Data needs a lifecycle

Many data projects think carefully about ingestion and use, but barely consider what follows. Documents are uploaded, tables merged, conversations indexed and outputs retained. The collection grows with every step. What is often missing is a system that determines when an object becomes active, who reviews it, what supersedes it and how it is responsibly retired.

A lifecycle is not a circular graphic for presentations. It is a state model with accountabilities, rules, events and testable transitions. That is what keeps a knowledge space current, legitimately usable and controllable by AI systems.

The lifecycle begins before capture

The first question is not where to store a file. It is why the information is needed. Purpose, intended use, affected people, protection needs and evidence requirements determine which data should be captured and what quality is necessary.

Collecting everything because it may become useful merely defers the decision. Unknown copies, conflicting versions and ownerless stores follow. Personal data is also subject to context-specific legal principles, including purpose limitation, minimisation and storage limitation. Other information may be governed by statutory, contractual or professional retention duties.

At entry, an object should therefore receive at least purpose, source, accountable owner, protection class, status, validity period, review date and intended disposition rule. Missing values create lifecycle debt from the first day.

States and transitions matter more than phase names

Labels such as create, store, use, archive and delete provide orientation, but real data does not move in one direction. An archived record may be restored for an audit. A source may be withdrawn, corrected and republished. A draft may be rejected before becoming active.

A state model is more precise. A knowledge object might be draft, in_review, approved, active, restricted, superseded, archived, legal_hold or disposed. Every transition has a trigger, authorised role, preconditions, event record and downstream effects.

object_id: contract-guide-v4
state: active
valid_from: 2026-07-01
review_due: 2027-01-15
owner_role: legal_content_owner
retention_class: business_guidance
supersedes: contract-guide-v3
on_expiry: block_retrieval_and_review

Value comes from enforcement. When review_due passes, the system should not merely send an email and continue unchanged. Depending on risk, it labels the object, removes it from default retrieval, requests review or continues only with a visible limitation.

Raw data, curated data and active evidence are different

A robust architecture separates layers. The raw zone preserves the original capture unchanged and protected. The curated zone contains cleaned, normalised or enriched versions with documented derivation. The active knowledge space includes only sources approved for a purpose, period and audience. The archive preserves selected historical objects and evidence without injecting them into every AI request.

This prevents two common failures. A correction is not silently written over the original, and "findable" is not confused with "currently usable". Historical records may remain valuable without overriding a current decision out of context.

Quality and approval gates sit between layers. They assess identity, format, completeness, provenance, rights, protection class, version and purpose. A gate is a recorded decision, not merely a script that exited successfully.

Every use creates new lifecycle objects

AI systems do not only consume data. They produce chunks, embeddings, summaries, classifications, prompts, tool results, logs and answers. Each artefact has its own purpose, sensitivity and lifecycle.

A document may be deleted while its chunks remain in a vector index. A false figure may disappear from the knowledge base but survive in a stored summary. Personal information may leave the interface while remaining in debugging logs. Provenance must therefore expose derivations in both directions.

For retrieval-augmented generation, the chain might be original → extraction → curated document → chunk → embedding → index entry → retrieved evidence → answer. A source-state change triggers defined actions along this chain: block, re-extract, re-embed, invalidate the index, review dependent summaries and flag affected outputs.

Not every output should be retained indefinitely. Some run data is useful only for short-term debugging; other evidence may be needed longer for audit, safety or professional accountability. Purpose and risk — not available storage — drive the decision.

Retention asks two questions: how long and in which state?

"Seven years" is not a complete retention rule. It needs a starting event, exceptions, accountable role and final action. Does the period begin at creation, contract termination, last modification or closure of a case? What happens during litigation or investigation?

Retention also does not imply that data stays active in AI context for the whole period. An object may be protected in an archive for evidence while excluded from search, training and generation. Access, encryption, integrity control and recoverability should match the archive function.

A legal_hold or comparable preservation status may temporarily suspend ordinary disposal. It must be justified, authorised, reviewed and eventually released. Permanent exceptions indicate weak governance.

Deletion is a chain, not a button

Removing a database row does not automatically end the lifecycle. Copies may remain in caches, search indexes, exports, data lakes, analytical tables, logs, backups and user devices. Derived artefacts may preserve part of the content.

Disposition therefore needs scope. The system discovers dependent objects, applies the intended method, records success and failure and checks residual stores. Media sanitisation is a separate technical discipline: NIST SP 800-88 Rev. 2 frames it as rendering access to target data infeasible for a given level of effort, with methods selected according to sensitivity and context.

Backups require honest treatment. Individual objects often cannot be removed immediately from immutable backup sets. They may be deleted from production, registered for suppression during restore and age out through a documented backup cycle. Restoration must not silently reactivate disposed data.

Nor should a deletion record preserve the entire deleted content. It should minimally record identifier, governing rule, time, method, executing role, affected systems and outcome.

Lifecycle control needs events and owners

Calendar dates are not enough. State changes also follow new versions, consent withdrawal, contract termination, source correction, role changes, incidents, model updates or altered purposes. These events should trigger automated checks.

A lifecycle matrix connects data type, purpose, owner, active state, review, retention, archive target, disposal method and dependent systems. Accountability is distributed: domain owners confirm meaning and validity; data stewards maintain rules and metadata; security defines protection and sanitisation; privacy and legal functions assess applicable duties; engineering enforces transitions; audit tests evidence.

Humans remain involved where meaning or exceptions must be decided. Automation handles deadlines, dependency analysis, blocking, re-indexing and evidence collection. It must not "solve" uncertainty through silent deletion or infinite retention.

Data and model lifecycles intersect

Training data, evaluation sets and knowledge bases shape models and measurements. With RAG, a withdrawal can often be handled through a controlled index update. Removing the influence of one source from a trained model is substantially harder. Decisions about training use therefore require stronger controls before ingestion.

Model versions should reference data snapshots, filters, transformation code and evaluation sets. A dataset should in turn reveal which models, indexes or reports used it. Change can then trigger impact analysis instead of only cleaning a source folder.

The lifecycle does not end at deployment. Drift, new failure modes and changed operating conditions can reduce the fitness of earlier data. Scheduled review should be complemented by events: quality decline, unusual retrieval results, new categories and professional complaints.

Test a lifecycle with scenarios

A diagram does not prove operation. Test real transitions:

1. An active policy is superseded. Does the previous version leave default retrieval?

2. A source is withdrawn. Which chunks, embeddings, summaries and answers are identified?

3. A review date passes. Is use blocked, limited or escalated?

4. A deletion request reaches production data, logs and backups. Is each system covered?

5. A backup is restored. Do previously disposed objects remain suppressed?

6. A legal_hold ends. Does the normal rule resume?

Measure transition time, failures, undetected dependencies, manual exceptions and evidence completeness. A strong lifecycle is not invisible because nothing ever fails. It is dependable because deviations are detected and controlled.

Data needs a lifecycle because knowledge is time-dependent. Purpose begins, validity ends, relations change and risk travels through derivation. Storage management organises a collection. State and transition design creates accountable information infrastructure.

Worksheet: Build a lifecycle matrix

Choose a dataset with at least five sources and one AI use case.

1. Define purpose and entry. Record the capture reason, permitted use, owner, protection class and required metadata.

2. Model states. Define at least eight states and permitted transitions, including roles and preconditions.

3. Map derivations. Connect original, processing, chunk, embedding, index, summary and output.

4. Plan review and retention. Define triggers, period start, exceptions, archive state and final action.

5. Simulate withdrawal. Withdraw one source and record every blocking, re-indexing, review and deletion action.

6. Test restoration. Simulate a backup restore and prevent reactivation of disposed objects.

7. Review evidence. Can an independent reviewer determine who triggered each transition and why?

All materials to download — the topic overview and the worksheet:

Scope: There is no universal lifecycle with identical stages or retention periods for all data. Legal duties, legitimate archival purposes, contractual requirements, evidentiary needs and technical capabilities vary. This article offers a professional and technical design model, not legal advice. Deleting an application record does not automatically sanitize media, backups, indexes or derived artefacts.