Reproducible AI Projects

A good AI result can look convincing and still be methodologically weak.

Nobody may be able to explain which sources were used. The decisive prompt may have been overwritten several times. A model switch may have happened halfway through the project. An agent may have called a tool, but its side effect was never recorded. An architecture decision may be visible in the current code while the reason for rejecting the alternative has disappeared. A reviewer may have approved one version that was changed afterwards.

In the end, the result exists — but there is no robust history of how it came into existence.

That is where the reproducibility problem begins.

Classic software projects already rely on version control, build logs, tests, and tickets. Agentic AI projects need more. They also produce prompt versions, model calls, tool interactions, handoffs, dynamic decisions, retrieval results, evaluations, and sometimes non-deterministic outputs.

The professional question is therefore not only:

Does the result work?

It is also:

Can we later demonstrate how this project state came into existence, what evidence supported it, and which decisions shaped it?

Reproducibility begins where memory ends

As long as a project is small, memory works surprisingly well.

You still remember why one variant was rejected. You remember the decisive prompt. The conversation is still open. The source files are visible next to it. Changes are recent.

As project duration grows, this model fails.

People forget. Conversations change. Models are updated. Files receive new versions. Agents create intermediate outputs that later become invisible. New contributors join without knowing why an existing decision was made.

A dangerous illusion appears: the current state looks unambiguous even though its origin is no longer unambiguous.

Reproducibility is therefore not primarily an archive problem.

It is a property of project architecture.

A result is not evidence of how it was produced

A finished presentation does not prove how it was created.

A working prototype does not prove which assumptions were tested.

A final prompt does not prove which earlier variants failed.

A Git commit does not automatically prove why a decision was made.

And a conversation history does not prove which files, tool states, and external data were relevant at the moment a response was generated.

A reproducible AI project therefore needs several connected evidence layers.

Only together do these layers form a robust project history.

Prompt history is necessary — but not sufficient

In AI projects, reproducibility is often reduced to “save the prompt.”

That is a useful start. It is not enough.

An identical prompt can still produce a different result when at least one of these conditions changes: model or model version, system instructions, temperature or other sampling parameters, retrieval content, tool versions, external API responses, order of parallel steps, previous session state, files or datasets, or time-dependent web content.

The prompt is only one part of the experimental setup.

If you archive only prompts, you archive the question — not necessarily the conditions under which the answer was produced.

A conversation history is not an audit trail

A conversation is excellent for exploration.

As the only evidence source, it is weak.

Why?

It mixes tasking, discussion, correction, decision, and output. It may not contain every tool state. Some systems compress older content. Files can be replaced. External pages change. An agent can create side effects outside the conversation. Platforms do not always expose every instruction layer.

An audit trail needs structured events instead.

For example: RUN-028-014 started on 28 August at 14:32, used prompt version P-12.3, worked on source snapshot SRC-07, used runtime configuration M-04, produced artifact A-19.2, received a finding in review R-44, led to decision D-31, and was superseded by RUN-028-016.

That turns chronology into provenance.

Observability is not the same as provenance

Observability primarily answers an operational question: What is the system doing, or what did it technically do?

Provenance answers an origin question: Why does this state exist, and which inputs, decisions, and checks produced it?

A trace may show that an agent called three tools and created two files. Without a Decision Record, it may still be unclear why variant A was accepted and variant B rejected. Conversely, a good Decision Record may exist without making the actual execution technically reconstructable.

Professional projects therefore need both:

Observability for runtime events, failures, tool calls, and system behavior.

Provenance for origin, rationale, versions, and evidence relationships.

This distinction prevents a common mistake: many logs do not automatically make a project reproducible. They only make more events visible. Reproducibility emerges when relevant events are connected to sources, decisions, and artifacts.

Project history is a graph, not a story

People tell project stories linearly:

First we researched. Then we built a draft. Then we changed the architecture. Finally we reached version 3.

Technically, history is rarely linear.

An artifact may derive from multiple sources. A single decision may influence three runs. A review may reopen an earlier decision. Parallel agents may produce competing variants. A later test may invalidate an earlier assumption.

A provenance graph is therefore more useful than a timeline alone.

Its most important node types are:

Source — source, dataset, reference.

Instruction — prompt, system rule, task contract.

Run — concrete execution.

Artifact — file, code, analysis, or output.

Finding — test or review finding.

Decision — accepted, rejected, or superseded decision.

Checkpoint — confirmed project state.

Edges describe relations such as USES, PRODUCES, REVIEWS, SUPPORTS, REJECTS, SUPERSEDES, and DERIVES_FROM.

The central question changes from “What happened when?” to:

Which evidence supports which decision — and which artifacts depend on it?

The Minimal Run Record

Not every project needs a sophisticated observability platform.

But every relevant AI run should leave a minimal structured record.

The record does not need to duplicate every raw input.

It needs to reference them unambiguously.

Model identity belongs in the evidence

“We used GPT” is not a reproducible statement.

Even “we used model X” may be too vague when a provider changes behavior behind the scenes.

For relevant runs, capture — as far as available and appropriate: provider, model name, model or API version, date, relevant sampling settings, tool configuration, enabled retrieval sources, structured-output schema, and special runtime options.

Not every provider exposes every detail.

That is exactly why a project should record what was known and what was not.

Hidden model changes are not a reason to abandon reproducibility. They are a reason to document uncertainty explicitly.

Reproducibility in generative AI does not mean byte identity

A common mistake is to expect the same prompt to generate exactly the same text later.

With generative systems, that is often neither realistic nor necessary.

Sampling, non-deterministic hardware behavior, changing retrieval results, and provider updates can all create variation.

It helps to separate three goals.

Replay

Can the same stored inputs, instructions, and tool steps be executed again?

Rebuild

Can a new run recreate the same functional or project state from the documented prerequisites?

Re-evaluation

Can an independent reviewer judge whether the original decision was reasonable under the documented conditions?

For many AI projects, re-evaluation is more important than reproducing the exact prose.

Decisions need their own data type

A decision should not exist only as a sentence inside a conversation.

It should be a first-class project object.

A useful Decision Record contains at least:

This prevents orphaned decisions: project states whose origin can no longer be found.

An Activity Log and a Decision Log are not the same

An Activity Log says: the agent opened a file, a tool was called, a test ran, a draft was saved.

A Decision Log says: variant B is rejected because test T-14 violates a core requirement; architecture A is accepted despite higher cost because reversibility and data control are stronger; version 3.1 will not ship until finding F-22 is closed.

Both are useful.

But they answer different questions.

Activity explains movement. Decisions explain direction.

The most important link is: decision → evidence

A project can look extensively documented and still be weak.

That happens when decisions are recorded without their basis.

“We chose variant A” is documentation.

“We chose variant A because tests T-07 and T-09 met the acceptance criteria, source S-03 supported the critical assumption, and variant B failed the cost stress test” is evidence-backed reasoning.

Now the Decision Record is reviewable.

The decision is not merely present in the project.

It points to its evidence space.

Evidence density matters more than log volume

A project can store millions of log lines and still be difficult to understand.

The important property is not event volume but the density of meaningful connections. A good evidence set answers a concrete question with a small number of linked objects: Which assumption was tested? Which test ran? What result appeared? Which decision followed? Which artifact changed because of it?

This suggests a useful design rule:

Do not store every event with equal importance. Mark project-impacting events.

These include state changes, approvals, rejections, tool side effects, new source states, findings that were closed or reopened, and decisions that affect later work. Low-level debug detail can still exist, but it does not need the same retention and review priority.

This keeps provenance readable without weakening observability.

Sources need snapshots or at least identity

Web sources change.

Documents are updated. Pages disappear. API documentation receives new versions. Tables are corrected.

If a project later stores only a URL, nobody necessarily knows what the original run actually saw.

For critical sources, capture where possible: retrieval date, title and publisher, version number, archived snapshot when legally permitted, hash of local source files, or at least the relevant extracted section with provenance information.

The rule is simple:

A source is reproducibly referenced only when its identity at the relevant time is sufficiently determined.

Artifacts need lineage

A folder full of files is not version control.

final.docx, final_new.docx, and final_v2_really_final.docx are filenames — not provenance.

An artifact should know at least which version it is, which run produced it, which inputs were used, which decision allowed its release, which earlier version it replaces, and whether it is draft, reviewed, approved, or superseded.

For machine-readable artifacts, a hash can additionally identify the concrete file.

That turns “the file” into an addressable state.

Hashes prove identity — not quality

A cryptographic hash is useful because even a small file change produces a different value.

It can therefore verify whether the exact same file is present.

It does not tell you whether the file is correct, whether the source was trustworthy, whether the prompt was good, or whether the decision was sound.

Hashes solve the identity problem, not the quality problem.

That distinction matters.

Checkpoints do not freeze a process — they stabilize it

Long AI projects need moments where a state is deliberately confirmed.

A checkpoint may contain the current project baseline, active Decision Records, approved artifact versions, open findings, relevant source states, current runtime and tool versions, and the next allowed work area.

A checkpoint does not mean “nothing may change after this.”

It means:

If we later need to reconstruct where the project stood at this point, this is the confirmed reference.

A checkpoint needs a fingerprint

A robust checkpoint should be uniquely addressable.

That can be a version number, commit, manifest, or a combination of hashes.

For example: CP-28-04 points to Charter v5, Decision Register v8, Source Manifest v6, Artifact Set v12, Open Findings v3, Runtime Manifest v4, and one concrete Git commit.

This creates a compact state address.

Prompt diffs are more valuable than prompt collections

A prompt library shows which prompts exist.

A prompt diff shows why a prompt changed.

That is far more useful for reproducibility.

For example: version 4 caused source mixing, version 5 added separation between primary and secondary sources, version 6 introduced a hard output schema, and version 7 added uncertainty labels after a reviewer finding.

Prompt engineering becomes a versioned method history, not merely a collection of clever texts.

Tool calls belong to project history

Agentic systems fundamentally change the evidence problem.

A model no longer produces text only. It can write files, modify databases, call APIs, create tickets, execute code, change external systems, delegate to other agents, or request human approval.

A useful trace does not need to store every internal token movement.

It needs to expose project-impacting events.

OpenAI, for example, describes traces as end-to-end records of agent runs with spans for generations, tool calls, handoffs, guardrails, and custom events.^[1] The principle matters more than the vendor: complex agent runs require structured events, not only a final response.