Test, don't admire

A successful AI answer can feel remarkably convincing. That is precisely its risk: when it is fluent, fast and plausible, it is easy to mistake a strong first impression for reliability. Anyone who wants to work with AI needs less admiration and more small, repeatable tests.

Not every correct answer is a good test

One example tells you almost nothing about the quality of a workflow. Perhaps the task was unusually easy. Perhaps the available material happened to fit perfectly. Perhaps the answer sounds convincing while a decisive point is missing. A test begins only when you define in advance what counts as usable.

This is not an invitation to distrust for its own sake. It is a form of care. You are not testing whether a system is magical; you are testing whether it helps you usefully under clear conditions.

Three scenarios reveal what actually happens

The positive scenario describes the normal case: a clear task, suitable material and a result in an agreed format. Here you check whether the work is genuinely made easier. A good result is not simply long or elegant; it serves the agreed purpose.

The negative scenario sets a boundary. What should happen when the basis is missing, the request is unclear or a claim cannot be supported? A useful system may ask a question, mark uncertainty or refuse at that point. That is not a failure; it is often evidence that the boundary is visible.

The repairable scenario goes one step further. You intentionally supply a small error, a conflicting instruction or incomplete material. Then you test not only whether the problem is noticed, but whether the path to correction is clear: What is missing? What changed? What must a person decide?

Acceptance criteria make quality visible

An acceptance criterion is a short, observable statement. For example: “The response only names claims found in the supplied material.” Or: “When information is missing, it asks a question first.” These statements matter because two people can check them without arguing about a vague impression.

Keep criteria small. Three to five clear points are more useful than a long wish list. Be equally clear about what is not being tested. A test for a structured summary does not need to judge creativity, tone, research and legal interpretation at the same time.

An error log is not failure

When a test fails, the first question is not, “Why is the AI bad?” Ask instead: Was the task clear? Was the material sufficient? Was the rule understandable? Could the criterion actually be checked? Often the most useful insight is not in the answer but in an unclear hand-off.

Record errors as repair assignments. Describe the input, expected behaviour, actual behaviour and next change. A failed attempt then becomes a better system — not merely a memory that something once did not work.

An example: the invoice summary

Suppose an assistant should summarise incoming invoices into a short list: supplier, amount, due date. The positive test gives it three clean invoices. The criterion: every row names exactly those three fields, with no invented values.

The negative test gives it an invoice with no due date. The safe behaviour is not to invent a date but to mark the field as “missing”. The repairable test builds in a transposed digit in the amount — and you watch whether the assistant surfaces the discrepancy instead of smoothing it over.

Only these three runs together tell you something: the first shows the task succeeds in the normal case; the second shows the boundary holds; the third shows whether an error becomes a human moment or slips through quietly. A single elegant output would have revealed none of it.

The test card for an AI task

# TEST CARD

**Task**
The AI should …

**Positive scenario**
With these materials and this question, I expect …

**Negative scenario**
If … is missing or unclear, the AI should …

**Repairable scenario**
With this deliberate error, it should …

**Acceptance criteria**
1. …
2. …
3. …

**Next change**
After the test, I will adjust …

The best AI answer is not the one that impresses you fastest. It is the one whose limits you know, whose errors you can interpret and whose behaviour you can deliberately improve after a test.

Worksheet: Three tests

Choose a recurring task, such as a summary, a draft or a check. Before using it again, write three short scenarios and three acceptance criteria.

Define the positive case. Describe a normal, well-prepared request. What exactly should exist at the end?

Set a negative boundary. Describe a request with missing or unclear basis. What safe behaviour do you expect?

Build in a repair. Deliberately include a small contradiction or gap. How will you know it became visible and workable?

Check the criteria. Assess the result only against your three criteria. Then write one change for the next attempt.

Both working materials for this article — the topic overview and the worksheet with a reflection space to fill in — are available for download here: