Human-in-the-loop needs decision boundaries

"A person checks it at the end" is not a control concept. Oversight becomes meaningful only when a person has enough information, time and authority to alter or stop a machine-supported outcome.

Human-in-the-loop sounds like a simple answer to automated-decision risk. A model analyses data, produces a proposal and a person approves the result. Responsibility appears to have been added to the process diagram.

In practice, that control can be empty. The reviewer may see only a traffic light, process hundreds of cases per hour, lack data provenance or have only a confirmation button. They may theoretically disagree but face a cumbersome exception path. Approval may occur after an irreversible action has already started. A human in the interface is not necessarily a human in control.

Oversight is a system capability

Effective oversight is not a role appended to the pipeline. It is a joint property of process, data, interface, permissions, organisation and monitoring. Five questions expose whether it is substantive:

1 · Knowledge: Can the person inspect inputs, sources, rules, uncertainty and alternatives?

2 · Competence: Can they assess professional, technical and, where relevant, legal consequences?

3 · Time: Is there enough time for review rather than reflexive confirmation?

4 · Authority: Can they modify, reject, escalate and stop the process?

5 · Effect: Does intervention occur before the consequential step and change what happens?

If one condition is missing, human involvement may become theatre. Nominal accountability without practical authority is especially dangerous: the person bears consequences although the system constrains or blocks meaningful action.

Define the boundary before the output

Not every case needs identical oversight. A reversible spelling correction differs from advice affecting credit, employment, treatment or educational access. Control depth follows potential harm, reversibility and context.

A decision boundary marks the point at which automation may not continue silently. Triggers can include high impact, weak reversibility, missing or contradictory evidence, sensitive data, vulnerable people, unfamiliar cases, distribution shift, conflict between rules and output, unclear authority, poor calibration or use beyond the permitted purpose.

A single confidence score is insufficient. Model probability may be miscalibrated and says nothing about corpus completeness, processing lawfulness or error severity. A defensible boundary combines multiple signals with an explicit risk class.

Five operating states, not one approval button

A controllable system distinguishes at least five states. Assist organises information without proposing the decision. Recommend presents an option for independent assessment. Approve requires active review before execution. Escalate transfers the case to a differently qualified or authorised role. Stop prevents further processing or effect.

Each state needs triggers, an owner, a maximum waiting time and permitted next actions. A stop with no reachable owner is incomplete; so is escalation without a deadline. The system also needs a safe default. Missing data, service failure or ambiguous ownership must not silently push the workflow in the riskiest direction.

Decision rights become explicit. Who may correct a proposal? Who can override a rule by exception? Who may pause but not reject? When is dual control required? Which changes require a new policy or model approval? Responsibility then becomes testable rather than rhetorical.

Give the reviewer a case file, not an assertion

Review is limited by what is visible. A useful case file contains the original request, data used, relevant source passages, proposed decision, applied rules, known gaps, plausible alternatives and consequences of each option. It separates facts, model output and derived interpretation.

Explanations must fit the task. A feature-importance chart does not establish that data are correct or processing is permitted. A long rationale can simulate certainty even when generated by the same model. Source passages and rule references are more inspectable than a model's linguistic self-justification.

Sequence matters. A persuasive recommendation shown first can anchor judgement. For critical tasks, independent initial assessment can help: the professional reviews core information before the model suggestion becomes visible. Other workflows benefit from counterevidence, alternatives or a requirement to record the decisive reason for approval and rejection.

Training alone does not remove automation bias

Automation bias is more than blind trust. People may miss a problem because the system raises no warning, or execute a bad recommendation because it looks authoritative. Research describes omission and commission errors among both inexperienced and expert users. Workload and rare exceptions create attentional problems.

Controls therefore belong in the workflow: independent samples, deliberately withheld recommendations, counterexamples, mandatory reasoning for critical releases, rotating reviewers, quality time rather than throughput-only targets and tests containing intentionally wrong advice.

Explanations are not a universal cure. Research on cognitive forcing shows that interactions which deliberately slow judgement can reduce overreliance, yet effort, acceptance and effectiveness vary by task and user. Good oversight does not optimise convenience alone. It evaluates appropriate reliance: accepting sound recommendations, detecting bad ones and escalating uncertainty.

Capacity is part of the safety architecture

An organisation can defeat a good interface with unrealistic performance targets. A reviewer expected to close several complex cases per minute becomes a confirmation mechanism. Expected review depth must match the time budget. Rare high-risk cases need reserve capacity, questions and specialist access.

Workload is planned like technical capacity. Monitor queues, review time, abandonment, escalation backlog and unexamined default approvals. If load rises, the system must prioritise, restrict functions or enter a safe mode rather than silently lowering quality.

Competence is case-specific. General AI literacy does not replace domain expertise. Reviewers need to know the use case, common model limits, relevant data failures and their authority. Multidisciplinary decisions may require a team with explicit handovers rather than one nominal reviewer.

Logs must reconstruct both decision and influence

An audit record should not merely say "human approved". It captures the model proposal, visible evidence, system version, decision, edits, rationale, time, role and executed action. Order matters: was independent judgement recorded before or after the recommendation appeared? Could the reviewer still stop the consequential step?

These records support learning. Where are suggestions often corrected? Which case types are escalated disproportionately? Which rationales are missing? Where is a stop bypassed? Monitoring must not become indiscriminate worker surveillance. Purpose limitation, access controls, retention rules and aggregated analysis belong in the design.

Human corrections must not flow unreviewed into training or the knowledge base. People also err, follow local habits and work under pressure. Feedback is qualified, conflict-checked, versioned and approved before it becomes a learning signal.

Oversight needs contestability

The affected person is not the same as the internal reviewer. Consequential decisions may require information, an opportunity to state a position and an accessible route to challenge the result. An internal click does not replace an intelligible explanation or independent reconsideration.

Legal regimes must remain distinct. Article 14 of the EU AI Act specifies human oversight for high-risk AI systems, including proportionate abilities to understand limitations, recognise automation bias, disregard or override output and stop a system. Application is staged and must be checked against the current legal timeline.

GDPR Article 22 instead concerns certain decisions based solely on automated processing that produce legal or similarly significant effects. Its exceptions and safeguards, including human intervention, are not synonymous with every human-in-the-loop design. A nominal signature does not automatically transform a substantially automated process into meaningful human judgement.

Evaluate the oversight itself

Model accuracy alone does not show whether the combined system is safer. Evaluation covers the human, model and process as a team. Useful measures include detected model errors, incorrectly overridden correct proposals, missed boundary cases, escalation precision, time to stop, post-review outcome quality, differences across groups and roles, reversal rates and rationale completeness.

Tests include correct, incorrect, overconfident, contradictory and unanswerable recommendations. They vary time pressure, information volume and case frequency. This reveals whether oversight works under realistic conditions or only in a demonstration.

Human-in-the-loop is therefore not a station but a control architecture. Its quality is not proven by a human click. It is proven when the system stops early enough, the right person receives decisive information and a reasoned intervention genuinely changes the outcome.

Worksheet: Design a real decision boundary

Choose an AI-supported process with meaningful consequences and complete eight steps.

1. Classify impact. Describe harm, affected people, reversibility and urgency.

2. Define states. Separate assist, recommend, approve, escalate and stop.

3. Specify boundary signals. Use evidence gaps, conflicts, sensitivity, novelty, uncertainty and purpose deviation.

4. Assign decision rights. Name role, authority, deputy and dual-control cases.

5. Build the case file. Show data, sources, rules, alternatives, uncertainty and consequences.

6. Test automation bias. Include wrong recommendations, missing warnings and time pressure.

7. Enable challenge. Define correction, escalation, affected-person information and contest.

8. Measure effectiveness. Set metrics, regression tests and a review cycle.

All materials to download — the topic overview and the worksheet:

Scope: Human-in-the-loop is neither one technical architecture nor a blanket legal requirement for every AI application. Article 14 of the AI Act concerns high-risk AI systems within its legal scope; GDPR Article 22 has a different and narrower test. This article presents a risk-based design pattern and does not replace assessment of the concrete system, use context or current transitional law.