The failure mode nobody plans for
Every responsible AI framework says the same thing: keep a human in the loop. It is good advice and it is almost always where the design thinking stops.
Because a gate does not fail loudly. Nobody removes it. Nobody announces that oversight has been abandoned. What happens is quieter: the queue grows, the approvals are almost always correct, the reviewer starts scanning rather than reading, and within a few months a person is clicking approve on forty items a day with a median dwell time of two seconds.
That is a log entry that looks like oversight, which is materially worse than having no gate at all, because the organisation now believes something is being checked, and the audit trail says so.
A rubber stamp does not just fail to catch errors. It launders them.
So the design question is not "should there be a human in the loop". It is: what would have to be true for this person to realistically say no?
Three conditions a real gate needs
1. Enough context to disagree
An approval screen that shows only the AI's output is asking the reviewer to rubber-stamp, because they have nothing to check it against. To disagree, a person needs to see what the system saw and why it concluded what it did.
In practice that means the source document alongside the extraction, with the specific fields the system was unsure about marked. The rule that fired, named. The prior decisions on similar cases. The thing that makes disagreement possible is the ability to compare, and comparison needs two things on screen.
2. Enough time, which means few enough items
A gate on every transaction in a high-volume process will degrade, not because the reviewer is careless, but because the arithmetic does not work. Forty approvals a day at a minute each is most of an hour, competing with the rest of that person's job.
The fix is almost always to gate less, not to ask for more diligence. Route only the items that warrant judgement: below a confidence threshold, above a value threshold, outside a policy boundary, or of a type with a known error history. A gate that catches ten items a day and is genuinely read beats one that catches four hundred and is not.
This feels like reducing oversight. It is increasing it, because oversight that happens is worth more than oversight that is nominal.
3. A reason to believe rejection matters
If rejecting an item sends it into a void, with no follow-up, no correction to the system and no visible consequence, reviewers learn that their judgement is decorative. Rejection has to do something: route to a named person, correct the record, and feed the case into the evaluation set so the same mistake becomes measurable.
The strongest signal you can send is showing reviewers that their rejections changed the system. It costs a line in a monthly summary and it is the difference between a gate people take seriously and one they do not.
Where gates belong, and where they don't
Gates are not free. Each one adds latency, consumes attention, and spends a limited budget of human patience. Spending them evenly across a process wastes them.
Gate these, essentially always:
- Money leaving the business, or a commitment to pay
- A binding commitment to a customer: a price, a delivery date, a contractual term
- Anything a regulator will see, particularly with a statutory deadline attached
- A decision affecting an individual's employment, enrolment, entitlement or care
- Anything irreversible
Do not gate these:
- Reading, classifying, extracting and summarising, where the output feeds a later gated step
- Internal drafts a person will obviously read before using
- Reversible system writes with a clean audit trail
- High-volume, low-consequence routing that has a measured accuracy record
The test is not "could this be wrong". Everything could be wrong. The test is what does being wrong cost, and can it be undone.
Designing the approval surface
Most of whether a gate survives is decided by the interface, and approval interfaces are consistently the least-designed part of an AI system.
What works:
- Source and output side by side. Always. The document on the left, what the system read on the right.
- Uncertainty marked. Highlight the two fields the system was unsure about rather than presenting fourteen fields uniformly. Directing attention is the single highest-value thing the interface does.
- A reason for the route. "Held: variance $412 exceeds the $250 tolerance." The reviewer starts from a diagnosis instead of a puzzle.
- Reject with a reason, from a short list. Free text is never analysed. Four or five categories become data, and the distribution tells you exactly where the system is weak.
- One item at a time. Bulk-approve buttons are how a gate dies. If the volume makes one-at-a-time impractical, the answer is to gate fewer items, not to add a select-all.
- A visible queue with an owner and an age. Nothing consequential should be able to sit unnoticed.
And one thing to avoid: pre-selecting approve as the default action. It sounds like a small convenience. It measurably shifts outcomes.
Measuring whether a gate is still real
A gate is a control, and controls need monitoring. Four numbers, reviewed monthly, tell you almost everything:
| Signal | What to watch for |
|---|---|
| Rejection rate | Trending toward zero over months: the classic rubber-stamp signature |
| Median dwell time | Falling below the time it physically takes to read the item |
| Queue age | Growing, which means the gate is under-resourced and about to degrade |
| Post-approval error rate | Errors caught downstream that the gate should have caught |
A rejection rate near zero is not automatically bad. It may mean the system is genuinely accurate. Read it alongside dwell time. High accuracy with unchanged dwell time is a good system. Falling rejections and falling dwell time together is a gate that has stopped functioning.
When the numbers say a gate has degraded, there are three honest responses: narrow what it catches so the volume is reviewable, invest in the interface so review is faster and better directed, or accept the evidence and move that action to a lower autonomy level. What is not an honest response is reminding staff to be more careful.
The thing this is actually for
Approval gates are usually justified in the language of risk. That is true and it is not the main benefit.
The main benefit is that a gate is how trust gets built on evidence rather than assertion. A system that starts at suggest-only, accumulates a few thousand reviewed decisions, and can show you its rejection rate and its error distribution has earned an argument for more autonomy. One that was deployed autonomously from day one has only a promise.
Which is why we design gates to be moved rather than to be permanent. The dial starts conservative, the evidence accumulates, and each action moves independently as its record supports it. Some actions reach full autonomy in a month. Some never should, and knowing which is which is the entire point of having built the gate properly in the first place.