We often treat human oversight as the safety net for AI.

Let the AI prepare the recommendation and propose the action, and then put a human in the loop to approve it.

Problem solved.

Except there is one uncomfortable question:

What happens when the human stops reviewing and just starts clicking?

As AI agents take more actions, the number of decisions requiring human approval can grow quickly. A person who receives one unusual request may examine it carefully. A person who receives hundreds of similar requests every day may not.

At some point, “human in the loop” can become human just clicking approve.

Human oversight is not automatically meaningful oversight.

The assumption sounds reasonable: humans provide judgement, context, and accountability that AI lacks. But simply adding a person to a workflow does not guarantee any of those things.

NIST has cautioned against assuming that a human overseer will automatically provide effective governance. Human decision-makers bring their own cognitive biases, and the effectiveness of different human-AI configurations depends heavily on how roles and responsibilities are designed.

Research also shows how easily people can over-rely on automated recommendations.

In one experiment involving 292 participants, people followed algorithmic recommendations more closely than equivalent human recommendations. More importantly, when participants were able to monitor and adjust the automated recommendation, they were less likely to intervene when the algorithm made its largest errors. The researchers concluded that adding a human in the loop increased uptake of automation while decreasing decision accuracy in their experiment.

That is automation bias: the tendency to rely too heavily on automated output.

The EU AI Act explicitly recognizes this risk. For high-risk AI systems, Article 14 says people responsible for human oversight should be able to understand system limitations, correctly interpret outputs, override or reverse them, and remain aware of the tendency to over-rely on AI-generated recommendations.

In other words, the regulator does not consider the mere presence of a human sufficient. The human must actually be capable of challenging the system. In this sense, approval does not always mean a judgement has been made.

Imagine an AI agent handling procurement.

Most transactions are routine. The agent prepares an action and asks an employee to approve it.

Approve.
Approve.
Approve.
Approve.

After hundreds of correct recommendations, another approval appears.

How carefully does the employee review number 237?

This problem becomes even harder with autonomous agents. A traditional application may require occasional approval. An agent can potentially perform hundreds or thousands of actions as it works toward a goal.

OpenID contributors have raised precisely this scalability concern: per-action approval can lead to consent fatigue and reflexive approval, undermining the control it was supposed to provide.

So perhaps the question should not be on where we can insert human approval, but where does human judgement genuinely add value? Maybe humans should manage boundaries, not clicks.

Meaningful human oversight may require a different model.

Instead of asking people to approve every routine action, organizations can define in advance what the agent is allowed to do independently and what requires escalation.

For example:

An agent may process standard transactions below a certain risk threshold.

But an unusual payment, policy exception, sensitive data access, or action above a defined financial threshold may require human review. The human then focuses attention where judgement matters rather than becoming a bottleneck for thousands of predictable decisions.

NIST’s recent work on post-deployment AI monitoring identifies scaling human-driven monitoring as an open challenge and asks how oversight should vary according to risk and use case.

That may be the more important principle.

Human oversight should be designed around risk, not around the comforting presence of an “approve” button.

Test the human in the loop, too

Before declaring a process safe because it includes human approval, ask:

  • How many decisions does this person need to review?
  • Do they have enough information to challenge the AI?
  • Do they understand what they are accountable for?
  • How often do they override the recommendation?
  • Would they recognize an unusual or dangerous request?
  • Are we measuring whether the oversight actually works?

Human oversight can be extremely valuable, but only when the human is actually overseeing.

Because, ultimately, saying “a human approved it” and “a human meaningfully reviewed it” are not the same thing.