Why AI agents need kernel-style governance, not more humans

Key Takeaways

• Human-in-the-loop review creates alert fatigue and operational bottlenecks at scale
• The shift from capabilities to responsibilities reframes agent design around what agents SHOULD do, not what they CAN do
• Governance by exception means humans design policy, runtime enforces it, and only edge cases require review

The human-in-the-loop safety model for AI agents is breaking down. When an agent flags a decision for review, a human approves it. Then another arrives. Then dozens more. The queue grows. The human starts clicking through without reading the JSON payloads. They hit 'Approve' because the backlog is piling up and nothing has exploded yet.

That's the core argument from Mike Loukides at O'Reilly Media, writing on the company's Radar blog. His latest piece, "From Capabilities to Responsibilities," makes the case that enterprise AI agents need a different governance model entirely. The current approach works fine for development environments and low-frequency pipelines. In production systems running dozens of agents making hundreds of decisions per hour, it collapses into what Loukides calls the Scalability Trap.

What breaks when humans can't keep up?

The failure mode is simple and familiar to anyone who's worked in security operations. Alert fatigue turns governance into throughput management. Reviewers stop evaluating and start processing. The approval checkbox becomes a formality.

Loukides is explicit that this isn't about human weakness. It's technical debt in the governance layer, created by routing too many binary decisions through a manual queue. The article cites Tyler Akidau's observation that the industry invested heavily in agent capability while neglecting the infrastructure for authority, constraint, and accountability.

The target audience here is narrow. This isn't about RAG chatbots or research copilots that only retrieve and summarize. Loukides is focused on high-stakes agentic systems, ones that mutate external state by moving money, changing infrastructure, or modifying critical records.

How does responsibility-based design differ from capability-based?

The dominant framing in enterprise AI asks: what can this agent do? What tools does it have? What APIs can it call? Loukides argues this is the wrong question entirely for production systems.

A responsibility statement bounds what an agent is permitted to do, not what it's technically capable of doing. The distinction maps to how organizations already handle human roles. You can't dictate how a person thinks, but you can strictly define what they're authorized to execute.

The article walks through concrete examples across domains:

• Finance: A capability is 'can execute equity trades.' A responsibility is 'authorized for up to $50,000 per order, highly liquid equities only, maximum 2% daily drawdown.'
• Healthcare: A capability is 'can reschedule appointments.' A responsibility is 'authorized to rebook non-critical outpatient visits within 14 days, no specialist double-booking.'
• Supply Chain: A capability is 'can reroute freight.' A responsibility is 'authorized to redirect non-hazardous cargo up to $5,000 SLA penalty budget.'

The gap between these two statements is the gap between a demo and a production deployment.

Why prompts aren't a sufficient control surface

The current approach often handles this gap with prompts. Give the LLM an API key, tell it to 'be careful with position sizing,' and hope alignment holds under adversarial inputs and edge cases. In low-risk contexts, that may be tolerable. For systems touching money or medical records, it isn't.

Loukides proposes borrowing from operating system design. A privileged Kernel Space validates every proposed action before it touches the real world. Humans design policy, the runtime enforces it deterministically, and only genuinely exceptional cases escalate to human review.

This builds on concepts from his previous article, "The Missing Layer in Agentic AI": idempotency to ensure actions can be safely retried, just-in-time state verification to confirm the world hasn't changed since the agent made its decision, and deterministic function ID tracking for audit trails.

What does governance by exception look like in practice?

The scalable alternative Loukides advocates is Governance by Exception. The system enforces policy at machine speed. Humans intervene only when actions fall outside defined boundaries or when the system genuinely cannot determine the right path.

This isn't a call to remove humans from the loop. It's a call to move them to a different layer. Policy design, boundary definition, and exception handling are human work. Real-time approval of routine actions at scale is not.

The article references Carl Hewitt's Actor model from distributed systems as prior art. These problems aren't new. Privilege separation and capability control were solved decades ago in operating system design. The question is whether the AI industry will learn from that history or reinvent it poorly.

Frequently asked questions

Frequently Asked Questions

What is the main problem with human-in-the-loop AI agent oversight?

At production scale with dozens of agents making hundreds of decisions per hour, human reviewers experience alert fatigue. They stop evaluating decisions carefully and start clicking through approvals to clear the queue, which defeats the purpose of the safety mechanism.

What is governance by exception for AI agents?

A model where humans design policies and boundaries, the runtime enforces them automatically at machine speed, and only genuinely exceptional cases that fall outside defined parameters escalate to human review.

What's the difference between AI agent capabilities and responsibilities?

Capabilities describe what an agent can technically do (execute trades, reschedule appointments). Responsibilities define what it's authorized to do within specific constraints (trade up to $50K in liquid equities with 2% max drawdown).

Which types of AI systems does this governance model apply to?

High-stakes agentic systems that mutate external state: moving money, changing infrastructure, or modifying critical records. Not RAG chatbots or research assistants that only retrieve and summarize information.

Why are prompts insufficient for controlling high-stakes AI agents?

Prompts asking an agent to 'be careful' rely on alignment holding under adversarial inputs, unusual conditions, and edge cases. For systems touching money or medical records, deterministic policy enforcement is a more reliable control surface.

Source: Stack Overflow Blog