Autonomy needs an off switch, and a ceiling

Everyone is racing to ship agents that act without asking. Almost nobody is racing to ship the controls that make that acceptable. That is backwards, because the agent that acts is the easy part. A model with tools and a loop is the quick part. The slow part is everything that lets you say no.

So I audited Team-X, my own project, against that standard at release v3.4.0, and I am going to tell you where it holds and where it does not.

Team-X, an open-source, local-first desktop app for running AI-agent organizations, treats autonomy as a grant the operator can revoke. At v3.4.0 that grant has five controls: an arm switch, monthly dollar caps, an approval inbox, a pre-flight doctor, and an append-only event log. Some are deep. One is mostly a facade, and I will name it.

What does an operator need before letting an agent act?

Five things, and regulators and engineers keep arriving at the same list.

An off switch comes first. The EU AI Act’s human oversight article requires that a person can “interrupt the system through a ‘stop’ button or a similar procedure that allows the system to come to a halt in a safe state”, per Article 14 of Regulation (EU) 2024/1689. That text governs high-risk systems, and most agent apps are not those. The engineering principle applies anyway.

Cost comes second. Anthropic’s guidance on agents is blunt: “The autonomous nature of agents means higher costs, and the potential for compounding errors. We recommend extensive testing in sandboxed environments, along with the appropriate guardrails.”

Approval comes third. The OWASP entry on excessive agency lists the control directly: “Utilise human-in-the-loop control to require a human to approve high-impact actions before they are taken.”

Visibility closes the list. A 2024 paper on governing agents names three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging. Of those three, the one Team-X implements is the logging.

The failure mode is not hypothetical. In July 2025, The Register reported that a founder claimed an AI coding tool “deleted a database despite his instructions not to change any code without permission”. That is his account, and it is a claim, not an adjudicated finding. But it shows what an instruction is worth without enforcement: nothing.

The NIST AI Risk Management Framework frames the organizational version. Its Govern 3.2 outcome asks that “Policies and procedures are in place to define and differentiate roles and responsibilities for human-AI configurations and oversight of AI systems.”

Where is the off switch, and what does it actually stop?

It is a real switch, persisted, and off by default. It is also wired to less than the screen suggests.

The Proactive Mode widget landed on the Mission Control dashboard in v3.4.0. The changelog calls it a “previously backend-only surface, now operable from the flagship view”. The widget is mounted in the dashboard and when disarmed it says “Proactive mode is disabled. Agents will only respond to direct commands.”

The flag defaults to false in the settings repo. Both proactive actions, the scan and goal decomposition, check it before doing anything. Disarmed, the scan returns zero queued items and goal decomposition returns without starting a run.

Now the facade. The widget shows Active Work, Queued Work, and Last Scan tiles, polled every five seconds. The handler behind them returns this, verbatim:

      return {
        enabled,
        activeWork: 0, // TODO: track active work count
        queuedWork: 0, // TODO: track queued work count
        lastScanAt: null, // TODO: track last scan timestamp
      };

At v3.4.0 those tiles read zero, zero, and Never, whatever the system is doing (handlers.ts). The switch is real. The gauges are placeholders. I shipped a dashboard that implied telemetry I had not written.

What does “autonomy level” mean in code?

Less than the label implies. It is a stored string with three values, and the proactive service consults it in one place.

The type is 'balanced' | 'conservative' | 'autonomous', default balanced. In the proactive trigger service, the only branch on it is that conservative blocks goal decomposition, with the explanation “Conservative autonomy mode requires explicit approval for goal decomposition”. Balanced and autonomous behave identically there.

The widget shows the mode as a read-only tag. It is a readout, not a control.

How often does the proactive scan run?

Only when you press the button. I went looking for a timer and found none.

The scan has two callers in the renderer: the dashboard button and a button in settings. The picture of armed, idle employees going to look for their own next ticket is not what this code does. The scan lists open, unassigned tickets, and for each one picks the first non-system, non-officer employee, whether or not that person is idle.

The code carries its own admission. The thread and message ids it queues are placeholders, per the comment “In production, this would use the threadsRepo to create/get thread”. A separate dispatcher that does create threads and checks budgets exists in the orchestrator folder, but v3.4.0 only exports it. Nothing constructs it. I call that an orphan.

What does run on a clock is the routine service, which polls every 60 seconds per company. Routines create tickets and wake the assignee, and they pass through budget admission first.

What does a dollar cap block, and where is it checked?

It blocks admission of new work, at the start of a run. This is the deepest control in the set.

A policy has a scope, a monthly hard cap in dollars, a warning threshold, an optional approval threshold, and an auto-pause flag. The scopes are company, employee, runtime profile, and routine. The only period is monthly.

Setting	Value at v3.4.0
Scopes	company, employee, runtime-profile, routine
Period	monthly only
Warning threshold default	80 percent
Auto-pause default	false
No policy configured	execution allowed

The check lives in assertExecutionAllowed. Once spend reaches the cap, it returns a refusal with the reason “Budget cap reached for” the scope. It then pauses the company only if the policy has auto-pause on, and the default is off.

Callers include the orchestrator before each agent turn (line 1742), the agentic loop, routines, and the copilot analyzer. The loop’s own caps are covered in Eight turns, 8,000 tokens, 120 seconds. A blocked turn records a checkpoint and throws “Execution is blocked by budget policy before the run can start.”

Be exact about the limit. Spend is written to the ledger after a run ends, by recordRunSpend, which skips runs still in flight. A cap stops the next run, not the current one. One expensive run can cross it.

The approval threshold adds a middle state. Above it and below the hard cap, execution waits on a budget exception in the inbox. Approve it and runs proceed for that month. Deny or dismiss it and they stay blocked.

Which decisions reach the approval inbox?

Three kinds, though the type names eight. That gap is the honest number.

The enum lists budget exceptions, authority requests, delegation requests, and planner, runtime, routine, deliverable, and artifact items. The list function only assembles three: budget exceptions, delegation requests, and authority requests. I found no non-test code that creates the other five.

Delegation is the gate I trust most. When a manager agent calls delegate_subtask, the tool writes a pending row “that the operator must approve”. The ticket does not exist until a human approves it. That is the OWASP principle implemented at the tool layer, where an agent cannot talk its way around it.

What does the doctor check, and does it stop anything?

It runs eleven checks and stops nothing. It is a report.

The check list covers database integrity, migrations, backup posture, runtime profiles, secrets, sessions, ticket checkouts, workspace paths, MCP health, provider health, and budget blockers. Findings are warning or blocked. When a probe is not connected, it says so, as in “Migration table checks are not wired in this runtime.”

The doctor’s only caller is the IPC handler behind the Doctor tab. Nothing consults it before dispatch. Its budget finding tells you to review budgets “before launching more runtime work”, and then trusts you to.

The benchmark service is more modest still. Its mode constant is control-plane-simulated. It exercises the control plane, not live agents. The self-improvement loop creates tickets from failure signals, but it opens them unassigned as the system reporter. It proposes work; it does not run it.

Where is the audit trail?

In an append-only events table, and it is the most solid control of the five.

Every event the bus emits is persisted before it fans out, so a listener never sees something the log does not hold. The events repo exposes append and read methods and no update or delete. That is append-only by API, not by a database trigger, and I will not claim more.

Budget warnings, budget caps, approval requests, and proactive toggles all emit into it. Each row carries an actor, a kind, and a JSON payload.

Control	Maturity at v3.4.0
Arm and disarm	Real, persisted, off by default
Proactive status tiles	Placeholders
Autonomy mode	Stored value, one branch
Dollar caps	Enforced at run admission
Approval inbox	Three kinds live of eight named
Doctor	Eleven checks, report only
Audit log	Append-only by API

What should you do with this?

Audit your own agents the same way. None of these steps needs new tooling.

1. Find the off switch. If it is not on the first screen you use, or it is not persisted, you do not have one.
2. Set a company-level monthly cap before you arm anything, and remember that no policy means no limit.
3. Read your approval list in code, not in the UI, and count the kinds that can actually be created.
4. Treat any status tile as unproven until you have changed the underlying state and watched it move.
5. Do not treat a pre-flight report as a gate until something refuses to start because of it.

The gaps above are boring to fix, and I would rather you read them here than discover them later. The code is in the repository, and the autonomy control plane docs cover each panel.