←
AI-Ready Foundation
Visionary · M4 · lesson 4 of 5 · queued
Preview — browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll →
Governed Autonomy: Multi-Agent Systems, Agent Identity, and Kill Switches
📖
now learning

Governed Autonomy: Multi-Agent Systems, Agent Identity, and Kill Switches

15 min

Attackers ran most of an espionage campaign through an AI agent, with only about 4 to 6 human decision points per campaign, by Anthropic's own assessment published on 13 November 2025. Your own agents are starting to hand work to each other, and most organizations still cannot say how many agents they run, who owns each one, what each may spend, or how many minutes it would take to stop one. Agentic AI governance closes that gap: every agent gets its own identity and a named owner, an explicit ceiling on what it may do and spend alone, approval points where the stakes are high, and an off-switch that has actually been tested. This lesson teaches the five levels of agent autonomy, why a prompt instruction is not a control, and an eight-step, half-day workshop that ends with a live kill-switch drill and a finished Agent Register and Autonomy Policy.

The Agentic AI Governance Problem: Authority Without an Off-Switch

What did Anthropic's November 2025 espionage disclosure show?

It showed that an agent can carry out most of a hostile operation while each step looks harmless. On 13 November 2025, Anthropic disclosed a campaign it had detected in mid-September 2025 and attributed, in its own assessment and "with high confidence", to a Chinese state-sponsored group. The attackers used an agentic coding tool against roughly 30 targets. Anthropic estimates the AI did 80 to 90% of the work.

How did they get past the safeguards? They split the work into small, innocent-looking tasks and posed as a security tester. Read alone, each request looked routine. Only the sequence was hostile. The model also hallucinated credentials and overstated its findings.

Three lessons transfer to your own agents:

  • A review of each call passes what a review of the session would stop. Splitting a task defeats per-request checks, so audit the whole session.
  • Few human decision points means the controls live in the plumbing. Only permissions, ceilings and logs watch the steps no person sees.
  • The agent's account of itself is not evidence. A model that invents credentials can misreport what it changed, so logs, rollback and the kill switch must sit outside the agent, in systems it cannot edit.

You have met the permissions side before. In Lesson 17 on agent tools, MCP permissions and human approval, a coding agent at Replit deleted a production database during a freeze it had been told to respect, and every fix that followed changed what the agent could reach rather than what it was told. Together the two cases give you the rule this lesson rests on: a rule written in a prompt is a request; a missing permission is a control.

What does the same gap look like inside a normal company?

Illustrative, but you will recognise it. An operations agent reconciles stock overnight under a script's old service account that can still write to production, and its only rule about live data is a sentence in its prompt. Three teams built agents on three platforms, so nobody can say which agents exist, who owns each one, what each may do or spend alone, or how fast one could be stopped. Ask the service desk to stop that agent now, and the honest answer is "we would have to find out".

How big is the gap between agent ambition and agent control?

Large: every survey finds plans racing ahead of controls. The figures are self-reported by executives, so read them as a direction of travel.

  • The headline gap. 42% of companies expect to give agents real decision authority by 2030, while only 5% have the full set of controls today, according to BCG's press release of 30 September 2026 (1,330 CxOs and senior leaders).
  • Adoption is real. 40% of respondents from large organizations report scaling AI agents, up from 27% in 2025 (McKinsey, The State of AI, August 2026).
  • Governance trails plans. Close to three-quarters of companies plan to deploy agentic AI within two years, but only 21% of those report a mature agent-governance model (Deloitte, State of AI in the Enterprise, 21 January 2026).
  • Control separates leaders. 31% of organizations say they are fully equipped to control and secure AI agents, against 75% of the most AI-ready group (Cisco AI Readiness Index, 14 October 2025; a vendor survey).
  • The downside forecast. Gartner predicted in a press release on 25 June 2025 that more than 40% of agentic AI projects will be canceled by the end of 2027, because of escalating costs, unclear business value or inadequate risk controls. It is a prediction, and two of its three reasons are governance problems you can design against.

Put the BCG pair on the first slide of your steering committee pack. Forty-two against five is the whole problem: many firms intend to let software make decisions, and almost none have finished building the brakes.

Why do capable teams get agent governance wrong?

Rarely through carelessness. Each of these six patterns is a locally sensible choice.

  1. Instructions are treated as controls. "Never refund more than 500 euros" in a prompt is a hope. A payment tool that rejects any call above 500 euros is a policy.
  2. Capability is confused with autonomy. Teams assume a smarter model has earned more freedom. It has not.
  3. Agents inherit a human's permissions. The builder connects the agent with their own credentials, and convenience at build time becomes blast radius at run time. Blast radius means everything an agent could damage if it went wrong, set by its access, not its intentions.
  4. Checks happen per request, not per session. A task split into innocent-looking pieces passes every check, as in the espionage case.
  5. Budgets are set per month, not per agent. Finance sees the bill when the invoice lands, so a looping agent spends for hours unseen.
  6. "Agent washing" hides what you bought. The same Gartner release estimated that only about 130 of the thousands of vendors marketing "agentic" products are real. If you cannot tell whether a product acts on its own, you govern a label.

Where this lesson sits on the ladder

This is Level 5, AI Native, lesson 2 of 5. In Lesson 17 you built an Agent Permission and Approval Matrix for one agent, and in Lesson 18 you gave AI systems traces you can replay. The previous lesson, Lesson 21 on AI-native process redesign, left you with a Process Redesign Charter, and that redesign usually produces several agents passing work along a chain. Lesson 17 governed one agent; this lesson governs the fleet: identity, ceilings on action and spend, hand-offs and the off-switch. The next lesson, Lesson 23 on the data flywheel, turns the approvals and overrides you start logging today into feedback data that improves the system every release.

How Governed Autonomy Works: The Hotel Key Card

Think about how a good hotel treats a new housekeeper. Nobody hands them the master key with a note saying "please stay out of the safe room". They get a key card with their own name on it. It opens floors three and four, between 8 in the morning and 4 in the afternoon, and can sign for supplies at the stores counter up to a set amount per shift. Every door it opens is logged against that name. If something goes wrong, the front desk cancels the card in seconds, and no lock has to change. Nobody argues about intentions: the card simply does not open the safe room.

  • The named card is the agent's own identity.
  • The floors and hours are its scoped credentials and tool allow-lists.
  • The signing limit is its spend and call ceiling.
  • The door log is the action log tied to that identity.
  • The front desk is the kill switch, run by people, outside the agent.
  • The manager who signs off the card is the agent's named human owner.

Every control in the workshop is one of those six things, and together they form a working AI agent governance framework.

What is governed autonomy?

Governed autonomy means every agent has an explicit ceiling on what it may do without a person, set by how reversible and how valuable each action is, and enforced by permissions rather than wording. The ceiling is set per task, not per agent, because one agent can do a harmless thing and a dangerous thing in the same afternoon.

What is AI agent identity?

AI agent identity means each agent has its own directory identity, a named human owner, least-privilege credentials, a lifecycle (creation, review, retirement) and logs that tie every action to it. Least privilege means the agent holds only the access its current tasks need. Without its own identity, an agent cannot be limited precisely, traced, metered for spend, or stopped without stopping whatever shares its account.

What is an AI agent kill switch?

A kill switch is a tested way to revoke an agent's credentials, pause its work queues and restore any state it changed, with a measured time to do so. A kill switch nobody has drilled is a theory; the only honest unit for it is minutes on a stopwatch.

What is a cost kill switch for an agent fleet?

A cost kill switch is a spend ceiling and a call ceiling per agent, enforced outside the agent, that suspends it automatically when either is reached. It is the financial twin of the safety kill switch. Fleets need it more than single assistants, because agents loop, retry and fan work out: if each of three chained agents retries a failed call five times, the model behind the last one can be called 125 times for one task (5 × 5 × 5, illustrative arithmetic).

Mandiant's September 2026 AI risk and resilience report describes an accounting agent stuck in a reasoning loop that made over 15,000 high-cost API calls in under an hour, running up about USD 50,000 in cloud charges; Lesson 14 on AI FinOps and controlling the AI bill takes that case apart. The OWASP Top 10 for LLM Applications 2026 (August 2026) names the risk Unbounded Consumption (LLM06:2026).

Lesson 14 set budgets per use case. At Level 5 the budget moves down to the agent: agent fleet cost limits, four numbers per agent, enforced at the gateway that issues its keys.

  • Calls per task: the most model or tool calls one task may make before it stops and asks a person.
  • Calls per hour: a rate ceiling that catches a loop in minutes, not at month-end.
  • Spend per day: a currency ceiling per agent, from its expected unit cost plus headroom.
  • Workflow ceiling: a total for all agents in one chain, so a fan-out cannot hide under each agent's own limit.

When a ceiling trips, the key is suspended, the queue paused and the owner paged: the safety kill switch, pressed by a meter.

What are the five levels of AI agent autonomy?

The clearest scale is "Levels of Autonomy for AI Agents" by Feng, McDonald and Zhang, published by the Knight First Amendment Institute (June 2025, revised July 2025). It defines each level by what the human does, not by how clever the agent is.

LevelHuman's roleWhat it looks like at work
L1OperatorThe user drives and the agent assists on demand.
L2CollaboratorPlanning and execution are shared; the user can take over at any point.
L3ConsultantThe agent leads and the user gives feedback, for example accepting or rejecting a proposed credit-limit change.
L4ApproverThe agent runs alone, and the user is needed only for blockers and consequential sign-offs.
L5ObserverThe user can only watch logs and use an emergency off-switch.

At L5 the off-switch is the only control a person still holds, so you build and drill it before any agent nears the top of the scale. The paper's central argument is that autonomy is a design decision, separate from capability. A highly capable agent can run at L2 because the action is dangerous; a modest one can run at L4 because the action is trivial and easy to undo. You choose the level. The model does not earn it.

What is an autonomy certificate, and why does it matter for change control?

An autonomy certificate is the Knight paper's proposal for a document, issued by a third party after evaluation, stating the maximum level an agent may operate at given its model, prompts, tools and environment. It is invalidated when the tools or environment change. Use the idea internally without a third party: an approved level belongs to one configuration, and a new model version, prompt or tool list lapses it until the agent is evaluated again. A new tool is a new set of doors on the key card.

What makes multi-agent systems harder to govern?

The hand-offs. Each agent can look correct alone while the chain does something nobody approved; the espionage campaign was exactly that, a sequence of innocent steps adding up to an attack. So multi-agent systems governance reviews the chain, not just the links. The constructive version is the Allianz claims chain from Lesson 21, where one of seven agents exists only to summarize the others' decisions for the claims professional who decides the payout (company-reported, November 2025). Build that audit summary in from day one and the sign-off stops being a rubber stamp.

Which threats should you model for agents?

Use the OWASP Top 10 for Agentic Applications for 2026, published by the OWASP GenAI Security Project on 9 December 2025, as your checklist:

IDThreatPlain question for your workshop
ASI01Agent Goal HijackCan content the agent reads change what it is trying to do?
ASI02Tool Misuse and ExploitationCan a legitimate tool be used for an illegitimate end?
ASI03Identity and Privilege AbuseDoes the agent hold more access than its tasks need, or borrow someone else's?
ASI04Agentic Supply Chain VulnerabilitiesWhich third-party tools, connectors or models could change under you?
ASI05Unexpected Code ExecutionCan the agent run code you did not expect it to run?
ASI06Memory and Context PoisoningCan bad data in its memory steer later decisions?
ASI07Insecure Inter-Agent CommunicationDoes agent B check who sent the instruction, or just obey it?
ASI08Cascading FailuresIf agent A goes wrong, how far does it travel down the chain?
ASI09Human-Agent Trust ExploitationWill a busy approver sign whatever the agent's summary says?
ASI10Rogue AgentsCould an agent keep acting outside its mandate without anyone noticing?

After the espionage case, ASI09 deserves a second look: approval points work only if the approver sees evidence from outside the agent.

Is there an official standard for AI agent identity yet?

No finished one, as of November 2026. On 5 February 2026 NIST's National Cybersecurity Center of Excellence published a concept paper on identity and authorization for software and AI agents, asking about identification, authorization, auditing, non-repudiation and prompt-injection controls; comments closed on 2 April 2026. Regulators are holding back too: in April 2026 the Federal Reserve, OCC and FDIC replaced SR 11-7 with SR 26-2, revised model risk guidance dated 17 April 2026, whose footnote places generative and agentic AI models outside its scope as "novel and rapidly evolving".

Vendor directories for agents exist, but a registry entry is not a governed identity, so test any platform for the pattern (own identity, named owner, scoped access, spend ceiling, lifecycle, logs), not the label. Nobody will hand you the bar; the workshop below is how you set it.

The Workshop: Build Your Agent Register and Autonomy Policy in Half a Day

About four hours with a short break, in eight steps, including a 30-minute live drill. You leave with a register entry for every agent in production or pilot, a maximum autonomy level per task, ceilings per agent, and a stopwatch time for stopping one.

Who needs to be in the room, and what should they bring?

Accountability splits four ways: security owns the controls, the AI platform lead the register, finance the money, and each business owner what their agent may do.

  • People: the CISO or a delegate, the identity and access management lead, the AI platform lead, the business owners of two or three live agents, an SRE who can actually revoke credentials, the FinOps lead or a finance controller, the change manager, and internal audit or risk as an observer.
  • Inputs: a list of agents and automations in production or pilot, their service accounts and permissions, three months of usage by key, your Process Redesign Charter from Lesson 21, any Agent Permission and Approval Matrix from Lesson 17, unit costs from Lesson 14, and last quarter's incidents.
  • Pre-work, one week ahead: the SRE and the change manager raise a standard change for the drill, naming a non-critical agent, the credentials to revoke, the restore step and a 30-minute window, so the change advisory board has approved it before anyone touches production.

The eight steps

  1. Inventory every agent (35 minutes). One row per agent: what it does, where it runs, which tools it can call. Give each its own identity, a named human owner and a review or expiry date. Anything on a shared account is flagged red on the spot. Ask each business owner, "what else in your area acts on its own?"
  2. Score each task for reversibility and value at risk (25 minutes). For every agent and task pair: can this be undone, and how quickly, and what is the most one action could cost? Band each: easily reversed, reversible with effort, irreversible; low, medium, high.
  3. Set the ceiling and enforce it structurally (40 minutes). Read the scores against the ceiling rule below and write the maximum level in the register; an owner may always choose lower. Name the mechanism that makes each ceiling true: scoped credentials, tool allow-lists, caps inside tools, separate development and production. Then back every rule written as prompt text with a permission, scope or cap, or delete it.
  4. Define approval points (20 minutes). List the consequential or irreversible actions that need sign-off, which role signs, and what that person sees: a short summary plus evidence from systems of record, not only the agent's own explanation.
  5. Log at the identity and at the session (20 minutes). Record every action against the agent's identity with inputs, tool calls and outcome, plus a human-readable audit summary. Decide who reviews whole sessions, how often, and which patterns trigger a closer look.
  6. Set spend and call ceilings per agent (25 minutes). With the FinOps lead, write the four numbers for each agent and workflow from expected unit cost plus headroom. Name where each is enforced and who is paged when it trips. An agent whose spend cannot be separated from other systems goes red, like a shared account.
  7. Drill the off-switch inside the approved change (30 minutes, live). In the approved window, with the agent's business owner on the call, have the SRE revoke the agent's credentials, pause its queues and restore one piece of state it changed while someone runs a stopwatch. Record the change reference with the time. Then walk every other agent on paper: who would stop it, and what else would stop with it.
  8. Threat-model the hand-offs and set re-certification triggers (35 minutes). Walk the OWASP list against your chain, with extra time on ASI01, ASI02, ASI03 and ASI08, and record the top three risks with an owner and a date. Then wire a rule into change management: any new model version, prompt or tool sends the agent back through evaluation.

How do you choose an agent's autonomy level?

Use reversibility and value at risk, not how impressive the agent is. This rule is a starting point to adapt, not an industry standard:

Action profileMaximum levelCondition
Easily reversed and low value at riskL4 ApproverLogged, sampled in session reviews, kill switch drilled
Reversible with effort, or medium valueL3 ConsultantAgent proposes, a named role decides
Irreversible or high valueL3, or L4 under a hard capA cap in the tool keeps each action small; anything above it needs human sign-off
Anything at L5 ObserverBy exception onlySigned by the business owner and the CISO, with a drilled off-switch and a tested restore

Worked example: three agents in order-to-cash

Illustrative only. Picture a 2,500-person wholesale distributor that built three agents for order-to-cash during its Lesson 21 redesign. Its register at the end of the session:

AgentOwnerIdentity todayTaskReversible?Value at riskMax levelStructural limitTime to revoke (drill)
Order-intake agentHead of customer serviceOwn identityParse orders, create draft ordersYesLowL4 ApproverCan write drafts only, cannot release3 min
Credit-check agentCredit managerShared finance service account (red)Recommend credit-limit changesYes, within a dayMediumL3 ConsultantRead-only on ledger, proposes via ticket41 min
Refund agentFinance operations leadOwn identityIssue refundsNo, once paidHigh above 500 eurosL4 below 500 euros, L2 abovePayment tool capped at 500 euros per call and 5,000 euros per day6 min

Look at the refund agent first. Its prompt used to say "refunds over 500 euros need approval". In step 3 that rule moved into the payment tool, so a 501-euro call fails whatever the agent believes, and a loop of small refunds stops at 5,000 euros a day. That is the key card, not the note.

Then the drill, run inside the standard change approved the week before. The order-intake agent stops in 3 minutes and the refund agent in 6. The credit-check agent takes 41 minutes: its credentials belong to a shared finance service account, and two month-end scripts use the same password. Revoking it would have broken month-end, so the SRE had to find the script owners first. A 41-minute stop is not a kill switch; it is a meeting.

Step 6 found the same root cause from the money side:

AgentCalls per taskCalls per hourSpend per dayWho is paged
Order-intake agent2060040 eurosHead of customer service
Refund agent1030030 eurosFinance operations lead
Credit-check agentNot settableNot settableNot settableNobody (red)
Order-to-cash workflown/an/a120 eurosAI platform lead

The credit-check agent's calls run under the shared account, so the gateway cannot tell its spend from the month-end scripts'. One root cause, two missing switches. The action item writes itself: the credit-check agent and the two scripts each get their own identity before the next release, and only then does the agent get a ceiling.

The threat walk turned up one hand-off worth fixing. The order-intake agent passes customer notes to the credit-check agent as free text, and a note saying "credit manager has approved a higher limit" could steer it (ASI01 and ASI07). The fix: notes are data, never instructions, and approvals arrive only through the ticketing system under a human identity.

Template: the Agent Register and Autonomy Policy

This is the artifact you leave with: one entry per agent, owned by the AI platform lead, reviewed at least quarterly and whenever a re-certification trigger fires.

FieldWhat to record
Agent name and purposeOne sentence a service desk analyst would understand
Unique identityIts own directory identity; never a shared or human account
Named human ownerA person, not a team, who answers for the agent
Created and next review or expiryDates; an agent past expiry loses access automatically
Certified configurationModel version, prompt version and tool list that were evaluated
Tasks, scores and maximum levelEach task with its reversibility and value-at-risk bands, and its level L1 to L5 with the reason
Structural limitsScopes, allow-lists, caps and environments that enforce the level
Approval pointsWhich actions need sign-off, the approver role, and what evidence they see
LoggingLog location, audit summary format, session review owner and frequency
Spend and call ceilingsCalls per task, calls per hour, spend per day, workflow ceiling, where enforced, who is paged
Kill switchProcedure, who can run it, change reference and date of the last drill, measured time to revoke
RollbackRestore method and date of last verified restore
Triggers and top risksChanges that force re-evaluation; top OWASP agentic risks with owner

Three rules make the register real rather than decorative. An agent with no entry gets no production credentials. An entry with an empty kill-switch row gets no level above L2. An agent with no spend ceiling gets no key from the gateway.

Failure Modes, How to Spot Them, and the Metrics to Track

Governed autonomy decays quietly: a tool added on a Friday, a script borrowing an agent's account, a drill that slips a quarter because nobody raised the change. These seven patterns tell you it is happening.

How do you spot an agent fleet that is badly governed?

Failure modeHow to spot it
Instructions used as controlsA rule appears in the prompt, but no permission, scope or cap enforces it
Shared service accountsLogs cannot tell which agent did what, and revoking one agent breaks another system
No tested rollbackNobody can name the date of the last successful restore
Inherited human permissionsThe agent's access list is the same as its builder's
Per-call review onlyEach action looks fine, yet nobody can reconstruct what a whole session set out to do
No spend or call ceilingThe first sign of a loop is the invoice, and nobody can say what stops the next call
Unclear ownership at hand-offsWhen agent A passes work to agent B and it fails, two teams each point at the other

A quick test for the first one: find the word "never" in any agent's prompt and ask what would stop the agent if it ignored that sentence. "Nothing" means a note on a master key.

Which agent governance metrics should you track?

MetricHow to measure it
Share of agents with own identity and named ownerRegister count divided by agents discovered in the inventory
Actions by autonomy levelShare of agent actions executed at each level, monthly, from logs
Approval and override ratesApprovals requested, approved, changed and rejected, per agent
Time to revokeStopwatch result from the quarterly drill, per agent, run under an approved change
Rollback success in drillsRestores attempted against restores completed and verified
Ceiling trips and spend against ceilingTrips per agent per month, and peak daily spend as a share of the ceiling

Read the ceiling metric both ways: a ceiling never approached is too loose to catch a loop early, and one tripped weekly is too tight or pointing at a bug. No public benchmark exists for an acceptable incident or override rate, so track your own trend. A rising override rate may mean approvers are paying attention, and each override is the signal Lesson 23 teaches you to capture.

Frequently Asked Questions

How do you audit what an AI agent actually did?

Audit the session, not the single call: log every action against the agent's own identity with its inputs, tool calls and outcome, and have a named person review whole sessions for patterns such as many small actions adding up to one large one. Check its summary against systems of record; an agent's account of itself is not evidence.

Can an AI agent run under a shared service account or its builder's login?

No, not in production. A shared or borrowed account means you cannot limit, trace or meter the agent, or stop it without stopping everything else on that account; in the worked example that turns a 3-minute revoke into 41 minutes.

How do you stop an agent fleet from running up a huge cloud bill?

Set a spend ceiling and a call ceiling for each agent, plus a total for each workflow, and enforce them where the agent's keys are issued, so a tripped ceiling suspends the key and pages the owner. Mandiant's September 2026 report describes a looping agent that made over 15,000 calls in under an hour, exactly the pattern an hourly call ceiling exists to catch.

Can we drill a kill switch on a live agent without breaking change control?

Yes, if the drill is itself a pre-approved change. Raise a standard change a week ahead so the change advisory board approves the revocation, run it in the agreed window on a non-critical agent with its business owner on the call, and record the change reference and the measured time in the register.

How can we tell whether we are ready to let agents act alone?

Measure agentic AI readiness agent by agent: an agent is ready for L4 only when it has its own identity and named owner, structural limits that enforce its ceiling, a spend ceiling, and a kill switch drilled and timed in minutes. The share of your fleet that meets all four is your readiness figure for the steering committee.

Key Takeaways

  • A rule in a prompt is a request; a missing permission is a control.
  • Audit whole sessions and hand-offs, not single calls: the espionage campaign Anthropic disclosed passed per-request checks by splitting hostile work into innocent-looking tasks.
  • Give every agent its own identity, a named owner, least-privilege credentials and an expiry date. A shared account turns a 3-minute stop into a 41-minute meeting.
  • Choose autonomy by reversibility and value at risk, task by task, using the five human roles from Operator to Observer. Capability does not earn autonomy, and a new model, prompt or tool sends the agent back to evaluation.
  • Put spend and call ceilings on every agent and workflow, enforced where keys are issued. A cost kill switch is the safety switch pressed by a meter.
  • A kill switch revokes credentials, pauses queues and restores state, and counts only when drilled and timed inside a change your change advisory board approved in advance.
  • Leave with the Agent Register and Autonomy Policy. Next, in Lesson 23, the approvals and overrides it logs become your data flywheel.