The commissioner of a state revenue department, Yusuf Castellanos, approved an AI system to flag fraudulent tax refund claims. In its first filing season it caught $47 million in fraud the old rules missed, a triumph the agency put in a press release. Six weeks later a different number surfaced: roughly 9,000 legitimate taxpayers, disproportionately low-income filers claiming the earned-income credit, had their refunds frozen for weeks by the same system. The model had learned that low-income returns looked riskier because they had been audited more in the past. Yusuf had deployed AI into a mission-critical function, and discovered that on the functions that matter most, the cost of being wrong is measured in citizens harmed, not just errors logged. This lesson is about leading AI into the functions where failure is not an option.

What Mission-Critical Actually Means

Not all government work carries the same stakes. Automating the cafeteria menu and automating fraud detection are not the same gamble. A mission-critical function is one where failure causes irreversible or severe harm: lost livelihoods, denied legitimate benefits, security breaches, physical danger, or the collapse of public trust. The defining test Yusuf now applies is whether damage at scale can be undone. If the answer is no, the function is mission-critical and the ordinary rules for IT projects are insufficient.

The term is used loosely, so it helps to have criteria rather than instinct. The source material applies four. Loss of life implication: failure can directly or indirectly cause death or serious injury. Economic harm: failure can cause catastrophic damage to the nation, a sector, or key infrastructure. Critical capability loss: failure prevents the government from performing a legally or politically required function. Irreversible trust damage: failure erodes public confidence in ways that are difficult to rebuild. A function that meets any one of these is in scope; most of the systems that hurt people meet more than one at once.

The federal framing sits on top of that. Presidential Policy Directive 21, issued February 2013, identified sixteen critical infrastructure sectors: chemical, commercial facilities, communications, critical manufacturing, dams, defense industrial base, emergency services, energy, financial services, food and agriculture, government services and facilities, healthcare and public health, information technology, nuclear reactors, transportation systems, and water and wastewater. National Security Memorandum 22, dated April 30, 2024, replaced that directive and restated the framework, with a sector risk management agency coordinating each sector. Departmental safety and security guidelines issued in April 2024 address AI specifically across those sectors.

The federal AI framework connects to this through a definition worth memorising. OMB Memorandum M-24-10, issued in 2024, designates safety-impacting AI as AI that can materially affect life, health, safety, critical infrastructure, or the environment. Those are the federal AI systems most commonly aligned with mission-critical functions, and the designation carries minimum practices rather than aspirations. If your system meets that definition, the question is not whether safeguards apply but which ones, and your agency's AI governance office rather than your programme office owns that answer.

Where Yusuf's Peers Live

Four domains dominate the category in ordinary government experience, and they share a structure worth seeing clearly.

• Revenue collection. Tax administration, fraud detection, benefit-overpayment recovery. Wrong here freezes the money families live on.
• Border security. Screening, identity verification, risk assessment. Wrong here detains the innocent or admits the dangerous.
• Emergency management. Disaster prediction, resource dispatch, emergency call triage. Wrong here costs response time when minutes are lives.
• Critical infrastructure. The power grid, water systems, transportation control. Wrong here cascades across millions of people.

The Risk-and-Reward Equation

The reason to use AI in these functions is real, not hype. Yusuf's system did catch $47 million in fraud humans missed. AI can find patterns across millions of records no analyst could review, operate around the clock, and respond at machine speed in emergencies. The upside is precisely largest in mission-critical functions, because that is where scale and speed matter most, and a leader who refuses on principle is also making a decision with consequences for the people the function serves.

But the risk scales identically. The same system that catches $47 million in fraud at scale can freeze 9,000 legitimate refunds at scale. In mission-critical functions, reward and risk are not separate dials; they are the same dial. A model powerful enough to be worth deploying is powerful enough to do serious harm when it errs. Leadership at this altitude is not about maximizing the reward. It is about capturing the reward while bounding the harm, which is a different optimisation with different evidence requirements.

The Asymmetry That Changes Everything

Yusuf's hard lesson was about error asymmetry. His system optimized to catch fraud. Catching more fraud meant flagging more claims. But the two ways to be wrong are not equal in human terms. A false negative, fraud that slips through, costs the state money it can often recover later. A false positive, a real family's refund frozen, costs that family rent, groceries, and their faith in the agency, frequently in ways that cannot be recovered at all in the moment they are needed.

The model did not understand this asymmetry; it treated both errors as equally bad arithmetic. The reframe Yusuf imposed was to decide which error is worse for citizens before tuning the system, and to tune deliberately toward the less harmful failure. For refunds, a slightly higher fraud-leakage rate was an acceptable price for not freezing thousands of legitimate families. That is a values decision a leader must own, not a threshold an engineer should pick alone, and it should be written down with a name against it so that the next tuning cycle knows it was a decision rather than a default.

The Domain Landscape and Its Specific Disciplines

Mission-critical AI is not one practice. Each domain has its own authorities, safety discipline and characteristic failure, and a leader moving between them should expect the vocabulary to change. What follows is a map of where the sources locate the work, not a substitute for the domain expertise each one demands.

In defense command and control, AI runs through the Chief Digital and AI Office's Task Force Lima for generative AI, Project Maven for intelligence from imagery, the ADVANA platform for data integration, and the Joint All-Domain Command and Control concept, with AUKUS Pillar 2 accelerating cooperation with the United Kingdom and Australia on AI. DOD Directive 3000.09, updated January 2023, governs autonomy in weapons systems and requires appropriate levels of human judgment over the use of force.

The department's responsible AI strategy and implementation pathway emphasises test, evaluation, verification and validation, and DoDI 5000.87 provides the software acquisition route. The design distinction that matters is between advisory AI, operator-confirmed AI, and fully autonomous systems, and the source states that DOD policy disfavours full autonomy for lethal force absent specific authorization.

In the intelligence community, AI supports imagery analysis, signals exploitation, counterintelligence and open-source triage under ethics principles issued in 2020 and an accompanying risk management framework, with ICD 503 governing risk management for community systems. Classification and compartment controls require air-gapped or cross-domain deployment, which constrains architecture in ways commercial patterns do not anticipate. Analytic judgment remains human-led under the community's tradecraft standards: AI accelerates triage and pattern detection but does not replace the analyst's sourced, reasoned judgment. Red-teaming matters here because adversaries actively work to manipulate or evade analytic AI; foreign-origin components are screened, and foreign investment review applies to vendor transactions. The Cyber Threat Intelligence Integration Center provides coordination across the community.

In air traffic management, AI supports arrival and departure prediction, weather-impact analysis, runway assignment and conflict detection, with surface surveillance capability integrating sensors and machine-learning conflict prediction. The source describes certification standards under Parts 25, 23, 27 and 29 being updated for AI-enabled avionics through advisory guidance such as AC 20-174 and standards work including RTCA DO-178C and DO-254 with machine learning extensions, with parallel European effort under an AI roadmap; confirm any of those references with your certification authority before relying on them. Human-in-the-loop is the standard in safety-critical avionics and air traffic control, and investigations of automation-related incidents drive design.

The discipline this domain contributes to everyone else is graceful degradation stated as a requirement: if the AI is unavailable or uncertain, controllers and pilots retain full capability, and the operational reporting system captures the near-misses that reveal whether that is true in practice.

In disaster response, AI supports damage assessment, predictive resource positioning, satellite imagery analysis and public alerting through the Integrated Public Alert and Warning System, structured by the National Response Framework and the National Incident Management System under Stafford Act authority, with cross-agency coordination spanning agriculture, environmental and health emergencies. The source is explicit that AI is currently advisory in disaster response and that final resource decisions remain with human incident commanders. Its named risks are instructive for every domain: model drift as climate change shifts disaster patterns, equity concerns because damage assessment algorithms have in some studies favoured higher-value properties, and data quality varying across incident types.

On the energy grid, national laboratory and departmental work applies AI to demand forecasting, state estimation, stability assessment and distribution management, under reliability regulation and critical infrastructure protection standards. The asymmetry here runs both ways: false triggers can cause outages and missed indicators can cascade. The 2003 Northeast blackout and the 2021 Texas winter storm illustrate what grid failure costs. Nuclear applications carry additional regulatory oversight, and distribution utilities adopt more slowly because their capital cycles are long, which is a reminder that adoption speed in infrastructure is set by asset replacement rather than by software.

In public health, AI supports syndromic surveillance, outbreak forecasting through a forecasting centre established in 2021, genomic epidemiology and vaccine effectiveness monitoring, under the Public Health Service Act and the Pandemic and All-Hazards Preparedness Act. The pandemic exposed variable electronic health record data quality, limited data sharing with state and local partners, and reporting latency. The response was durable data pipelines, academic partnerships through Insight Net, publicly available code and evaluation, and integration with the established Morbidity and Mortality Weekly Report communication channel. Critically, the source states that forecast accuracy during novel outbreaks is limited, and that the centre uses ensemble approaches and explicitly communicates uncertainty rather than presenting a single number.

In tax administration, AI and advanced analytics support audit selection, earned income tax credit review, identity verification, fraud detection and taxpayer service, funded by the Taxpayer First Act of 2019 and the Inflation Reduction Act of 2022. The source records a facial recognition identity-verification rollback in 2022 as a cautionary tale, with current identity verification running through a shared federal login service. Taxpayer rights are enumerated, audit selection AI must be contestable, taxpayers have statutory appeal rights, and independent review comes from the taxpayer advocate alongside audit bodies. Generative AI for taxpayer service is advancing cautiously because hallucination is particularly dangerous in tax advice.

In disability adjudication, AI assists case processing, medical evidence retrieval and decision support across millions of annual claims handled by state-level determination services and administrative law judges, under the Social Security Act, its regulations at 20 CFR Parts 404 and 416, and Administrative Procedure Act due process, with advisory committee structures providing oversight. Historical criticism centres on variability across judges and processing delays, and AI could standardise but also risks amplifying historical bias.

The equity stakes are stark: claimants face processing times long enough that some die waiting, and disparate impact on rural, limited-English-proficiency and disability subgroups is a recognised risk. Transparency to claimants about AI involvement, and human oversight of it, is a due process concern under Mathews v. Eldridge and related jurisprudence.

Choosing the Oversight Level Deliberately

For mission-critical AI, the operating model must specify a human oversight level, and the choice should be explicit rather than inherited from a vendor's default. Human-in-the-loop means a human authorises each AI decision before it takes effect, which the source treats as appropriate for high-stakes individual decisions such as benefits adjudication and weapons release. Human-on-the-loop means the AI operates autonomously while a human can intervene, appropriate where the timescale makes per-decision authorization impossible, such as air defense. Fully autonomous means no human intervention during execution, reserved for pre-authorized rules, bounded scenarios and robust safety cases.

The federal minimum practices require human oversight for safety-impacting and rights-impacting AI, and the autonomy directive requires appropriate human judgment over lethal force. But the source puts the real design question well: it is not whether human oversight exists, but whether the human has a meaningful ability to understand, evaluate and override, meaningful in terms of time, information and cognitive load. A reviewer with four seconds, no visibility into why the system flagged the case, and a queue behind them is not an oversight control. They are a person who will be blamed for an outcome they could not have changed.

Automation complacency is the well-documented failure that follows. Humans defer to systems that are usually right, and the deference grows precisely as the system gets better, which means the oversight control weakens as the model improves. Training, workflow design and audits mitigate this. They do not solve it. Assume some rate of rubber-stamping in any high-volume review process and design to detect it: sample overturned and non-overturned decisions, look at review times, and treat a reviewer who never disagrees with the model as a finding rather than a good employee.

Resilience, Fallback, and the Difference Between a Plan and a Capability

Mission-critical AI must have resilience designed in rather than added. Failure modes and effects analysis identifies how the AI can fail and what follows from each failure. Fail-safe design ensures that failure defaults to a safe state; fail-operational design maintains operation in a degraded mode. Graceful degradation preserves useful function when AI is unavailable or uncertain. Fallback procedures specify what humans and systems do without AI. The source also names the tradeoff most programmes miss: redundancy at data, model and infrastructure layers increases availability but also increases attack surface, and that has to be managed rather than celebrated.

Cross-domain dependencies on power, communications and cyber infrastructure need assessing, because a resilient application on a fragile substrate is not resilient. NIST Special Publication 800-160 on systems security engineering and the measure function of the NIST AI Risk Management Framework both address this ground. After-action reviews following incidents, including near misses rather than only actual failures, feed continuous improvement. Joint exercises across agencies, including tiered emergency management exercises and large-scale defense exercises, test end-to-end resilience with AI components in the loop rather than in the appendix.

Now the part that gets skipped. A fallback procedure that has never been exercised is a document, not a capability. The people who would execute it have not done it, the credentials have not been tested, the manual process has probably lost the staff who knew it, and the runbook references a system that was decommissioned two years ago. The same applies to a rollback: an untested rollback is a plan to discover a problem during an outage. Exercise the fallback on a normal Tuesday, with the people who would actually be on shift, and measure how long the function takes to run without the AI. That number is your real resilience, and it is usually worse than anyone expected.

Monitoring carries the same caveat and it is worth saying plainly, because live dashboards are the most oversold control in this field. Monitoring surfaces only what you chose to watch, in the categories you defined, at the thresholds you set. Yusuf's system was monitored. The metric it was monitored on was fraud caught. Nobody had instrumented the distribution of frozen refunds by income, which is why the problem surfaced six weeks later through complaints rather than through a dashboard. Choosing what to watch is a substantive design decision about who can be harmed invisibly, and it should be reviewed by someone whose job is not to make the system look good.

A pilot is not a proof either. A successful pilot is evidence about performance under pilot conditions: a curated population, attentive staff, low volume, and everyone paying attention because it is new. Production adds volume, staff turnover, edge cases the pilot population did not contain, and the ordinary erosion of attention. Treat pilot results as a reason to proceed carefully with monitoring designed for the failure modes the pilot could not have surfaced, not as a safety case. The source lists deploying without test, evaluation, verification and validation as a contributor to most federal AI failures, and a pilot is not that testing.

The Non-Negotiables for Mission-Critical AI

Across all four of Yusuf's domains, the same safeguards separate responsible deployment from recklessness. He treats these as conditions of approval, not as nice-to-haves.

• Meaningful human control over consequential decisions. A frozen refund, a flagged traveler, a benefit denial: a human reviews before the harm lands on a person, with real authority and real time to override. Not a rubber stamp, and monitored for whether it has become one.
• A fast, human path to reverse errors. When the system is wrong, the citizen can reach a person and get it fixed in days, not months. The appeal channel must be funded and staffed as part of the system, not as an afterthought.
• Tested for disparate impact. Before and during deployment, measure whether errors fall unequally on protected or vulnerable groups, which is exactly the test that would have caught the earned-income-credit bias.
• Graceful failure and an exercised manual fallback. When the AI goes down or degrades, the function keeps running, and the fallback has been rehearsed by the people who would run it.
• Continuous monitoring, not launch-and-leave. Models drift as the world changes. Mission-critical systems need live measurement of error rates and equity, with a named person accountable for looking and for adding metrics when the harm turns out to be somewhere nobody instrumented.

These map onto the NIST AI Risk Management Framework, a voluntary federal guide for governing AI risk, and onto the federal requirement under OMB Memorandum M-24-10 that rights-impacting and safety-impacting AI carry specific safeguards including human oversight and a path to remedy. Yusuf does not cite these to auditors so much as use them as a ready-made checklist for whether the work was done properly. The checklist is a floor. Clearing it means you have not omitted anything obvious, which is different from having a system that is safe to run.

A Usable Artifact: The Mission-Critical AI Go and No-Go Gate

No mission-critical AI system reaches production in Yusuf's agency without clearing every gate below. A single no stops deployment, and the gates are answered in writing by named people rather than discussed in a meeting.

Learning From Failures in Other People's Domains

Every domain in this lesson has a history of automation failure that the others could have learned from and mostly did not. In defense, a Patriot missile battery in 2003 accidentally engaged friendly aircraft through an interaction between automation and human oversight, and the Aegis incident involving Iran Air 655 in 1988 showed automation bias under stress. In aviation, Air France 447 in 2009 demonstrated the risk in an automation handoff, Asiana 214 is treated as a further example of automation complacency, and the 737 MAX MCAS case is the software instance of the same pattern.

The transferable point is not that automation is dangerous. It is that these failures share a shape: a system that behaved as designed, in conditions the design did not anticipate, in front of humans whose training and situational picture had quietly been reorganised around the assumption that the system would handle it. Ignoring analogous failures is itself listed in the source as an anti-pattern, and it is the cheapest one to avoid. Before deploying, read the incident history of the closest analogous system in any domain, and ask which of its failure conditions your design would also fail.

The Leadership Altitude

Yusuf fixed his system. He rebalanced it toward fewer false positives, funded an appeal line with a 72-hour service commitment, and added equity monitoring disaggregated by income. The next season it still caught tens of millions in fraud and froze a small fraction of the legitimate refunds it had frozen before. He is careful, now, not to describe that as solved. It is a system performing acceptably under this year's conditions, with monitoring that would probably catch a recurrence of last year's failure and might not catch a new one.

The lesson he carries to every peer commissioner is about what the leader's question should be. It is not to approve the impressive demo. It is to ask, relentlessly, what happens when it is wrong, because it will be: who gets hurt, how would we find out, how fast can we make them whole, and does the function still run while we do. Those four questions are answerable before deployment, they are cheap to ask, and every case in this lesson is a case where somebody senior did not ask them.

Anti-Patterns

• Autonomy creep. A system designed for advisory use gradually becomes authoritative without explicit reauthorization, as staff learn to accept its output and the workflow reorganises around it. Nobody decides this; it accumulates. Re-examine the oversight level on a schedule, not on incident.
• Automation complacency. Humans defer to AI without critical review, and the deference deepens as the system gets more reliable. Detect it by sampling review times and overturn rates rather than by asking reviewers whether they are paying attention.
• Single-point dependency. The mission-critical function depends on one vendor, one region, or one data source, and a failure anywhere in that chain cascades into the mission.
• Testing in production. Deploying without genuine test, evaluation, verification and validation. The source identifies this as a contributor to most federal AI failures, and a successful pilot does not substitute: it is evidence under pilot conditions, not a safety case for production load.
• Metric-only safety. Measuring accuracy without considering rare but catastrophic failure modes. An excellent aggregate number is compatible with a specific population being harmed systematically, which is precisely what happened in this lesson's opening.
• No fallback, or an unexercised one. No documented procedure for AI unavailability, or a procedure nobody has run. An unexercised fallback is a document, not a capability, and you will discover the difference during the outage.
• Missing incident response. No playbook for AI-specific failure, so the first hour of a real incident is spent deciding who is in charge.
• Over-trusting the vendor safety case. Accepting vendor claims about safety without independent verification. The vendor tested what the vendor chose to test, on data the vendor selected, for conditions the vendor anticipated.
• Certification-free operation, and its mirror image. Federal safety-critical systems typically require a certification basis, and operating without one is an anti-pattern. So is treating certification as proof of safety: it establishes that a defined standard was met at a point in time, on a system that has since drifted.
• Ignoring analogous failures. Repeating a known failure pattern from another domain because it happened in aviation, or defense, or another country, and therefore did not feel relevant.
• Believing redundancy is free. Redundancy at data, model and infrastructure layers raises availability and also enlarges the attack surface and the number of things that can be misconfigured. It is a tradeoff to manage, not a safeguard to accumulate.
• Trusting the dashboard you designed. Monitoring surfaces what you chose to watch. The harm that reaches you through complaints rather than metrics is the harm you did not instrument, and no amount of dashboard refresh rate fixes a missing dimension.

Practice Prompts

• Apply the four mission-critical criteria to your own portfolio: loss of life, catastrophic economic harm, loss of a required capability, and irreversible trust damage. Which systems qualify that you have been managing as ordinary IT?
• For one qualifying system, write the error asymmetry decision explicitly. Which error is worse for citizens, who decided that, when, and is the system currently tuned in line with that decision or with an engineering default?
• Name the oversight level for each mission-critical system you run, then check whether the reviewer actually has the time, information and authority the level implies. Where the answer is no, you have a nominal control.
• Schedule and run a fallback exercise for your most critical AI-dependent function, on a normal working day, with the people who would really be on shift. Record how long the function takes to run without the AI.
• List every metric currently monitored on one production system, then list the populations that could be harmed by it. Find the harm nobody instrumented, and add the metric before you need it.
• Take an automation failure from a domain other than your own and write down which of its conditions your system would also fail. Bring the list to your next design review.

Reflection

Think about the most consequential AI-assisted decision your organization makes about an individual person. Trace one such decision end to end from that person's side: how the system reached it, what they were told, who reviewed it, how long a reviewer actually spent, what it would take for them to contest it, and how long the whole cycle runs. Then ask how you would find out, today, if that path had quietly stopped working. If the honest answer is that you would find out through complaints, litigation or press coverage, that is the state of your monitoring regardless of what the dashboard shows.

Then consider your own relationship to the good news. Yusuf's system produced a press release before it produced a problem, and the press release was accurate. The failure was not that the fraud number was wrong; it was that the fraud number was the only number anyone had chosen to watch. Every leader running a mission-critical system has a headline figure that makes the programme look successful. Ask what that figure would look like if the system were harming a specific group systematically, and whether you would be able to tell the difference from where you sit.

Glossary

• Mission-critical function. A function where failure causes loss of life, catastrophic economic harm, loss of a legally or politically required capability, or irreversible damage to public trust.
• Safety-impacting AI. Under the federal memorandum, AI that can materially affect life, health, safety, critical infrastructure, or the environment, which triggers minimum practice requirements.
• Human-in-the-loop. An operating model in which a human authorises each AI decision before it takes effect.
• Human-on-the-loop. An operating model in which the AI operates autonomously while a human retains the ability to intervene.
• Error asymmetry. The condition where a false positive and a false negative impose different kinds and magnitudes of harm, so the tuning point is a values decision rather than a technical one.
• Failure modes and effects analysis. A structured method for identifying how a system can fail and what consequence follows from each failure mode.
• Fail-safe and fail-operational. Fail-safe design defaults to a safe state on failure; fail-operational design continues to operate in a degraded mode.
• Graceful degradation. Preserving useful function when a component is unavailable or uncertain, rather than failing wholesale.
• Fallback procedure. The defined way humans and systems continue the function without the AI, which counts as a capability only once it has been exercised.
• Automation complacency. The documented tendency of human overseers to defer to a system that is usually right, weakening oversight as reliability improves.
• Model drift. Degradation of a model's fitness as the world it was trained on changes, which in mission-critical settings is a safety issue rather than a maintenance one.
• Test, evaluation, verification and validation. The discipline of establishing that a system does what it is specified to do under the conditions it will meet, distinct from a pilot.

Related Lessons

• Rights-Impacting and Safety-Impacting AI Safeguards specifies the federal safeguards this lesson treats as conditions of approval.
• AI in Defense and National Security goes deeper into the defense and intelligence considerations sketched here.
• Continuity of Operations with AI develops the fallback and resilience practice in operational detail.
• AI Incident Response Planning supplies the playbook whose absence is one of this lesson's anti-patterns.
• Testing and Validating AI Systems covers the verification discipline that a pilot does not substitute for.
• Bias Detection and Mitigation at Scale details the disparate impact testing that would have caught this lesson's opening failure.
• Moving from Pilot to Production addresses the transition where pilot evidence stops applying.
• When Government AI Goes Wrong examines the failure record across government in depth.
• AI in Infrastructure extends the critical infrastructure discussion into sector practice.

Closing

Mission-critical government functions span defense, intelligence, air traffic, disaster response, energy, public health, tax administration and disability adjudication. Each has its own authorities, its own safety discipline and its own characteristic failure, and none of them can be governed from a generic AI policy alone. What travels between them is a small set of habits: choose the oversight level deliberately, decide the error asymmetry as a values question, exercise the fallback, instrument the harm rather than only the benefit, and read the incident history of systems unlike your own.

What does not travel is the reassurance. No safeguard in this lesson guarantees that a mission-critical system keeps working. Redundancy adds attack surface. Monitoring sees what it was told to see. Certification describes a moment. Human oversight decays as the system improves. A pilot proves something about a pilot. The honest position for a leader is that these controls reduce the probability and the blast radius of failure, and that the remaining risk is real and is being accepted deliberately by a named person. Yusuf's agency is safer than it was, and he would be the first to say it is not safe.

Key Takeaways

• Mission-critical means harm is severe or irreversible. Four criteria decide it: loss of life, catastrophic economic harm, loss of a required capability, and irreversible trust damage.
• Reward and risk are the same dial. A model powerful enough to be worth deploying is powerful enough to harm at scale when it errs.
• Error asymmetry is a values decision. Decide which failure is worse for citizens, record who decided, and tune deliberately toward the lesser harm rather than accepting an engineering default.
• Choose the oversight level explicitly. In-the-loop, on-the-loop and fully autonomous carry different requirements, and the design question is whether the human has the time, information and authority to actually override.
• Automation complacency worsens as the model improves. Detect rubber-stamping by sampling review times and overturn rates; a reviewer who never disagrees is a finding.
• An unexercised fallback is a document. Rehearse it with the people who would run it, and measure how long the function takes without the AI.
• Monitoring only surfaces what you chose to watch. The opening failure in this lesson reached the agency through complaints because nobody had instrumented refunds by income.
• A pilot is evidence, not a safety case. Pilot conditions omit volume, turnover, edge cases and the erosion of attention that production supplies.
• Fund the remedy path. A fast, human way to reverse errors is part of the system, staffed and budgeted, not an afterthought.
• Read other domains' incident histories. Defense, aviation and infrastructure failures share a shape, and ignoring analogous failures is the cheapest anti-pattern to avoid.
• Lead from when it is wrong. Who gets hurt, how would we find out, how fast can we make them whole, and does the function still run: four questions, all answerable before deployment.

Frequently Asked Questions

How do I know whether a system of mine counts as mission-critical? Apply the four criteria rather than your intuition about the programme's profile. Can failure cause death or serious injury, catastrophic economic harm, loss of a legally or politically required capability, or irreversible damage to public trust? Then apply the reversibility test: if this is wrong at scale, can the harm be undone? Systems that clear both tests as low-stakes are genuinely ordinary IT. Everything else should be governed with the safeguards in this lesson, and your agency's AI governance office rather than your programme owns the formal designation.

Is human oversight enough to make a high-stakes system safe? No, and this is the most common overstatement in the field. Oversight is a control whose strength depends entirely on whether the reviewer has time, information and authority, and it weakens as the system becomes more reliable because deference grows with accuracy. Treat human oversight as one layer alongside asymmetry decisions, disparate impact testing, exercised fallbacks and instrumented monitoring, and design specifically to detect the moment it has become a rubber stamp.

Our pilot went well. Can we scale it? You can proceed, with monitoring designed for what the pilot could not surface. A pilot runs on a curated population, with attentive staff, at low volume, while everyone is watching because it is new. Production adds volume, staff turnover, edge cases and the ordinary erosion of attention. The source lists deploying without genuine test, evaluation, verification and validation as a contributor to most federal AI failures, and a successful pilot is not that testing.

We have redundancy and a documented rollback. Are we resilient? Not yet, on the evidence. Redundancy increases availability and also increases attack surface and configuration complexity, so it is a tradeoff to manage rather than a safeguard to stack. And a rollback that has never been executed is a plan to discover a problem during an outage. Run the fallback on a normal day with the staff who would actually be on shift, and treat the elapsed time as your real resilience number.

How should we decide the false positive and false negative balance? As a leadership decision, documented, with a name against it. Establish which error is worse for the people affected, in their terms rather than in the agency's, and accept the cost of the lesser harm explicitly. In the opening case, a slightly higher fraud-leakage rate was the correct price for not freezing thousands of legitimate refunds. Do not leave this to a threshold chosen during model development, and revisit it whenever the system is retuned.

What single practice would have prevented this lesson's opening failure? Deciding in advance which populations could be harmed and instrumenting the system for that harm. The bias was findable: error rates disaggregated by income would have surfaced it in the first weeks rather than after six weeks of complaints. Disparate impact testing before launch and disaggregated monitoring after it are the same discipline applied at two moments, and both are cheaper than the remediation that follows discovering the problem from the outside.