Inés Haddad-Quiroga chaired the AI governance board at a large federal agency, and she had a spreadsheet of forty-one AI systems in front of her. The board's first real task was deceptively simple: sort each one into a risk category. She had assumed it would take an afternoon. It took three weeks, because almost every system sat in a gray zone, and because the people who built each one had a strong incentive to argue their system was low-risk and therefore exempt from the heavy compliance obligations the high-risk categories carried. A document-summarization tool used by the benefits division, was that rights-impacting? It only summarized; a human still decided. But the human decided based on the summary, and the summary could omit the one fact that mattered. Inés learned that risk classification is not a clerical sorting exercise. It is the single most consequential governance judgment her board would make, because the category determines every obligation that follows.

This lesson is about how to classify AI risk in government using the two categories that drive federal AI policy: safety-impacting and rights-impacting. These are not academic labels. Under OMB Memorandum M-24-10, "Advancing Governance, Risk Management, and Responsible AI Use in Government," a system's classification determines whether it must undergo an impact assessment, satisfy minimum risk-management practices, provide human oversight, and offer affected people notice and an appeal. Get the classification right and the rest of governance follows. Get it wrong, usually by under-classifying, and the agency carries undocumented liability into its next audit.

Why Classification Comes First

Not all AI systems pose the same kinds of risk. A system that flags potential benefits fraud for manual review poses different governance challenges than one making final eligibility determinations. A chatbot answering general questions about visa requirements presents different concerns than a system used in border screening. Risk classification gives an agency a common language for talking about impact, and it is what lets you decide what oversight level is appropriate, what validation is needed, and how to allocate finite governance resources before you have built anything.

M-24-10 emphasizes that agencies must classify AI systems by impact, document their risk profile, and implement appropriate oversight. Federal agencies operate under constitutional constraints, statutory requirements, and public accountability principles that require deliberate management of these risks. In practice the classification answers a specific set of downstream questions: how often should we audit this system, weekly, monthly or quarterly? Who signs off on deployment, the technical team, legal, or an executive? What level of human oversight is required, sampling, full review, or random spot-checks? What testing must happen before deployment, fairness analysis, edge case testing, domain expert review? And what happens if the system fails, can we simply turn it off or do we need a contingency plan?

The Two Categories That Matter

M-24-10 defines two categories of consequential AI, each with a presumptive list of covered uses.

Safety-impacting AI is AI whose output could meaningfully affect human life, health, or the safety of physical infrastructure or property. Stated the other way round, it is a system whose failure or malfunction could reasonably result in significant injury or severe harm to human health or safety, or could reasonably result in loss of life. Think of systems that control or inform the operation of dams, power grids, water treatment, transportation, or emergency response; systems used in clinical decision support; and systems that affect the integrity of elections or critical infrastructure. The defining question: if this system is wrong, could someone be physically harmed or could critical infrastructure fail?

Rights-impacting AI is AI whose output serves as a principal basis for a decision or action affecting an individual's civil rights, civil liberties, privacy, equal opportunity, or access to critical government resources and services. Put in terms of failure, it is a system whose failure, malfunction, or misuse could meaningfully impact civil rights, civil liberties, or privacy, including by discriminating against individuals or groups or depriving someone of a fundamental right. This covers AI that influences eligibility for benefits, housing, credit, employment, or education; that informs law-enforcement or immigration decisions; that affects access to government services; or that monitors or profiles individuals. The defining question: if this system is wrong, could a person lose access to something they have a right to, or be treated unequally?

A system can be both. Inés's hardest case was a wildfire-risk model that prioritized which households received evacuation alerts: safety-impacting because lives depended on it, and rights-impacting because unequal alerting across communities raised equal-protection concerns. Both classifications applied, and both sets of obligations attached.

Safety-Impacting Systems in Practice

Government examples of safety-impacting AI include a system that assists in aviation safety decisions at the FAA; a system that helps identify critical infrastructure vulnerability points; AI used in emergency dispatch to prioritize response; a system that assists in nuclear facility operations monitoring; and AI that predicts bridge infrastructure failure patterns for maintenance prioritization. What makes these safety-impacting is that their failure mode is human injury or death, and the chain of causation is direct and relatively short. If the system makes a wrong recommendation and that recommendation is followed, people can be harmed.

Four characteristics recur. The consequences of failure are physical: injury, death, or damage to critical infrastructure. These systems often operate in time-sensitive environments where humans cannot easily second-guess a recommendation. Their failure consequences are frequently irreversible or extremely costly. And they may run with partial automation, where human oversight formally exists but is limited by time pressure or by the reviewer's expertise. That last point matters more than it sounds: an oversight design that assumes a human has minutes to think is not oversight in a setting that gives them seconds.

The governance that follows is correspondingly heavy. Safety-impacting systems require robust testing including edge cases, adversarial testing, and failure mode analysis. They need formal risk assessment and safety validation before deployment. They typically require continuous monitoring with automated alerts for performance degradation. They need clear escalation procedures and human override capabilities. They may need independent expert review before deployment and periodically thereafter. And they demand detailed documentation of system design choices and failure modes, because the design rationale is exactly what an investigator will ask for after an incident.

Rights-Impacting Systems in Practice

Government examples of rights-impacting AI include a system that assists in criminal sentencing recommendations; a system that predicts recidivism to inform parole decisions; AI used to screen job applicants for government positions, carrying disparate impact risk; a system that identifies potentially fraudulent benefit applications, affecting eligibility for public services; AI that flags individuals for enhanced screening in security or immigration contexts; a system that determines loan eligibility for small business administration programs; AI used to assess creditworthiness for federal credit programs; and a system that categorizes vulnerability levels for social services eligibility.

What makes these rights-impacting is that failure or misuse affects someone's access to fundamental services, opportunities, or liberty, and the impact falls hardest on individuals and groups who lack alternative channels to remedy the harm. The characteristic pattern: impact on access to government services or opportunities; potential for disparate impact on protected classes or vulnerable populations; frequent involvement of sensitive personal information such as race, gender, or disability status; a failure mode that includes discrimination or unfair treatment rather than accuracy failure alone; effects felt over time or across many decisions; and individual harms that accumulate across a population until they are visible only in aggregate.

The governance obligations differ in kind, not just in intensity. Rights-impacting systems require bias detection and fairness analysis before deployment. They need meaningful human review for high-stakes decisions, not just sampling. They demand transparency about how the system works and what factors drive its decisions. They require appeal or reconsideration mechanisms for individuals affected by adverse decisions. They need demographic performance analysis and ongoing monitoring for bias drift. They demand clear documentation of validation methodology and fairness metrics. And they often require legal review for compliance with civil rights laws and equal protection principles.

The Distinction That Shapes Oversight

A safety-impacting system primarily poses a risk of physical harm. When it fails, it has typically failed to prevent a dangerous outcome, and the underlying problem is accuracy and robustness under challenging conditions. A rights-impacting system primarily poses a risk of unfair treatment or discrimination. It fails when it produces different outcomes for people in similar situations, especially along demographic lines, and the crucial complication is that it can be highly accurate in aggregate and still discriminate. Accuracy and fairness are separate properties; a system can be 95% accurate and still treat one group markedly worse than another.

That difference sets the question each oversight regime is built to answer. For safety-impacting systems the question is "is this system working?" and the effort goes into preventing failure and detecting degradation. For rights-impacting systems the question is "is this system fair?" and the effort goes into equitable outcomes and detecting discrimination. Where a single system is both, the requirements layer rather than substitute. An AI system used in emergency medical dispatch can affect safety, if it fails to alert responders to critical situations, and rights, if it systematically under-allocates resources to certain neighborhoods; that system needs robust failure testing and fairness analysis, not a choice between them.

The "Principal Basis" Test

The most contested word in the rights-impacting definition is "principal basis." Sponsors argue that because a human makes the final decision, the AI is merely advisory and therefore not rights-impacting. Inés learned to push past this reflexively. The test is not whether a human is nominally in the loop; it is whether the AI's output is a principal basis for the decision in practice.

Her benefits-summarization tool was the textbook case. Technically a human caseworker approved or denied each claim. But the caseworker handled 60 claims a day and relied on the AI summary to decide which claims to scrutinize and which to wave through. In practice, the summary was a principal basis for the outcome. The board classified it rights-impacting over the objection of the program office. That classification was correct, and it is the one that under-classifying agencies most often get wrong: a human rubber-stamp does not demote a system from rights-impacting to advisory.

The working rule Inés wrote into the board's charter put it plainly. If a reasonable caseworker, given the volume and the workflow, would in practice defer to the AI's output most of the time, then the AI's output is a principal basis for the decision, regardless of who signs the form. Nominal human review does not change the classification. Meaningful human review might, but only if it is real, and the burden of showing that it is real sits with the sponsor claiming it.

A Methodology You Can Repeat

How do you actually classify a system? Five questions, asked in order, will resolve most cases and will make the hard ones legible.

• What is the failure mode? If this system gives the wrong answer, what happens? If the answer is "someone could be injured or die," you are likely looking at safety-impacting. If the answer is "someone might not get access to a service they deserve" or "someone could face discrimination," you are likely looking at rights-impacting.
• Who bears the consequences of failure? Safety failures typically harm the individuals directly affected by the decision. Rights failures may harm individuals but also damage broader population equity and public trust in government.
• How much human review is feasible? Can a human realistically review every decision? If not, the system may require higher governance standards precisely because humans cannot catch the problems. If so, you may be able to mitigate some risks through human oversight, subject to the principal-basis test above.
• What is the sensitivity of the domain? Does this decision affect access to fundamental government services or constitutional rights? Higher sensitivity domains such as voting, criminal justice and benefits raise the rights-impact concern; lower sensitivity uses such as informational assistance potentially lower it.
• Does the system use sensitive information? If the system incorporates demographic data or other sensitive information, the risk of disparate impact rises and a rights-impacting classification becomes more likely.

Worked Case Examples

A tax fraud detection system. An AI system flags suspicious tax filings for enhanced review by IRS auditors. Failure mode: it flags non-suspicious filings as suspicious, burdening taxpayers with audits, or it misses fraudulent filings and reduces tax compliance. Rights impact: yes. Individuals incorrectly flagged face a compliance burden and an intrusive review, and if the system systematically over-flags returns from certain demographic groups it creates disparate impact. Safety impact: probably not directly, since failure does not cause physical harm. Classification: rights-impacting, requiring fairness analysis, demographic performance testing, and human review procedures that protect taxpayers from unfair targeting.

A runway inspection system. An AI system analyzes runway surface imagery to detect cracks and damage requiring maintenance. Failure mode: it misses critical damage that could cause aircraft accidents. Safety impact: yes, failure could directly contribute to loss of life. Rights impact: probably not, since inspecting runways does not make individual-level decisions affecting people's rights or services. Classification: safety-impacting, requiring rigorous edge-case testing, robustness testing under various weather conditions, and continuous monitoring for performance degradation.

A benefits eligibility screening system. An AI system performs initial screening of applications for social services benefits and flags applications for full human review. Failure mode: it incorrectly denies eligible individuals or misses eligibility factors. Rights impact: yes. Citizens are denied benefits they qualify for, and disparities across demographic groups would create discriminatory impact. Safety impact: possibly indirect, since benefits denial can contribute to housing or food insecurity for vulnerable individuals, though that chain is indirect. Classification: strongly rights-impacting and possibly also safety-impacting, requiring fairness analysis, demographic testing, human review procedures, and appeal mechanisms.

Why Under-Classification Is the Real Danger

There is structural pressure to classify low. Higher classification means impact assessments, fairness testing, documentation, human-oversight redesign, and notice-and-appeal mechanisms, all of which cost time and money. The sponsor who wants to deploy quickly is motivated to argue "advisory only." A governance board that yields to that pressure is not saving effort; it is deferring liability to the moment something goes wrong and an inspector general asks why a rights-impacting system was never assessed.

Inés built two safeguards against under-classification. First, the burden of proof ran the other way: a system presumptively fell into the higher category if it appeared on the M-24-10 lists, and the sponsor had to make an affirmative, documented case for a lower classification, not the reverse. Second, every classification, high or low, was recorded with its reasoning in the AI inventory, so a future reviewer could see not just the answer but why. A documented "we classified this advisory because the human independently re-derives the decision from primary records in every case" is defensible. An undocumented low classification is just a gap waiting to be found.

The mirror-image error is real too, and worth naming so the board does not overcorrect into theatre. Classify everything at the top and you spread finite governance capacity thinly across systems that never needed it, which starves the systems that did. The answer to both errors is the same and it is not a posture: objective criteria, applied consistently, with the evidence written down. Classification decides which systems receive which oversight. It does not by itself make that oversight effective, and a correctly classified system with a nominal review process is still an exposed system.

From Classification to Obligations

The reason the category matters is that it switches on a specific set of duties. For rights-impacting and safety-impacting AI, M-24-10 requires the agency to, before deployment, complete an AI impact assessment, test the system for performance in a real-world context, and assess for fairness and disparate impact; and, in operation, provide ongoing monitoring, meaningful human oversight, and, for rights-impacting systems affecting individuals, notice to affected people and a path to contest or appeal adverse decisions. Systems in neither category still warrant basic good practice but do not carry the full apparatus.

Inés produced a one-page classification record for each of the forty-one systems: the system name, owner, a one-line description, the classification, the specific covered-use category it matched (or the documented reason it did not), and the resulting obligations with due dates. Of the forty-one, nine were rights-impacting, two safety-impacting, one both, and the rest neither. That single artifact became the spine of the agency's entire AI compliance program, every downstream obligation traced back to a classification on that sheet.

Two other instruments sit alongside the memorandum and are worth knowing by name. The NIST AI Risk Management Framework offers guidance on impact assessment and system characterization; it is voluntary and non-binding, which makes it a useful structuring tool rather than a compliance obligation in itself. Executive Order 14110, "Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence," is the executive action under which much of this governance activity was framed; treat it as historical context and confirm the currently operative authorities with your own counsel before relying on any of it in a filing.

Reassessment as Systems Evolve

AI systems do not stay still, and a classification is a statement about a system as it was on the day it was assessed. Five scenarios should trigger a fresh look. Scope expands, when a system used in one office spreads to multiple regions. Decision authority increases, when a system that flagged cases for human review becomes fully automated. The population affected changes, when a system built for one group now serves a broader one. The use case expands, when a recommendation tool starts driving final decisions. Or performance changes, when monitoring shows degradation in particular subpopulations.

Reassessment should happen before major scope changes or deployment expansions; when performance monitoring reveals new issues; when new protected classes or vulnerable populations come into scope; when the deployment context changes significantly; and at least annually as part of routine system governance review. The annual floor matters because the other four triggers all depend on somebody noticing, and the whole point of a calendar-driven review is that it fires even when nobody has noticed anything.

Anti-Patterns

• Classification creep. A system is classified low-impact at the start and then quietly accumulates decision authority, users and population impact without ever being reclassified. Teams forget the original assessment context and keep treating it as lower-risk than it now is. Establish regular reassessment schedules, trigger formal reclassification when scope changes, document the assumptions the original classification rested on, and alert the team when those assumptions stop holding.
• Conflating accuracy with fairness. A team assumes that a system with high precision and recall is therefore fair and rights-compliant, focuses exclusively on accuracy metrics, skips fairness analysis, and discovers later that an accurate system systematically disadvantages a demographic group. Accuracy and fairness are separate concerns. Require accuracy testing and demographic parity analysis for rights-impacting systems, and treat a strong aggregate number as no evidence at all about subgroup performance.
• Arguing the classification down to avoid the governance. Leadership feels threatened by the compliance burden and pushes back on the assessment, arguing the system is lower-impact than the evidence supports. Establish clear, objective classification criteria in advance, document the evidence supporting each classification, and make clear that governance requirements are not punitive but exist to reduce real risks. Classify on actual impact, not on governance appetite.
• Treating a human in the loop as a downgrade. The sponsor points at the caseworker's signature and asks for advisory status. Nominal review changes nothing about the classification. Ask what the reviewer's actual workload and time per case are, whether they re-derive the decision from primary records, and how often they in fact disagree with the system. If nobody has measured the override rate, nobody has evidence that the review is real.
• Filing the classification and stopping. A correct classification recorded in an inventory is a starting condition, not a control. It records which obligations attach; it does not perform them, and it does not detect the day the system's scope quietly changed. Tie every classification to named obligations with owners and due dates, or the sheet becomes an artifact that describes a governance program nobody is running.

Practice Prompts

• Classify a system you know. Think of an AI system in your agency or one you are familiar with, and apply the five classification questions: what is the failure mode, who bears the consequences, how much human review is feasible, what is the domain sensitivity, and does it use sensitive information? Based on your answers, classify it as safety-impacting, rights-impacting, both, or neither. Document your reasoning, then list the aspects you remain uncertain about and what evidence would resolve each.
• Case study analysis. Classify each of these and document your reasoning: an AI system that predicts maintenance needs for federal buildings; an AI system that routes social security disability claims to appropriate specialists; an AI system that optimizes schedules for security checkpoint staffing; and an AI system that detects anomalies in federal financial transactions for investigation.
• Governance requirement design. Choose one system from the previous prompt. Based on your classification, design the governance approach: what validation testing is required before deployment, how frequently the system should be audited, what human review processes are needed, who should approve it before deployment, and what monitoring and alerting should be in place.
• Argue the other side. Take a system you classified as rights-impacting and write the strongest good-faith case a sponsor could make for a lower classification. Then write the evidence you would require before accepting it. This exercise is how a board learns the difference between an argument and a documented case.

Reflection

Take two or three minutes with a single question: what governance requirements would you want applied to an AI system that affects your own access to government services? Notice how your answer changes with what is at stake, a library card versus a disability determination, and with how much the system's output actually drives the outcome. Then ask whether the systems in your own agency's inventory would satisfy the standard you just set for yourself. That gap, between the protection you would want and the protection you currently require, is the most honest measure of whether your classification practice is doing real work.

Glossary

• Risk classification. The systematic categorization of AI systems based on the types and magnitude of impacts they could have if they fail or are misused.
• Safety-impacting system. An AI system whose failure could reasonably result in significant injury, severe harm to human health or safety, or loss of life.
• Rights-impacting system. An AI system whose failure, malfunction, or misuse could meaningfully impact civil rights, civil liberties, or privacy, including the potential for discrimination.
• Disparate impact. The effect of ostensibly neutral practices that fall disproportionately and negatively on members of a protected class, even where discriminatory intent is absent.
• Governance oversight. The systematic practices, reviews, and controls that ensure an AI system operates as intended, remains fair and safe, and complies with applicable policies and laws.
• Principal basis. The standard for whether an AI output drives a decision in practice, judged by real workflow and volume rather than by who signs the form.

Related Lessons

• NIST AI RMF: The GOVERN Function covers the framework layer that classification decisions plug into.
• NIST AI RMF: MAP, MEASURE, MANAGE takes the classification forward into impact mapping and measurement.
• Your Agency's AI Governance Structure addresses who holds the classification decision and how a board is constituted to make it.
• How AI Projects Differ from Traditional IT is the next step from classification into project planning and resource estimation.

Closing

The goal of risk classification is to match governance intensity to actual risk. You are not classifying systems to create bureaucratic burden; you are classifying them so the right kinds of failure get prevented by the right oversight mechanisms. Different organizations apply these frameworks with local variation, some agencies adding categories of their own or adding nuance to the two core ones. What is consistent across federal AI governance is the recognition that not all systems need the same oversight, and that the intensity and focus of governance should depend on what kind of failure the system could produce.

Classification is the foundation for everything that follows: validation approaches, monitoring frameworks, human oversight design, and incident response all sit on top of the category you assign here. Every new system goes through it, every major change triggers it again, and the practice gets easier with repetition once the criteria are written down and consistently applied. Inés's forty-one-row spreadsheet was not a compliance artifact. It was the document from which every other obligation in her agency's AI program was derived.

Key Takeaways

• Classification is the most consequential governance judgment. The category, safety-impacting, rights-impacting, both, or neither, determines every obligation that follows under M-24-10.
• Safety-impacting asks whether error could harm people or infrastructure. Rights-impacting asks whether error could cost someone access to a right, benefit, or equal treatment.
• The two categories call for different oversight, not just different amounts. Safety oversight asks "is this system working?" and centres on failure prevention and degradation detection. Rights oversight asks "is this system fair?" and centres on equitable outcomes, appeal rights, and demographic monitoring.
• A system can be both, and then both sets of obligations apply. Do not stop classifying once one category matches; the requirements layer rather than substitute.
• Accuracy is not fairness. A system can be 95% accurate and still discriminate, which is why aggregate performance tells you nothing about subgroup performance.
• "Principal basis" beats "human in the loop." If a caseworker in practice defers to the AI given real workflow and volume, the system is rights-impacting regardless of who signs.
• Under-classification is the real risk. Structural pressure pushes toward "advisory only." Reverse the burden of proof and require a documented case for any lower classification.
• Reclassify when the system changes. Scope growth, increased decision authority, a new affected population, an expanded use case, or measured degradation should each trigger reassessment, with an annual review as the floor.
• Document every classification and its reasoning. A one-page record per system, with the matched covered-use category and resulting obligations, becomes the spine of the agency's compliance program.

Frequently Asked Questions

Our system only makes recommendations. Is it still rights-impacting? Possibly, and the recommendation framing is not what decides it. Apply the principal-basis test to the workflow as it actually runs: what caseload does the reviewer carry, how much time do they have per case, do they re-derive the decision from primary records, and how often do they in fact depart from the system's output? If nobody has measured that last number, the agency has an assertion rather than evidence, and the presumption should run toward the higher classification.

What if a system is on the borderline between categories? Document the borderline. The board's job is not to produce a confident label; it is to produce a defensible record. Write down which covered-use categories were considered, which facts pulled each way, what the board decided, and what would change the answer. A future reviewer, including an inspector general, can work with that. A bare category with no reasoning behind it is what turns a defensible judgment into a finding.

Can classification be delegated to the system owner? The owner supplies the facts; the governance body makes the call. The structural problem is that the owner carries the cost of the higher classification and none of the cost of the harm, which is exactly the incentive that produces under-classification. Keeping the decision with a body that has no delivery deadline is what makes the burden-of-proof safeguard mean anything.

How does this classification relate to fairness testing requirements? Directly. A rights-impacting classification is what makes bias detection, demographic performance analysis, appeal mechanisms and ongoing bias-drift monitoring obligatory rather than discretionary. That is why sponsors argue about the label: the label is where the obligations attach.

Does classifying a system correctly mean it is now compliant? No, and this is the most common misreading. The classification determines which obligations apply. It does not perform the impact assessment, run the fairness tests, build the appeal route, or watch the monitoring dashboard. A correctly classified system with no owner assigned to those obligations is exactly as exposed as an unclassified one, with better paperwork.

How often should classifications be revisited? At minimum annually, and immediately on any of the five change triggers: scope expansion, increased decision authority, a change in the affected population, an expanded use case, or measured performance degradation in a subpopulation. The annual cadence exists precisely because the other triggers depend on somebody noticing and reporting a change, and quiet scope creep is the failure mode that most often defeats an otherwise sound classification.