Tamara Esposito got the call on a Tuesday in February. A constituent named Delores had been denied Medicaid renewal by the state's new automated eligibility system. The denial letter cited an income threshold exceeded, but Delores's income had not changed. She had been approved every year for eleven years. The system had reassigned her to a different income bracket based on a zip code level data estimate. There was no mechanism to contest the determination. There was no appeal path. There was no human being named anywhere on the denial letter who was responsible for the decision. "A letter told her she no longer qualified for healthcare," Tamara said at the next agency leadership meeting, "and there was no one on our side of that letter accountable for being wrong."

The accountability gap in automated decisions

Think of algorithmic accountability like a chain of custody in a legal case. Every piece of evidence must have a documented handler at every point: who collected it, who stored it, who transferred it, and who presented it. If the chain breaks, the evidence is compromised and the case falls apart, not because anyone proved the evidence was wrong but because nobody can prove it was handled right.

Automated decision making creates the same problem in a different medium. A decision that harms a person, whether a wrongful denial, an incorrect assessment or a biased determination, needs a documented chain of responsible parties running from the design stage through deployment and into the citizen's hands. Without that chain, accountability dissolves into the space between the AI vendor, the agency's technology office and the program office. Nobody is wrong because nobody is named. Every party gave an honest answer about their own part, and the person holding the letter still has no one to talk to.

Tamara's agency had the system. It did not have the chain. That distinction is the whole subject, and it is worth being precise about why it matters legally as well as morally: when a determination is challenged, an agency that cannot say who was responsible for what cannot explain the decision, cannot correct it at the source, and cannot credibly promise it will not recur.

Accountability registers

An accountability register is a formal document naming a responsible person for each function in an AI system's lifecycle. It is the organizational equivalent of the evidence chain, and it answers one question for every key function: who is accountable if this goes wrong. Not which office. Which person, by name and title, reachable, and aware that the name is theirs.

A complete register for a benefits eligibility system covers at minimum the six functions below. Each of them exists because a specific kind of failure has an owner only if somebody holds that role.

• System owner. The program executive who authorized the use of AI for this decision and is accountable for its outcomes.
• Model steward. The technical lead responsible for the model's accuracy, fairness and ongoing performance.
• Data owner. The person accountable for the quality, completeness and legal basis of the training and operational data.
• Equity reviewer. The civil rights or equity officer responsible for ongoing disparate impact monitoring.
• Citizen liaison. The program staff contact named on every automated decision notice.
• Appeal adjudicator. The named individual or panel authorized to review contested determinations.

The register is a living document. It has to be updated whenever named individuals change roles, leave, or transfer accountability, and the update has to be somebody's assigned duty rather than a good intention. Tamara's agency now treats an empty cell in the register as a system pause trigger: if no one is named as accountable for a function, the system does not run that function. That rule is severe on purpose. It converts a documentation gap, which agencies tolerate indefinitely, into an operational consequence, which agencies fix quickly.

What a register does not do

A register names people. It does not, by itself, make them accountable, and this is where the instrument is most often oversold. A name in a cell is accountability only when the named person has the authority to change the thing they are named for, the information to know when it is going wrong, and the standing to stop it. A model steward with no ability to halt a deployment is a designated recipient of blame. The test is simple and worth applying to your own register: for each name, what could that person actually do on the day the system started producing wrong answers, and how would they find out.

The same caution applies to the wider family of accountability artifacts. An impact assessment documents anticipated harms; it does not prevent them, and completing one has never made an inaccurate system accurate. A transparency report tells the public what the agency chose to disclose about a system it chose to describe. An appeal path creates a route for the people who find it, use it and persist. Each is genuinely valuable and none of them discharges the underlying obligation to make correct decisions and to fix incorrect ones. Treat every artifact in this lesson as evidence that a duty was taken seriously, never as evidence that it was satisfied.

An AI system with no named accountable owner is an unmanaged risk. If nobody is responsible when the system wrongs someone, the agency is responsible by default, and that accountability tends to surface in litigation rather than in a management meeting.

Appeal mechanisms

The legal floor

Federal due process requirements, the constitutional protections preventing government from depriving people of life, liberty or property without fair procedures, already require an opportunity to contest an adverse government decision. Automating the decision does not eliminate the obligation. What changes is the mechanism, and mechanisms designed for a caseworker's determination frequently do not function when the determination arrives from a system.

For benefits and eligibility systems, the minimum legally required appeal path includes notice of the determination in plain language, a statement of the reason, information about how to contest the decision, and access to a human reviewer. State agencies administering federal programs carry additional obligations under their grant agreements. If your agency's automated system does not currently produce all four of those elements on every adverse determination, it is out of compliance regardless of whether the AI itself is accurate. That is the source's formulation and it is the strict reading; the specific requirements that attach to your program come from your program's own statute, regulations and grant terms, so have counsel confirm what applies to you rather than working from a general statement in a training lesson.

Designing the appeal path

Appeal paths should be designed for the population that will actually use them, not for an idealized citizen who reads carefully, has reliable internet access and knows what an algorithmic determination is. The people most likely to be wrongly denied are frequently the people least equipped to challenge it, and a path that is technically available but practically unusable produces a low appeal rate that agencies then misread as evidence of accuracy. For Tamara's Medicaid population, the appeal path required five things.

1. A denial notice in plain English, below eighth grade reading level, in the top five languages spoken in the state, delivered by mail and email simultaneously.
2. A phone number answered by a human within three business days.
3. A web form that did not require creating an account in order to submit a challenge.
4. A 90 day appeal window, longer than the 30 day window the agency had previously treated as standard, to account for constituents managing health crises.
5. Automatic interim benefits continuation while the appeal is pending, to prevent harm during review.

Those five are Tamara's agency's design decisions rather than universal requirements, and the windows in particular vary by program and by statute. Take them as a worked example of designing for the real population, and check your own program's rules for the numbers that bind you. The design principle transfers even where the specifics do not: every element on that list exists to remove a barrier that was causing wrongly denied people to give up.

One element on that list deserves separate attention, because it is the one agencies believe they have already satisfied. A reason is not the same as a reason code. Delores's letter cited an income threshold exceeded, which is a true statement about what the system concluded and tells her nothing she can act on: not which income figure was used, not where it came from, not that it was an estimate rather than a reported number. A reason that lets a person contest a determination has to name the specific input the determination rested on and where that input originated, because the whole point of contesting is to say that one of those facts is wrong. Test your own notices against that standard by reading one as if you disagreed with it.

The human review standard

When a person appeals, the human reviewer must have authority to override the determination. This sounds obvious and is frequently not built. Many agencies have deployed systems in which the reviewer can annotate the output but not reverse it, or can reverse it only by escalating to someone who never sees the case file. That does not satisfy due process. The reviewer must be able to make an independent determination from the underlying facts, rather than assessing whether the algorithm followed its own rules correctly.

Authority alone is not sufficient either. A reviewer with formal override power, a caseload that leaves no time to look past the summary, and a default assumption that the system is usually right will affirm nearly everything, and the appeal path will still function as a rubber stamp with better paperwork. Give reviewers the underlying facts rather than the system's summary of them, measure the override rate and look hard at a rate near zero, and make sure nobody's performance rating rewards agreeing with the model.

What the appeal path costs

Appeal mechanisms cost money to build and to operate, and the cost belongs in the business case rather than in a surprise supplemental request. Tamara's agency estimated a one time implementation cost of $220,000 and an annual operating cost of $85,000 in additional caseworker hours. Those are two different quantities measured on two different clocks, and they should be carried separately in every document: one capital figure, one recurring figure. Collapsing them into a single range is the most common way this number gets misstated, and an executive who approves the implementation without the recurring line has funded a mechanism that will quietly degrade in its second year.

Both figures are one agency's estimate for one system, so use them as a template rather than a benchmark. The point is structural. An appeal mechanism is not a feature to be traded away when the build is over budget. It is a legal requirement and a line item, and a procurement document for an automated decision system that does not price it has understated the cost of the system rather than found a cheaper one.

Remediation procedures

Remediation is what happens after the system has wronged someone and the appeal has confirmed it. A remediation procedure answers four questions: what gets corrected, how fast, who is notified, and how similar future errors are prevented. Agencies routinely have an answer to the first and no answer to the other three.

Individual remediation

Individual remediation addresses the specific person's harm. For Delores this meant restoring Medicaid coverage retroactively to the date of the wrongful denial, paying medical bills incurred during the coverage gap, and issuing a written apology identifying what went wrong. Timelines should be set in advance rather than negotiated case by case: Tamara's agency commits to restoring benefits within five business days of a confirmed wrongful denial and to completing financial remediation within thirty days. Setting the timeline in advance matters because the alternative is a remediation speed that depends on how much attention the individual case has attracted.

Systemic remediation

Individual remediation leaves the larger question untouched: how many other people did the same error affect. Systemic remediation requires a lookback audit, meaning a retroactive review of every determination made by the same system while the error was active. For the zip code income estimation error that reached Delores, the lookback found 847 other affected individuals across a 14 month period. Those people were surfaced by the audit rather than by the appeal path, which is the part worth sitting with: the appeal path caught the error once, and the audit found everyone the appeal path had missed.

A systemic remediation plan should define the scope of the lookback, the method for identifying affected individuals, the outreach strategy for people who never appealed, the timeline for individual remediation at scale, and the technical fix preventing recurrence. The plan goes to the agency head and, where a statute requires it, to the relevant legislative oversight committee. Outreach is the element most often skipped and the one that determines whether systemic remediation is real, because people who were wrongly denied and gave up will not come back on their own.

When the model belongs to a vendor

Most agencies do not build these systems. That does not move the accountability, but it does change what has to be secured in advance, because the information the register depends on sits inside somebody else's product. Decide before award who can explain an individual determination and how fast, what documentation of model logic and version history the agency receives, whether the agency can obtain the specific inputs and outputs for a contested case, and what happens to all of that when the vendor updates the model. An appeal adjudicator who cannot obtain the inputs behind a determination cannot conduct an independent review, however clearly the register names them.

Governance across that boundary needs the same treatment as governance inside the agency: how joint decisions get made, how disagreements are resolved, and who is authorized to speak for each party when a determination is challenged publicly. What the agency cannot do is contract out the answer to the citizen's question. The vendor is accountable to the agency under a contract. The agency is accountable to the person holding the letter, and that relationship is not delegable regardless of what the statement of work says.

From design to deployment

Accountability mechanisms work only when they are embedded from the start rather than bolted on after a crisis. Tamara's agency now requires every AI procurement and development project to complete an accountability framework review as a condition of moving from design to deployment. The checklist covers named register entries for all six functions, a documented appeal path meeting due process requirements, a defined individual remediation timeline, a systemic remediation trigger and lookback protocol, and an annual accountability review date.

The checklist is signed by the program executive, the agency's legal counsel and the equity officer. No signature, no deployment. This is not bureaucracy for its own sake, and it is worth defending in exactly those terms when a schedule is under pressure: the checklist is the chain of custody that makes accountability real rather than aspirational, and the moment it becomes a formality is the moment the agency has returned to the position Tamara found herself in on that Tuesday in February.

Assessing readiness and resourcing the work

Before standing this up, assess where the agency actually is. Current capability: which deployed systems make determinations about people, which of them have a named accountable owner today, and where the largest exposure sits. Organizational readiness: is the agency prepared for an equity officer to pause a function over an empty cell, and what would make that stick. Stakeholder alignment: program leadership, legal counsel, the equity office, the technology office and the vendor each have interests here, and they are not the same interests. Resource constraints: what staff time and appeal capacity genuinely exist, given that an appeal mechanism nobody has resourced is the failure this lesson is about.

Then plan it. Goal clarity: what accountability means concretely in this agency, and what success looks like to someone outside the technology office. Action planning: which systems get a register first, in what sequence, with what resources. Risk management for the effort itself: the predictable failure is a register built once and never maintained, so design the maintenance before the first register is complete. Stakeholder engagement: the people who will operate the appeal path should help design it, because they know which barriers actually stop constituents and the design team does not.

Measuring whether accountability is real

Accountability that is never measured reverts to paperwork. Decide what you will count, who collects it, who receives it and what the report can change. The useful measures are mostly uncomfortable ones: how many determinations were appealed, how many appeals were upheld, how long the average appeal took from submission to decision, the override rate on human review, how many register cells are currently empty, how long remediation actually took against the committed timeline, and how many lookback audits were triggered and completed.

Read those numbers against each other rather than individually, because each is misleading alone. A low appeal rate can mean the system is accurate or it can mean the appeal path is unusable, and the two are distinguishable only by looking at who is appealing and comparing that against who is being denied. A near zero override rate can mean the model is right or it can mean reviewers are affirming by default. Send the results to the executive who signed the deployment checklist, not only to the office that runs the system, and let each review end in a decision recorded with its reasoning.

Sustaining it past the first crisis

These mechanisms are built under the pressure of a specific failure and then have to survive its absence. Scaling asks how a register and an appeal path for one system become standard practice across a portfolio, which mostly means templates, a defined place where registers live, and inclusion in the acquisition path so nobody has to remember. Funding asks who pays for appeal capacity and lookback audits in a normal year, when no crisis is justifying it. Organizational embedding asks whether this is a role or a person.

Personnel change is the specific threat to a register, because the register is a list of individuals. The countermeasure is procedural: tie register updates to the departure process, so a name cannot leave the agency without the cell being reassigned, and review the whole register on a fixed cycle regardless. Write down why each function is on the register and what failure it exists to catch, because a successor inheriting a list of names without reasons will eventually consolidate roles for efficiency, and the role that gets consolidated away is always the equity reviewer.

What happened next

Delores was restored, retroactively, with her coverage gap bills paid and a letter that said what had gone wrong. That took the agency's newly defined five business days once the appeal was confirmed. The 847 people the lookback identified took considerably longer, because most of them had to be found and contacted rather than simply corrected, and a number of them had already stopped expecting anything from the agency.

What changed structurally was smaller and more durable than the remediation. Every automated determination now carries a named person. Every adverse determination carries a reason, a route and a human at the end of it. An empty cell in the register pauses a function. A confirmed wrongful denial triggers a lookback rather than a case fix. And no AI system reaches production without three signatures from people who can individually refuse. None of that required new technology. It required deciding that a letter leaving the agency with nobody's name on it is not an acceptable thing for the agency to send.

Anti-patterns

• The unowned determination. An automated decision notice with no named person on it, so the recipient has an outcome and no one to ask about it, and the agency has no internal owner when it turns out to be wrong.
• The register as discharge. Treating a completed register as accountability. A name is accountability only when that person has authority to change the thing, information to know it is going wrong, and standing to stop it.
• The artifact as absolution. Treating an impact assessment, a transparency report or a published appeal policy as evidence the obligation is satisfied. Each documents diligence; none makes a wrong determination right.
• Review without reversal. Building human review that can annotate but not override, or that can override only by escalating to someone who never sees the case file. That does not meet a due process standard.
• The rubber stamp reviewer. Formal override authority combined with a caseload of minutes per case, the system's summary in place of the underlying facts, and a default assumption that the model is right. Watch the override rate.
• Appeal path for the idealized citizen. Requiring an account, a single language, reliable internet and a short window from a population managing health and income crises, then reading the resulting low appeal rate as accuracy.
• The unpriced mechanism. Omitting appeal build and operating costs from the business case, then trading the mechanism away when the build runs over, which understates the cost of the system rather than reducing it.
• The recurring cost that vanished. Funding the one time implementation and never the annual operating cost, so the mechanism degrades in its second year while remaining formally in place.
• Individual fix, systemic silence. Correcting the person who complained and never asking how many others the same error reached, which leaves the agency's exposure entirely determined by who happened to have the persistence to appeal.
• Lookback without outreach. Identifying affected people and waiting for them to come forward. People who were wrongly denied and gave up do not come back on their own.
• Accountability contracted out. Assuming that a vendor's responsibility under the contract answers the citizen's question. The vendor is accountable to the agency; the agency is accountable to the person holding the letter.
• The checklist that became a formality. A deployment gate whose signatures are collected after the deployment decision has already been made, which is the same as having no gate while feeling protected by one.

Practice prompts

1. Take one automated determination your agency issues and find the name on it. If there is none, draft the notice language that would put one there and identify who would have to agree.
2. Build the accountability register for one deployed system across the six functions. For each name, write what that person could actually do on the day the system started producing wrong answers, and how they would find out.
3. Walk your own appeal path as a constituent would: find the notice, read it, locate the contact route, and attempt the form. Record every point at which a person managing a crisis would stop.
4. Price the appeal mechanism for one system as two separate figures, one time and recurring, and check whether both appear in the current business case.
5. Write the systemic remediation plan for a hypothetical confirmed error: lookback scope, identification method, outreach strategy for people who never appealed, timeline at scale, and the technical fix.
6. Pull the override rate on human review for one system. If it is near zero, work out which of the two explanations applies and what evidence would distinguish them.
7. Review one AI contract for the appeal-critical terms: who can explain an individual determination, how fast, what case level inputs and outputs the agency can obtain, and what happens when the model is updated.
8. Run the readiness assessment: which systems make determinations about people, which have named owners today, what appeal capacity exists, and who would have to agree to pause a function over an empty cell.

Reflection

The detail worth staying with in Delores's case is not the zip code estimate. Estimation errors are ordinary and every system has them. The remarkable part is that a letter left a government agency, arrived at a person's home, ended her healthcare coverage, and carried no name, no reason she could act on, and no route back. That was not a technical failure. Every element of it was a design decision somebody made, or more precisely a decision nobody made, because the notice template was inherited and the accountability question was never anyone's assignment. Think about the adverse determinations your own agency issues automatically. Ask who is named on them, what a recipient who believes the decision is wrong is supposed to do in the next hour, and whether anyone in your organization has ever tried it.

Glossary

• Algorithmic accountability. The documented chain of responsible parties running from a system's design through deployment to the person affected by its output.
• Accountability register. The formal document naming a responsible individual for each function in an AI system's lifecycle, maintained as people change roles.
• System pause trigger. A condition, such as an unfilled register cell, that stops a function from running until it is resolved.
• Adverse determination. A decision that denies, reduces or terminates a benefit, status or entitlement for a specific person.
• Due process. The constitutional requirement that government follow fair procedures before depriving a person of life, liberty or property.
• Appeal path. The complete route by which an affected person contests a determination, including notice, reason, contact method, window and human review.
• Interim continuation. Maintaining a benefit while an appeal is pending, so that the review does not itself cause the harm it is meant to examine.
• Override rate. The proportion of reviewed determinations a human reviewer reverses, read as a diagnostic of whether review is substantive.
• Individual remediation. Correcting the specific harm to the specific person, including retroactive restoration, consequential costs and a written explanation.
• Systemic remediation. Correcting everyone affected by the same error, through a lookback audit, proactive outreach and a technical fix preventing recurrence.
• Lookback audit. A retroactive review of all determinations made by a system during the period an identified error was active.
• Disparate impact monitoring. Ongoing measurement of whether a system's outcomes differ across protected classes, reported by subgroup rather than in aggregate.

Related lessons

• Accountability Frameworks for AI Failures covers what happens organizationally after a failure is confirmed.
• Algorithmic Impact Assessments covers the structured pre-deployment analysis the register's design review should draw on.
• Rights-Impacting and Safety-Impacting AI Safeguards covers the classification that determines how much of this applies to a given system.
• Human-in-the-Loop: Design and Implementation covers building review that is substantive rather than nominal.
• Transparency: Citizens' Right to Know covers the notice and disclosure side of the same obligation.
• Public Reporting and Algorithmic Transparency covers what an agency publishes about systems that make determinations.
• Algorithmic Fairness in Government covers the disparate impact measurement the equity reviewer role depends on.
• The Blueprint for an AI Bill of Rights covers the non-binding blueprint issued by the White House Office of Science and Technology Policy, which sets out principles including notice and human alternatives.

Closing

Everything in this lesson reduces to a single question asked from the constituent's side of the envelope: if this decision is wrong, who fixes it, and how would they know. An accountability register answers who. An appeal path answers how a person reaches them. Remediation answers what happens when they are right, and a lookback answers what happens for everyone who never wrote in. Build all four and the agency can defend its automated decisions, correct them at the source, and say honestly that it will not happen the same way again. Build none of them and the agency is still accountable, just not to anyone in particular until it is to a court.

Key takeaways

• Name everyone in the chain. An accountability register identifies by name and title who is responsible for each function from design through citizen impact, and an empty cell is a system pause condition rather than a documentation backlog item.
• A name is accountability only with authority and information. For each person on the register, confirm what they could actually do on the day the system goes wrong, and how they would find out.
• Automating a decision does not waive due process. Every adverse automated determination should carry plain language notice, the reason, how to contest it, and access to a human reviewer, and a system missing any of them is out of compliance regardless of its accuracy. Confirm your own program's specific requirements with counsel.
• Human reviewers must be able to override. A reviewer who can only annotate, or who must escalate to someone who never sees the file, does not satisfy due process. Then watch the override rate, because authority without capacity produces a rubber stamp.
• Design the appeal path for the people who will use it. Plain language, multiple languages, phone access, no account requirement, a window that accounts for crisis, and interim continuation. A low appeal rate is as easily evidence of an unusable path as of an accurate system.
• Price the mechanism as two numbers. Tamara's agency estimated $220,000 one time and $85,000 annually for one system. Carry the capital and recurring figures separately in every business case, because funding only the first leaves a mechanism that decays in year two.
• No artifact discharges the obligation. Registers, impact assessments, transparency reports and published appeal policies are evidence of diligence, never evidence that determinations are correct.
• A confirmed error triggers a lookback. Systemic remediation means retroactive review of every determination made while the error was active, proactive outreach to people who never appealed, and a technical fix. The audit finds everyone the appeal path missed.
• Accountability is not contractible. Secure explanation rights, case level data access and update notification from the vendor before award, because the agency answers to the person holding the letter no matter who built the model.
• Make the framework a deployment gate. Signed review of the register, appeal path and remediation protocols before production, with signatures collected before the deployment decision rather than after it.

Frequently Asked Questions

Our appeal rate is very low. Does that mean the system is accurate?

It might, and it is equally consistent with an appeal path nobody can use. The two are distinguishable with evidence rather than assumption: compare who is appealing against who is being denied, and look for populations that appear in the denial data and not in the appeal data. Then walk the path yourself as a constituent would. Agencies that do this typically find at least one hard barrier, most often an account requirement, a single language notice, or a window that expires while someone is in hospital.

The vendor says the model's logic is proprietary. How do we explain a determination?

Settle this before award, because after award your leverage is a renewal conversation. What the adjudicator needs is not usually the model's internals: it is the specific inputs used for this person, the output and any score or reason code, the model version in effect on that date, and documentation of what the system was designed to weigh. Those are contractible without exposing anything genuinely proprietary. If a vendor cannot commit to providing them for a contested case, you are buying a system whose determinations you cannot defend.

Does a completed impact assessment mean we have met our accountability obligations?

No. An impact assessment documents the harms you anticipated and what you planned to do about them, which is valuable and is a different thing from the system being correct or the affected person having a route to challenge it. The same applies to a transparency report and a published appeal policy. Each is evidence of diligence. None of them makes a wrong determination right, and treating any of them as discharge is how agencies end up with excellent documentation and an unfixed error.

How far back should a lookback audit go?

As far back as the error was active, which is a factual question about the system rather than a matter of appetite. Establishing that window is the first task of the audit: when did the responsible logic or data source enter production, and when was it corrected. Where the start date is genuinely undeterminable, say so explicitly in the plan and choose a defensible boundary with counsel, rather than quietly picking a period that produces a manageable number of affected people.

We are a small agency with no dedicated equity officer. Who does that role?

Someone must hold it, and the honest options are a named individual with the responsibility added explicitly to their duties, or a shared resource across agencies. What does not work is leaving the cell blank and intending to address it, because the disparate impact monitoring role is the one that quietly disappears when roles are consolidated for efficiency, and it is the one whose absence surfaces years later in a pattern nobody was watching for. Write down what the role exists to catch, so a successor cannot consolidate it away without noticing what they are removing.

What is the single first step if we have none of this today?

Take the automated determinations your agency currently issues and check whether each carries a name, a reason and a route. That audit is cheap, it requires no new system, and it tells you exactly how exposed you are. Almost every agency that runs it finds at least one notice going out with none of the three, which is the same position Tamara's agency was in on the day Delores called, and unlike most findings in this field it can be corrected quickly.