Judge Marisol Reyes had presided over a state housing court for eleven years before an AI tool landed on her docket. A county housing authority had used a tenant-screening model to deny a Section 8 voucher renewal. The tenant's legal aid attorney argued the denial was unconstitutional: no human had reviewed the decision, the algorithm's reasoning could not be produced, and the training data was a trade secret the vendor refused to disclose. The county's lawyer had no answer to a simple question from the bench: "Can you show me how this decision was made?" Three months later, the agency settled, paid the tenant's costs, and quietly suspended the system. Nobody in the agency had ever imagined a courtroom would be the place their AI program died.

If you lead an agency, you are not just deploying technology. You are creating the factual record that a future court will examine. Every automated decision your agency makes is a potential exhibit, and the exhibit is assembled from artifacts that either exist at the moment of the decision or do not exist at all. This lesson is about understanding the legal terrain before a judge maps it for you.

Why Courts Are the Real Test of Government AI

Policy documents describe how AI should work. Courts decide what happens when it does not. That difference matters more in government than anywhere else, because a private company that deploys a bad model loses customers while an agency that deploys one takes something away from a person who had no choice about dealing with you. Three constitutional and statutory pressures converge on every consequential automated decision an agency makes, and they arrive together rather than in sequence. Understanding all three is what turns a legal risk you cannot see into a set of design requirements you can act on.

If your agency cannot explain a decision to a judge in plain language, you have not deployed a tool. You have created a liability you do not yet understand, and you will discover its size on someone else's schedule.

Due Process: The Central Battleground

When government takes away a benefit, a license, a job, or liberty, the Constitution requires fair procedure: notice of the decision, an explanation of the reasons, and a meaningful chance to contest it. A model that cannot explain itself collides directly with that requirement. It is not that the technology is disfavoured; it is that the constitutional obligation attaches to the decision regardless of what produced it, and an agency that cannot articulate its own reasoning has failed an obligation it would have had with a paper form.

The landmark example is from Idaho, where a state Medicaid agency used an algorithm to cut home-care budgets for people with disabilities. A federal court found the system unconstitutional in part because the state could not explain how the cuts were calculated. The math was deemed a trade secret. The court was unimpressed. Notice what the agency lost on: not the accuracy of the model, which was never really litigated, but its inability to produce the reasoning behind a specific reduction to a specific person's care. The remedy reached the whole program, not one claimant.

When AI Output Becomes Evidence

The second pressure is the evidentiary standard. When AI output enters a proceeding as evidence, courts ask whether it is reliable enough to be trusted. Forensic tools, risk-scoring instruments, and fraud-detection models have all been challenged on the ground that the opposing party cannot inspect the source code or the validation data. Several courts have ordered disclosure or excluded the evidence outright. An agency that cannot show how a score was produced may find that the score simply is not in the case, which can be worse for the agency than an unfavourable ruling on its merits.

This has a practical consequence people miss. Reliability here is not the vendor's benchmark. It is your validation record: what you tested, on whose data, under what conditions, with what error rates, reviewed by whom, and when. If that record was assembled during litigation it will be treated as advocacy. If it existed before the decision was made, it is contemporaneous evidence. The difference is not the content of the document; it is the date on it.

There is an asymmetry here that agencies consistently underestimate. The party best placed to demonstrate that a model is reliable is the party that built it, and that party is usually not you. The agency is the one in the courtroom, holding a system it did not construct, defending outputs it cannot reproduce, against an opponent entitled to test both. Every gap between what the vendor knows and what the agency can produce becomes the agency's gap the moment a challenge is filed, which is why disclosure obligations belong in the contract rather than in a support ticket you open under deadline.

The Administrative Record

The third pressure is the administrative record. Under most administrative law, an agency action can be struck down if it is "arbitrary and capricious," meaning the agency cannot show a reasoned basis for what it did. If the reasoned basis lives inside a model nobody can interrogate, your record has a hole in it, and the hole is where the reasoning was supposed to be. This is the quietest of the three pressures and often the most dangerous, because it does not require anyone to allege discrimination or a constitutional violation. It only requires a reviewing court to ask why, and to find no answer in the file.

The operational translation is straightforward once you accept it. For each decision the system informs, the record needs the inputs that were used, the version of the model that produced the output, the output itself, what the human reviewer did with it, and the reason communicated to the person affected. That is the file. If any element of it is not captured at the moment of the decision, it will not be there later, because a model that has since been retrained cannot reproduce what its predecessor concluded and a log that has rotated cannot be recovered by explaining that you meant to keep it.

The Four Questions a Court Will Ask

Reduce the legal complexity to four questions. If your program can answer all four with evidence that existed before the decision, your litigation exposure falls substantially. It does not fall to zero, because a well-documented system can still be a wrong one, but you move from defending a void to defending a record.

1. Can you explain the decision? Not the algorithm in the abstract, but this decision for this person. What inputs drove the outcome, and what would have changed it?
2. Did a human stay accountable? Was there a meaningful human review, or did a person merely rubber-stamp the output? Courts distinguish sharply between the two, and the distinction is drawn from your logs rather than your policy.
3. Can the affected person contest it? Were they told a system was used, given the reasons, and offered a real path to appeal to a human?
4. Can you prove the system works? Do you have validation testing, accuracy metrics by demographic group, and documentation of how you checked for bias before deployment?

A Worked Case: The Benefits Eligibility Engine

Return to a realistic version of Judge Reyes's courtroom. Suppose your agency runs a benefits eligibility engine that processes 40,000 applications a year and auto-denies roughly 6,000. A class action alleges the denials violate due process. The four questions decide the shape of the case, and each one turns on an artifact your team either kept or did not.

On explanation, opposing counsel demands the reason 6,000 people were denied. If your system produces a generic "does not meet eligibility criteria," you lose that point; a category code is not notice. If it produces a specific, individualised reason for each denial, such as an income threshold tied to a named wage record and its date, you have something to defend. That is not the same as winning. It means the argument moves from whether you gave notice to whether the reason you gave was correct, which is a much better argument to be having.

On human accountability, the plaintiffs will subpoena your audit logs. If the logs show a caseworker spent an average of nine seconds per denial, the court will treat the human review as a fiction, and no policy document saying review occurred will survive contact with that number. If the logs show genuine engagement with flagged cases and documented overrides, you have evidence that the review was real. Evidence, not proof: the logs establish that time and attention were spent, and the plaintiffs will still probe whether the reviewer had the authority and the information to reach a different conclusion.

On contestability, the question is whether denied applicants received a notice that named the automated system, stated the specific reason, and explained how to appeal to a person. A clear notice moves the dispute onto administrative ground where it can be resolved case by case, which is a far better posture than a constitutional challenge to the whole program. It does not cure a wrong decision or an untested model. It removes one defect, and it is the defect most cheaply removed before launch.

On proof of reliability, the plaintiffs' expert will run a disparate-impact analysis. If denials fall on one protected group at twice the rate of others and you have no pre-deployment bias testing to explain it, you are facing a discrimination finding with nothing to put against it. If you tested, documented and corrected, you have a contemporaneous record to defend with. A record is a much stronger position than silence, and it is still not an outcome. A disparity you found, explained and did nothing about reads worse in a courtroom than one you never looked for.

Notice that none of these defences can be built after the lawsuit is filed. They are built into the system before launch, which is exactly why this is a leadership problem and not a legal-department problem. Counsel can tell you what the standard is. Only the person who owns the deployment schedule can make the artifacts exist in time.

It is worth being clear about what a strong record buys you, because leaders routinely oversell it internally and then feel misled. It does not guarantee the agency wins. It changes what the case is about. Without a record, the dispute is whether your program can lawfully operate at all, and the remedy a court reaches for tends to be the whole system. With one, the dispute narrows to whether a particular decision was right, and the remedy narrows with it. That shift, from an existential challenge to an ordinary one, is the return on every hour spent documenting before launch.

Building a Litigation-Ready Program

The frameworks you already know map directly onto courtroom defensibility, and that mapping is the cheapest legal work available to you because the documentation is being produced anyway. The AI Risk Management Framework from the National Institute of Standards and Technology is voluntary and non-binding, so no court will treat it as the source of a duty. What it gives you is a discipline of documenting purpose, testing for validity, and tracing decisions, and that documentation is your evidence file when someone asks how a decision was reached.

The accountability framework from the Government Accountability Office stresses governance, data quality, performance monitoring, and the ability to be audited. An auditable system is not automatically a defensible one, but an unauditable system is reliably an indefensible one, because the agency cannot produce the record on which any defence would rest. Federal guidance to agencies on AI use, including the 2024 memoranda from the Office of Management and Budget, requires impact assessments and human alternatives for rights-affecting uses. Those impact assessments are the contemporaneous record a court wants to see, written before the harm rather than after the complaint.

Translate that into a standing practice. Treat every consequential AI decision as if it will be litigated, because eventually one of them will be. The practical version of that instruction is unglamorous: name the person who owns the evidence file for each system, agree what goes in it before launch, and review it on a schedule rather than when a letter arrives.

Legal Defensibility Checklist for AI Decisions

Use this checklist as a go/no-go gate before any AI system that affects a person's rights, benefits, or liberty goes live. Score each item: Yes, Partial, or No. Any No on items 1 through 6 should block deployment. Understand what a completed checklist does and does not tell you: it shows that the artifacts exist, not that the system is fair or the decisions correct. It is a floor for launching, never a certificate of lawfulness.

• Individualised explanation. The system produces a specific, plain-language reason for each adverse decision, not a generic category code.
• Notice of automation. Affected people are told in writing that an automated system was used in their case.
• Human review path. A documented, meaningful human review exists for adverse and contested decisions, with logged review time and override authority.
• Appeal to a person. There is a clear, accessible route to have a human reconsider the decision.
• Pre-deployment validation. Accuracy and error rates are measured, including separately by protected demographic group, and documented with dates.
• Bias testing on record. A disparate-impact analysis was run before launch and is repeated on a schedule, with results retained.
• Discoverable record. Inputs, model version, and outputs for each decision are logged and retrievable for the full records-retention period.
• Vendor disclosure clause. The contract requires the vendor to disclose, under protective order if needed, the information you would need to defend a decision in court. "Trade secret" cannot be the agency's defence.
• Accessibility. Notices and appeal interfaces meet federal accessibility requirements under Section 508 so the contest process is genuinely usable.
• Counsel sign-off. Agency legal counsel reviewed the deployment against due-process and evidentiary standards and signed the record.

The Trade-Secret Trap

One pattern sinks government agencies repeatedly: the vendor's intellectual property becomes the agency's legal liability. A vendor sells you a model, calls its inner workings a trade secret, and refuses disclosure even to a court. When a plaintiff challenges a decision, the agency cannot explain its own action and the vendor is not the party being sued. You inherit the loss. This is the same failure that ended Judge Reyes's case and the same one that decided the Idaho matter, which should tell you how routinely it recurs.

The fix is contractual and must happen at procurement, not at trial. Require in writing that the vendor support legal defence by producing model documentation, validation data, and decision-level explanations, subject to a protective order that shields genuine trade secrets while still satisfying the court. Negotiate it while you still have leverage, which is before award. If a vendor will not agree to that, they are selling you litigation risk dressed as software, and the price you were quoted does not include it.

Write the clause so it survives the moment it is needed. Name what the vendor must produce, which is model documentation, validation data and decision-level explanations rather than a vague duty to cooperate. Say that the obligation runs for as long as decisions made with the system remain contestable, not only during the contract term. State that a protective order is an acceptable mechanism, so refusal cannot be dressed up as confidentiality. A clause that leaves any of those three open is a clause your counsel will be arguing about at the worst possible time.

Anti-Patterns to Avoid

• Treating the checklist as the outcome. Every box ticked means the artifacts exist. It does not mean the model is accurate, the reasons are correct, or the appeal path works when a real person tries to use it. Test the appeal path with someone who did not build it.
• Notice as a cure-all. A well-drafted notice removes a procedural defect. It does not make an unvalidated system valid, and a notice that names the system while stating a generic reason gives a court two problems instead of one.
• Human review that exists only on the org chart. If reviewers lack the time, the information or the authority to reach a different answer, the logs will say so, and the audit trail you built for defence becomes the plaintiffs' best exhibit.
• Building the evidence file after the complaint. Documentation assembled during litigation is treated as advocacy. The same document written before the decision is contemporaneous evidence. Only the date differs, and the date is the entire point.
• Finding a disparity and filing it. Testing you performed and then ignored is worse than testing you never ran. If a test surfaces a gap, the record must show what you did next.
• Letting "the vendor says it is compliant" stand as the record. The vendor is not the respondent. Compliance claims that you cannot independently evidence are not evidence.
• Treating this as legal's problem. Counsel can state the standard. Only the programme owner can make the artifacts exist before launch, and after launch the window has closed.

Practice Prompts

• Pick the highest-stakes automated decision your agency makes and write the individualised explanation a person would receive. If you cannot write it without describing the model, you cannot give notice either.
• Pull the audit logs for one month of adverse decisions and compute the average review time per decision. Ask yourself how that number reads aloud in a hearing before you decide whether it is acceptable.
• Run the four questions against a system already in production. For each question, name the specific document or log that answers it and confirm it exists today. Anything you would have to create is a gap.
• Take one active vendor contract and find the clause that would compel disclosure of validation data under a protective order. If it is absent, draft the language you want in the next renewal.
• Have a colleague who did not build the system attempt the appeal path end to end, including the accessibility route. Record where it breaks.
• Write the one-page memo your General Counsel would need on the morning a complaint arrives about your largest system. Note every fact in it you could not currently evidence.

Reflection

Think about the systems your agency runs today and ask which of them could survive the question Judge Reyes asked from the bench. Not in principle, but on Tuesday, with the actual files your team could retrieve in an afternoon. The gap between what you believe your program does and what you could evidence is the real measure of legal exposure, and most leaders discover it is wider than they expected.

Then consider the incentive problem underneath. Documentation is unpaid work with no visible payoff until the day it matters enormously, and the people who cut it are usually under real delivery pressure rather than acting carelessly. What would have to change about how your program is resourced and reviewed for the evidence file to be built on time rather than reconstructed under subpoena?

Glossary

• Due process. The constitutional requirement of fair procedure, including notice, reasons and a meaningful chance to contest, when government takes away a benefit, licence, job or liberty.
• Arbitrary and capricious. The administrative-law standard under which an agency action can be struck down where the agency cannot show a reasoned basis for what it did.
• Administrative record. The body of material an agency relied on in taking an action, and the file a reviewing court examines.
• Disparate-impact analysis. Testing whether a decision process produces materially different outcomes for a protected group, independent of intent.
• Protective order. A court order limiting how sensitive material may be disclosed and used, allowing genuine trade secrets to be examined without becoming public.
• Individualised explanation. A specific, plain-language reason for one person's adverse decision, as distinct from a category code or a description of the model.
• Contemporaneous record. Documentation created at the time of the decision rather than assembled afterwards, which is what gives it evidentiary weight.
• Meaningful human review. Review by a person with the time, information and authority to reach a different conclusion, as opposed to a rubber stamp.

Related Lessons

• Rights-Impacting and Safety-Impacting AI Safeguards covers the notice, human alternative and appeal obligations this lesson treats as legal exposure.
• Algorithmic Impact Assessments covers producing the pre-deployment assessment that becomes your contemporaneous record.
• GAO AI Accountability: Four Principles in Practice works through the governance, data, performance and monitoring framework referenced here.
• Working with AI Vendors and Contractors covers the procurement leverage that closes the trade-secret trap.
• Accountability Frameworks for AI Failures covers what an agency owes when a system has already caused harm.
• When Government AI Goes Wrong covers the failure record behind these legal standards.
• AI Regulatory Design and Legislative Framework Development cover the rule-making side of the same problem.

Closing

The county in Judge Reyes's courtroom did not lose because its model was bad. Nobody ever established whether it was. It lost because when a judge asked how a decision about a specific person had been made, the agency had no answer and no right to obtain one. Every artifact that would have saved it was cheap to produce before launch and impossible to produce afterwards.

That is the whole discipline. Decide, before a system goes live, what you would need in your hand on the day someone challenges a decision it made, and then make sure that material exists and is retrievable. Do that and litigation becomes an ordinary risk you manage. Skip it and a courtroom becomes the place you find out what your program actually was.

Key Takeaways

• Courts are the true test. Policy describes how AI should work; a judge decides what happens when it does not, and the administrative record is your only defence.
• Due process is the central battleground. Government decisions that take away benefits, licences or liberty require notice, a real explanation and a meaningful chance to contest, which a system you cannot interrogate is unable to provide.
• Three pressures arrive together. Due process, the evidentiary standard for AI output in a proceeding, and the reasoned-basis requirement of the administrative record.
• Answer the four questions. Can you explain the specific decision, keep a human accountable, let the person contest it, and prove the system works? Evidence on all four substantially reduces exposure without eliminating it.
• Defences are built before launch. Individualised explanations, logged human review, appeal paths and dated bias testing cannot be manufactured after a complaint is filed, and the date is what gives them weight.
• The frameworks are your evidence file. The NIST framework is voluntary and creates no duty, but the documentation it produces, alongside GAO's accountability principles and OMB impact assessments, is the contemporaneous record courts want to see.
• Trade secret is not a defence. Require vendors by contract to disclose what you need to defend a decision in court, under protective order where necessary, or you are buying their litigation risk.
• Treat every decision as a future exhibit. If you cannot explain an automated decision to a judge in plain language, you have created a liability, not deployed a tool.

Frequently Asked Questions

Does a human reviewer protect us from a due process claim?

Only if the review is real, and the court will decide that from your logs rather than your policy. Reviewers who lack the time, the information or the authority to reach a different conclusion produce a record that looks like rubber-stamping, and an audit trail showing seconds per decision is more damaging than having no reviewer at all, because it documents the fiction. Design the review so that overrides actually happen and are recorded when they do.

Can we rely on our vendor's compliance certification?

No. The vendor is not the party being sued, and a certification you cannot independently evidence is not evidence. What protects you is a contractual right, negotiated before award, to obtain model documentation, validation data and decision-level explanations, disclosed under a protective order where genuine trade secrets are involved. Without that clause the vendor's intellectual property becomes your inability to explain your own action.

Is the NIST AI Risk Management Framework legally required?

No. It is voluntary and non-binding, and no court will treat it as the source of a duty. Its value here is practical: the artifacts it asks you to produce, covering purpose, validity testing and traceability, are the same artifacts that answer a judge's questions. Follow it because it builds the evidence file efficiently, not because compliance with it is a defence in itself.

We found a disparity in testing. Should we document it?

Yes, and document what you did about it in the same breath. Testing you ran and then ignored reads worse in a courtroom than testing you never ran, because it establishes that you knew. The defensible pattern is a dated finding, a documented analysis of cause, a remediation with an owner, and a re-test showing the effect. Suppressing the finding converts a fixable problem into a credibility problem.

Does giving notice that AI was used solve the problem?

It removes one defect, and it is the cheapest defect to remove. Notice moves a dispute onto administrative ground where it can be handled case by case rather than as a challenge to the whole programme. It does not validate an untested model, correct a wrong decision, or substitute for an appeal route that a real person can actually use. Treat notice as necessary rather than sufficient.

How long do we have to keep decision records?

Long enough to retrieve inputs, model version and outputs for any decision still capable of being challenged, which in practice means aligning your logging to your agency's records-retention obligations rather than to your engineering team's storage preferences. Model outputs used in decisions are agency records. Confirm the applicable retention schedule with your records officer and counsel before you set a log rotation policy, because a deleted record is indistinguishable from a record that never existed.