Dr. Renata Okafor, chief data officer for a mid-sized state health department, had spent eighteen months building a model that flagged Medicaid members at risk of an avoidable hospital stay. It worked. Readmissions in the pilot county fell by an estimated 11 percent. Then a state university researcher asked a question her team could not answer: "How do you know it isn't just flagging poor people?" Renata had no published study, no external validation, and no co-author who could defend the method in a peer-reviewed venue. The model was good. The evidence was an internal slide deck. When a journalist eventually asked the same question, she had nothing durable to point to.

That gap is what academic and research collaboration closes. For an agency leader, partnering with universities and research institutes is not about prestige. It is about producing evidence that survives staff turnover, records requests, litigation, and the next administration. This lesson treats collaboration as an instrument of governance. It carries Renata's readmission model through the decisions so the moves stay concrete, and it works through the federal instrument landscape that a larger agency has to navigate to get the same result at scale.

Why External Evidence Is a Governance Asset

An internal evaluation lives and dies with the team that wrote it. A peer-reviewed study, or an independent evaluation by a credible research partner, becomes an asset the agency owns regardless of who stays. It gives you defensible answers, because frameworks such as the GAO AI accountability principles ask agencies to document performance, monitor for drift, and validate results independently, and an external study is the cleanest way to satisfy the independence element. It builds a public record. And it attracts talent and funding, because agencies that publish become places researchers want to work and funders want to back.

There is a structural reason the bar is higher in government than in industry. A private firm's customers can walk away; an agency holds a monopoly of service. The tax authority is the only tax authority, the border agency is the only border agency, and the veterans health system is the only path to earned veterans benefits. Citizens cannot choose a competitor when an agency's verification, fraud detection, or eligibility model fails them. That monopoly is what raises the evidentiary bar, and it is why a contemporaneous scientific record matters more than a retrospective reconstruction assembled after litigation or an inspector general report forces the issue.

It is worth being precise about what published evidence actually buys, because overselling it is how agencies get embarrassed twice. A published finding of no significant disparity across income quartiles is a strong, citable answer produced by people with no stake in the result. It is not proof that the system is fair, and it does not end the argument on its own. It bounds the argument. It moves the conversation from "you have nothing" to "here is what was tested, by whom, on what data, with what limitations," which is a conversation an agency can hold in public and lose parts of without losing its credibility.

What the Case Record Shows

The pattern that produces public failures is consistent enough to be predictive. An agency builds or buys a system, deploys it against citizens, and learns about systemic error through litigation, journalism, or inspector general reports rather than through evaluation. Four cases, from two continents, show the shape.

The tax authority rollout of a commercial facial-verification service in 2022 collapsed within weeks under public reporting, letters from senators including Wyden and Warren, calls for review by the Government Accountability Office, and a Treasury Inspector General for Tax Administration review. A standing partnership with academic vision and biometrics laboratories, working to published protocols for demographic performance testing anchored to the NIST publications on bias in AI and on demographic effects in face recognition testing, might have surfaced the failure modes before the system met a single taxpayer. It would not have guaranteed that outcome; testing finds what it is designed to look for, and it is a great deal better than the alternative of finding out in public.

Michigan's MIDAS unemployment system generated roughly 40,000 false fraud accusations between 2013 and 2015. Court records and subsequent audits showed no external academic peer review of the underlying classifier, no published disparate impact analysis, and no human subjects protection equivalent for claimants. The state ultimately paid more than $20 million in settlements and adopted legislation restricting fully automated adjudication, and the harms to unemployed workers and their families were not undone by either.

Europe supplies two cases with the same structure and different institutions. In the Netherlands, the SyRI risk-scoring system was struck down by The Hague District Court in 2020 for violating Article 8 of the European Convention on Human Rights, in part for inadequate scientific validation and transparency. The Dutch childcare benefits scandal, the toeslagenaffaire, forced the resignation of the Rutte III cabinet in January 2021 after tax authorities used risk models that disproportionately flagged dual-nationality families.

The counter-example runs the other way. Where a health system holds academic affiliations covering more than ninety percent of the country's medical schools, relationships built originally for clinical training extend naturally into algorithm evaluation, because the data use agreements, the review boards, and the privacy controls already exist. The lesson for anyone building this capability is uncomfortable but simple: an agency's academic partnerships precede its AI deployments, or they arrive too late to matter. Where a border agency deployed biometric matching at scale before building those relationships, oversight reporting identified weak external validation as a consequence.

Four Models, and When to Use Each

Collaboration is not one thing, and matching the model to the problem prevents most of the disappointment in this area. Four models cover the usual range, and an agency serious about evidence will eventually run all four, in an order determined by which risk is most urgent rather than by which relationship is easiest to start.

• Independent evaluation. You built the model; a research partner audits it. Best when you need defensible validation, as Renata does. The partner must have data access and no stake in the result, which is a screening requirement rather than an assurance.
• Joint research program. The agency and university design a study together over multiple years. Best for hard, unsolved problems, such as which interventions actually reduce readmissions rather than who is at risk of one.
• Sabbatical and personnel exchange. A professor spends a year embedded in your agency, or a staff member takes a research fellowship. In federal practice this runs through Intergovernmental Personnel Act assignments and detail arrangements under Title 5, and it works in both directions. Best for building durable internal capability rather than buying a single answer.
• Publication partnership. You co-author papers and reports. Best for establishing the agency's voice and contributing methods other governments can reuse, which is a public good and also a recruiting instrument.

Renata needs all four eventually and starts with an independent evaluation, because her immediate risk is the unanswered fairness question rather than a gap in capability. Sequencing on urgency has a second benefit: the first collaboration teaches both institutions how the other one works, and it is better to learn that on a bounded evaluation than inside a five-year joint programme.

The Five Federal Vehicles

A state agency can often work through a single research agreement. A federal agency chooses among five legal instruments, and the choice determines how much the agency can direct the work, who owns what, and how fast anything starts. Getting this wrong is expensive in a specific way: the vehicle is chosen at the beginning, and the constraint it imposes is discovered in the middle.

Grants suit exploratory work where researcher autonomy is the point. Cooperative agreements suit measurement science and public health pilots where agency staff are genuinely co-developing method rather than receiving a report. Research and development contracts under FAR Part 35 suit agency-directed deliverables and are used heavily where mission research has a schedule attached. Cooperative research and development agreements allow federal laboratories to collaborate with non-federal parties on flexible intellectual property terms, which is why standards laboratories use them extensively. Other transaction agreements suit prototype work and carry participation conditions that vary; the Defense Innovation Unit has executed more than $5 billion in such agreements since 2016, many involving academic teams.

The mechanics of negotiating a bilateral agreement, including data use agreements and sponsored research agreements at state level, are worked through in Academic and Research Partnerships. What follows here is the governance layer that sits above whichever instrument you pick.

The Governance Layer Before You Sign

Before any vehicle is signed, a defined set of questions has to be answered by someone with authority to answer them. Under OMB Memorandum M-24-10, issued in 2024, federal agencies designate a Chief AI Officer, publish a use-case inventory, identify rights-impacting and safety-impacting AI, and apply minimum risk management practices that include independent evaluation, impact assessments, and ongoing monitoring. Confirm the current status of that guidance before relying on it, because OMB policy in this area has moved more than once, and the underlying discipline matters more than the memorandum number.

The checks themselves are stable regardless of which memorandum is in force. Is the use case in the agency's inventory? For rights-impacting or safety-impacting systems, has an impact assessment been completed, or is it scoped into the research itself? Is the data sharing consistent with the Privacy Act of 1974 at 5 USC 552a, with health privacy law where applicable, and with the agency's published systems of records notices? Are information security boundaries defined under FISMA with a control baseline drawn from NIST SP 800-53? Does the research plan specify disparate impact testing, anchored to a named method rather than left to the researcher's discretion?

Two anchors are commonly used for that last point. The NIST AI Risk Management Framework, which is voluntary rather than binding, includes a subcategory at MEASURE 2.11 covering evaluation of fairness and bias, and the federal uniform guidelines on employee selection procedures supply a long-standing method in the employment context. Naming the anchor in the research plan matters more than which one you choose, because a study that reports disparities without a pre-specified method invites the objection that the method was chosen after the results were seen. Where controlled unclassified information is involved, its handling requirements under 32 CFR 2002 apply to the research environment exactly as they apply to the programme.

Executive Order 14110, issued in October 2023, layered further obligations on this landscape, directing red-teaming guideline development, establishing a safety institute within the Department of Commerce, and creating reporting expectations for developers of dual-use foundation models. Treat that as history rather than as current law unless you have confirmed otherwise: executive orders are revoked and superseded, the institute it established has since been renamed, and an agency citing a revoked instrument in a research plan has created a problem rather than a citation.

Data, Security, and Compute

Where an academic team needs agency data, the options form a ladder from least to most controlled, and each rung trades convenience against exposure. Synthetic or de-identified extracts are simplest and may remove the variable the study needed. Agency-controlled enclaves running on authorized cloud environments keep raw records inside your boundary while letting external researchers work. Shared national research computing pilots, including the National AI Research Resource pilot launched in 2024 with partners across several federal agencies, let university researchers work alongside federal staff under a coherent governance model. Federally funded research and development centers host secure environments of their own. Classified environments apply where defence and intelligence work requires them.

Each option carries cost, latency, and talent consequences that should be discussed with the researchers rather than decided over their heads. A university that cannot support cleared staff at scale will need the enclave route regardless of what anyone prefers, and an enclave that is slow or unfamiliar will quietly reduce how much analysis actually gets done. Renata chose in-environment access so that raw records never left the department, which cost her partner some convenience and removed an entire category of argument with her privacy office.

De-identification deserves one caution, because it is the option agencies reach for when a deadline looms. A minimized extract that satisfies a privacy review may also destroy the analysis, and the reverse risk is worse: a small number of quasi-identifiers can be enough to re-identify individuals in administrative data. Treat de-identification as a risk reduction with residual risk to be assessed, not as a step that removes the data from the scope of the rules governing it.

Intellectual Property and Publication Terms

The negotiation over rights follows a well-worn path. Universities want to publish, retain pre-existing intellectual property, and license to spinouts. Agencies want operational use and control of their own data. The settlement usually has four parts: a nonexclusive, irrevocable, royalty-free license to the agency for government purposes; university retention of copyright and patent rights subject to the Bayh-Dole framework at 35 USC 200 to 212, with march-in rights preserved; a bounded pre-publication review window; and an explicit statement that data provided by the agency remains agency data and remains subject to the Privacy Act. Data rights clauses under the FAR govern the contract side, and the specific clause and its default should be confirmed with your contracting officer rather than assumed.

The review window is the term most often abused and the one most worth drafting carefully. Federal practice commonly uses a window of 30 to 90 days; Renata's agreement used 30 to 60. What makes the number defensible is the scope attached to it. Review should be limited to release of controlled unclassified information, exposure of vulnerabilities carrying active exploitation risk, and factual accuracy about agency programmes. Review is not permitted for disagreement with findings, for embarrassment, or for policy preference. Write both halves into the agreement, because a window without a scope is a veto with a deadline.

Publication also has an accessibility obligation that agencies routinely discover late. Public outputs must meet Section 508 accessibility standards so that people using assistive technology can read them, and conformance is a floor rather than a demonstration of usability. Budget the remediation of figures, tables, and supplementary material at the start, because retrofitting a published report is more expensive than producing an accessible one, and an inaccessible public evidence base undercuts the transparency argument that justified publishing at all.

The Collaboration Agreement Checklist

Before any partnership starts, your team and the institution's sponsored programmes office should answer every item below in writing. Treat unanswered items as red flags rather than paperwork, and treat the list as the agenda for a single early meeting rather than as a sequence of separate reviews, because sequential review by different offices is what turns a two-month agreement into a year.

• Scope and question. One sentence stating the research question and the decision it informs.
• Data plan. What data, at what sensitivity, accessed how, stored where, destroyed when, and under which governing privacy authority.
• Publication clause. A fixed review window with its permitted grounds written out, so that review cannot become veto.
• Intellectual property and licensing. Agency license to all artifacts for government purposes; researcher right to publish methods; ownership of improvements stated rather than implied.
• Independence guardrails. Conflict-of-interest disclosure, and no vendor of the system under study funding or staffing the evaluation.
• Deliverables and dates. Annual standalone outputs mapped to budget cycles, so a multi-year study survives a single-year appropriation.
• Equity analysis required. The study must report performance broken down across the populations the agency serves. This is the clause that would have saved Renata.
• Accessibility. Public outputs meet Section 508 standards, with remediation budgeted rather than assumed.
• Security boundary. The computing environment, its control baseline, and who is responsible for maintaining the authorization.
• Funding and roles. Who pays, who does what, and a named accountable owner on each side.

Where Collaborations Fail

Most collaborations fail on logistics rather than science, and the failures repeat. Data access is the first: decide early whether the researcher receives a de-identified extract, works inside your secure environment, or operates under a data use agreement, and decide it with the researchers present. Publication rights are the second: academics must publish to advance their careers and agencies fear premature or embarrassing findings, and the fix is a drafted clause rather than an argument after the fact. Intellectual property is the third, and it is usually settled by asking who needs what rather than who deserves what.

Pace mismatch is the fourth and the most underestimated. A dissertation runs three years and a budget cycle runs one, so structure the work in annual deliverables that each stand alone. Conflict of interest is the fifth: a researcher who consults for your vendor cannot independently evaluate that vendor's model, and in federal practice the standards of ethical conduct at 5 CFR 2635 and the criminal conflict-of-interest provisions at 18 USC 208 make dual appointments a matter for counsel rather than for judgment in the moment. Screen before signing, and screen the institution as well as the individual.

Beyond logistics, four failures are institutional. Using academic partners as cheap labour and never integrating findings into policy produces research that sits on shelves. Publishing without context so that findings are misread in the press damages the relationship that produced them. Sharing agency data under terms weaker than the governing privacy law requires is a compliance failure regardless of how good the science is. And accepting vendor-funded academic work that is effectively marketing produces evidence that will not survive its first serious challenge. To that list add one political failure: not briefing the committees that fund you, which reliably produces appropriations trouble later.

Funding That Outlives a Budget Cycle

Collaboration is cheaper than agencies assume, because the funding rarely comes from one pocket. Renata blended three sources: a small line in her existing analytics budget, an in-kind contribution of faculty time the university counted toward its public-engagement mission, and a federal research grant the university won with the agency as a named partner. The general lesson for leaders is that you are often the asset rather than the payer. Real-world data and a live policy problem are precisely what grant reviewers want a study to address, so position the agency as the field site and academic partners will pursue funding on your behalf.

Sustained partnerships need more than one grant. Single-year funding produces brittle relationships and high turnover, since the graduate students who understand your data leave when the money does. Stronger models include multi-year centre grants of the kind science agencies and health agencies already run, agency contributions to federally funded research and development centre core programmes, small business research programmes where a commercialization path exists, in-kind computing credits from shared national research infrastructure, and co-funding through the 501(c)(3) foundations attached to some agencies, which can accept non-appropriated gifts.

That last option carries a legal caution that is not optional. The Antideficiency Act at 31 USC 1341 and the augmentation-of-appropriations doctrine constrain what an agency may accept and how, and philanthropic co-funding models must be cleared by agency counsel before anything is promised to a partner. The failure mode here is not a fine. It is a partnership announced, staffed, and then unwound, which costs an agency the relationship and the credibility that made the next one possible.

A 90-Day Start

Renata's path is repeatable at any size. In month one, write the one-sentence research question and the data plan, and identify two candidate research partners. In month two, run the conflict-of-interest screen and draft the agreement from the checklist, with the privacy office, counsel, and any federal programme office in the same conversation rather than in sequence. In month three, sign, grant scoped data access, and set the first annual deliverable: an independent equity evaluation of the readmission model.

Within a year she has the published, defensible answer she lacked, and a partner positioned to take on the harder causal question next. The 90 days matter less than the shape. One question, one instrument, one dated deliverable, and a relationship that has now survived a negotiation is worth more than a multi-year memorandum of understanding that produces no evidence and expires quietly.

Anti-Patterns

• Treating a published study as a verdict. An independent evaluation bounds an argument, it does not end one. It reports what was tested, by whom, on what data, with what limitations. An agency that presents it as proof of fairness will be corrected in public by the study's own authors.
• Building the partnership after the deployment. Every case in the record runs the same way round: system first, evidence later, correction by litigation or journalism. Partnerships built after a system is live inherit the deployment's constraints and its politics.
• Using pre-publication review as a veto. A review window with no written scope becomes an approval right over findings. Write the permitted grounds into the agreement, and accept that the price of credible evidence is publishing findings you would rather not read.
• Letting conflict-of-interest screening slip. A researcher who consults for the vendor whose model is under study cannot produce independent evidence, whatever their integrity. Screen the institution as well as the individual, and route dual appointments to counsel rather than deciding in the moment.
• Funding a partnership one year at a time. Annual funding guarantees turnover among the people who understand your data, so each year restarts. Structure multi-year support, or structure annual deliverables that each stand alone.
• Commissioning research nobody will act on. Findings that never reach policy are an expensive way of appearing rigorous. Name, in the agreement, the decision the study informs and the person who owns that decision.
• Treating de-identification as a release from the rules. A minimized extract can still support re-identification and still sits under the authority that governs the source data. Treat it as risk reduction with residual risk, and document the assessment.

Practice Prompts

• Write the unanswered question. Name the question about one of your deployed systems that you currently cannot answer with evidence, in one sentence, in the words a journalist would use. That sentence is your first research question.
• Choose the instrument. For that question, decide which vehicle fits: a grant, a cooperative agreement, a research and development contract, a cooperative research and development agreement, or an other transaction agreement. Write down which constraint of your chosen vehicle will bite first.
• Run the governance checklist cold. Answer the inventory, impact assessment, privacy authority, security boundary, and disparate impact method questions for a system you already operate, and note which answers do not exist in writing.
• Draft the review clause. Write the pre-publication review window and, beside it, the exhaustive list of grounds on which review may require a change. Show it to a researcher and ask whether they would sign it.
• Map the money. Identify one multi-year funding route, one in-kind contribution your agency can make, and one grant programme where your data and problem would strengthen a university's application.
• Screen a hypothetical partner. Take a real research group in your domain and run the conflict-of-interest screen: vendor relationships, funding sources, and prior work for parties with a stake in the result.

Reflection

• Which of your production systems would survive the question Renata could not answer, and what evidence would you actually produce in the room?
• If a partner's evaluation found a disparity in one of your systems, who in your agency would decide what to do about it, and is that person part of the agreement?
• Where does your agency currently keep its answers to the recurring questions of privacy basis, intellectual property position, and publication terms? If the answer is one lawyer's memory, what happens when that lawyer leaves?
• What would your academic partner say they get from the arrangement, in their own terms rather than yours?
• Which of the four collaboration models has your agency never used, and what risk is going unaddressed because of that?

Glossary

• Independent evaluation. An assessment of a system by a partner with data access and no stake in the result, used to satisfy the independence element that internal review cannot supply.
• Cooperative agreement. A funding instrument resembling a grant but with substantial agency involvement, suited to work where agency staff co-develop the methodology.
• Cooperative Research and Development Agreement. An instrument at 15 USC 3710a allowing federal laboratories to collaborate with non-federal parties on flexible intellectual property terms.
• Pre-publication review window. A fixed period during which the agency may review a paper on written grounds, limited to controlled information, exploitable vulnerabilities, and factual accuracy about agency programmes.
• Government purpose license. A nonexclusive, irrevocable, royalty-free right for the agency to use research artifacts for its own purposes, typically paired with university retention of copyright and patent rights.
• Augmentation of appropriations. The doctrine, alongside the Antideficiency Act, limiting an agency's ability to supplement its appropriated funds from outside sources without authority. Counsel clears philanthropic co-funding against it.
• Agency enclave. A controlled computing environment inside the agency's security boundary in which external researchers analyse data that never leaves agency control.

Related Lessons

• Academic and Research Partnerships covers the bilateral negotiation mechanics: data use agreements, sponsored research agreements, technology transfer to production, and the student pipeline.
• Publishing on Government AI takes the publication side further, into what an agency writes in its own name.
• GAO AI Accountability: Four Principles in Practice works through the accountability framework this lesson leans on for the independence requirement.
• Minimum Risk Management Practices covers the risk management obligations that determine when independent evaluation is required rather than advisable.
• Privacy Impact Assessments for AI Systems covers the assessment work that has to be complete or scoped before research data moves.
• Oversight Mechanisms: IG, GAO, Congress covers the audiences your evidence is ultimately built for.

Closing

Renata's model was good and her evidence was a slide deck, and the distance between those two facts is the whole subject of this lesson. Nothing in the remedy is exotic. One question written in plain words, one instrument chosen deliberately, a governance layer answered before signature, a review clause with a scope attached, and a deliverable dated inside a budget cycle. What makes it hard is that every one of those steps has to happen before anybody is asking, and there is never an obvious quarter in which to do it.

The case record is the argument for doing it anyway. In each of those failures, the evidence that would have prevented the harm was producible in advance and was produced afterwards instead, by courts, auditors, and journalists, at enormous cost to the institution and greater cost to the people it served. Academic collaboration is how an agency chooses which of those two timelines it is on, while the choice is still available.

Key Takeaways

• External evidence is infrastructure. Published, independent validation outlives staff and administrations in a way internal analysis cannot, and it is the cleanest way to satisfy an independence requirement.
• A monopoly of service raises the evidentiary bar. Citizens cannot choose another tax authority or another benefits system, which is why government AI needs a scientific record contemporaneous with deployment.
• Match the model to the problem. Independent evaluation for defensibility, joint programmes for hard questions, personnel exchange for capability, publication partnership for voice.
• The instrument decides the constraints. Grants, cooperative agreements, research and development contracts, cooperative research and development agreements, and other transaction agreements differ in how much the agency can direct the work and who owns the result.
• Answer the governance layer before signature. Inventory status, impact assessment, privacy authority and systems of records, security boundary, and a pre-specified disparate impact method are prerequisites rather than paperwork.
• Bound the review window by scope, not only by days. Controlled information, exploitable vulnerabilities, and factual accuracy are legitimate grounds; disagreement, embarrassment, and policy preference are not.
• Require the equity breakdown in the agreement. Mandating disaggregated performance analysis converts the question that ambushed Renata into a planned deliverable.
• You are often the asset, not the payer. Real data and a live policy problem let academic partners win the funding, but sustained work needs multi-year structure and counsel-cleared arrangements for any outside money.

Frequently Asked Questions

Is an independent study enough to settle a fairness challenge?

It bounds the challenge rather than settling it. A published evaluation tells you what was tested, on which data, by whom, and with what limitations, which is a far stronger position than an internal deck and a good deal weaker than proof. Agencies that present external findings as a verdict tend to be corrected by the researchers themselves, which costs more credibility than the original gap did.

How do we choose between the federal vehicles?

Start from how much you need to direct the work. If researcher autonomy is the point, a grant fits. If your staff will co-develop the methodology, a cooperative agreement fits. If you need specific deliverables on a schedule, a research and development contract fits. Federal laboratory collaboration runs through cooperative research and development agreements, and prototype work often runs through other transaction agreements with participation conditions attached. Write down which constraint of your chosen vehicle will bite first.

Can researchers work with our data without it leaving the agency?

Yes, and that is often the right answer. An agency-controlled enclave keeps raw records inside your security boundary while giving external analysts access, which is what Renata chose. Shared national research computing pilots offer a similar model at larger scale. The trade is convenience and speed for control, and it is worth discussing with the research team rather than imposing, because an enclave nobody can work in reduces the analysis you actually get.

What stops pre-publication review from becoming censorship?

A written scope. Limit review to release of controlled unclassified information, exposure of vulnerabilities with active exploitation risk, and factual accuracy about agency programmes, and state explicitly that review may not be used for disagreement with findings, embarrassment, or policy preference. Agencies that use the window to suppress unwelcome results do not get a second partnership, and the university's other departments hear about it before the paper appears.

How do we fund this beyond a single year?

Combine sources deliberately. Multi-year centre grants, agency contributions to federally funded research and development centre core programmes, in-kind computing credits, and small business research programmes where a commercialization path exists all extend the horizon past one appropriation. Where an agency-affiliated foundation can accept non-appropriated gifts, that route must be cleared by counsel against the Antideficiency Act and the augmentation-of-appropriations doctrine before anything is promised to a partner.

Which frameworks should the research plan cite?

Cite the ones you have confirmed are current. The NIST AI Risk Management Framework is voluntary and supplies a named subcategory for fairness and bias evaluation. OMB guidance in this area has changed more than once, so verify what is in force rather than copying a memorandum number from an older plan, and treat executive orders as historical unless you have checked their status. The discipline the citation supports matters more than the citation.