David Chen runs the IT modernization program for a mid-size county. Two years ago he bought a "smart" eligibility-screening platform from a respected vendor. It worked. Then a citizen sued, claiming the tool had systematically denied her housing assistance. David's lawyers asked the vendor three questions: What data was the model trained on? Who built the part that calculated risk scores? Can we see the model's accuracy by race? The vendor's answer to all three was a version of "that's proprietary," and then a quieter admission: the risk-scoring module was actually built by a subcontractor in another country that the vendor "no longer had a relationship with." David realized he was on the hook for a system he did not understand, built partly by a company he had never heard of.

Almost no government agency builds its own AI. You buy it, license it, or reach it through a cloud service. You use cloud providers for infrastructure, commercial machine-learning platforms for training, contractor firms for development, and consulting companies for guidance. That means most of your AI risk is not in your building. It is in your vendors, and in your vendors' vendors. Each of those parties can mishandle data, deploy an insecure system, introduce a bias, or simply stop maintaining what they sold you. This lesson is about managing that risk before it becomes a lawsuit, a breach, or a headline.

Why third-party AI risk is its own discipline

You already manage vendors. AI adds three problems that ordinary procurement does not handle well, and the first is that the model is a black box you rent. With traditional software you can read the requirements and test the outputs against them. With AI, the behavior depends on training data and model internals the vendor may not share and may not fully understand itself. The second is that the risk changes after you sign: a vendor can retrain the model, swap in a new version, or change the underlying foundation model in a routine software update. The system you tested in the demo is not guaranteed to be the system running next quarter.

The third problem is depth. Your vendor likely builds on someone else's model, hosted on someone else's cloud, fine-tuned by someone else's contractor. David's risk-scoring module is the rule, not the exception. The federal AI risk management framework calls this the AI supply chain, and treating it as a single vendor relationship is the core mistake. The uncomfortable arithmetic underneath all three problems is simple: the more of your system depends on third parties, the more risk you have accepted. A vendor's security failure becomes your security failure. A vendor's bias becomes your bias. A vendor's decision to stop maintaining a component becomes your outage.

Third-party AI risk is unusually acute in the public sector for reasons that have nothing to do with the technology. Vendors often operate under less stringent governance than government does. Data flows to parties outside your direct control. You may not fully know what a vendor does with your data or your models. Vendor failures land directly on your operations rather than on theirs. And your own regulatory compliance frequently depends on the vendor's compliance, which means you can be found non-compliant for something you never did. Yet vendor oversight remains one of the most neglected areas of government AI governance, usually because it belongs to no single office.

The five categories of third-party AI risk

Before you can assess a vendor you need a vocabulary for what can go wrong. Third-party risks fall into five categories, and a serious assessment walks all five rather than stopping at security, which is where most agency reviews begin and end. The categories below come with a worked example each, because the abstraction is easy to nod at and hard to act on. Read each example as a question about a system you already run: if that happened next Tuesday, who would find out, how, and what would you be able to do about it?

Notice how the compliance row differs from the others. In the data, model and operational rows the vendor causes harm and you inherit it. In the compliance row the vendor's paperwork gap becomes your audit finding even when nothing has malfunctioned at all. That is the category agencies underweight most, because there is no incident to point at until an auditor arrives. It is also the category where you have the most leverage before signature and almost none afterwards, since compliance posture is expensive for a vendor to retrofit and cheap for them to promise. Ask for the evidence, not the assurance.

Before you sign: assessing the vendor and the model

This is due diligence on the AI itself, not just on the company. The mistake David made was assessing the vendor's financials and references but never the model. A real AI vendor assessment asks the vendor to answer, in writing, about provenance, subcontractors, performance by group, security posture and change control. Written answers matter more than a demo, because a written answer is something you can hold a vendor to later and a demo is something the vendor controls entirely. Ask for all of it before the selection decision, not after, when your only remaining lever is walking away from a completed procurement.

• Provenance. What foundation model or base model is this built on? What data was it trained on, and do you have rights to that data?
• Subcontractors. Name every party that touched the model's development or hosting. This is the question that surfaces the hidden subcontractor.
• Performance by group. Show accuracy and error rates broken down by the groups our decision affects, such as by race, age, and disability where lawful and relevant.
• Security posture. Is the hosting environment authorized for our data sensitivity level? For cloud services holding government data, that usually means a FedRAMP authorization, the federal program that vets cloud providers.
• Change control. How will you notify us before you retrain, update, or replace the model?

Those five questions are the AI-specific layer. Underneath them sits a broader vendor assessment covering six areas, each with red flags that should stop a procurement rather than generate a follow-up email. The value of writing the red flags down in advance is that it removes the argument from the room. When a vendor cannot articulate its own security controls in a live meeting, the question is no longer whether the program manager feels reassured. It is whether a documented disqualifier has been met, and the answer goes in the file either way.

One caution about certifications, because agencies routinely read more into them than they carry. A SOC 2 Type II report says that an auditor tested a set of controls the vendor itself defined, over a stated period, and reported what they found. It is meaningful evidence that a security program exists and operates. It is not a guarantee that the vendor will not be breached, and it says nothing whatsoever about how the model behaves, who it was trained on, or whether it treats applicants unevenly. Read the report, including the exceptions section and the scope statement, rather than the logo on the sales deck.

Turning promises into contract obligations

A demo is marketing. A contract is the only thing that holds at 2 a.m. during an incident. If it is not in the contract, you do not have it, and a vendor's reassuring email is not a right to audit. The six clauses below are the ones agencies most often wish they had written, drawn from the pattern of what went missing when things went wrong. Adapt them to your acquisition rules rather than lifting them verbatim, and route them through your contracting officer early, because a clause proposed after the solicitation closes is a clause you will probably not get.

A fuller contract framework organizes the same protections into eight requirement families, and it is worth seeing them as families because that is how contract review actually proceeds. Numbers shown in square brackets below are placeholders the agency sets during negotiation, not standards. Different agencies land on different windows for the same obligation, and you will notice that the incident clock in the table above and the one in the requirement family below are not the same. That is the point: pick your own window deliberately, document why, and apply it consistently across your vendor portfolio rather than inheriting whatever each vendor's template proposed.

• Security requirements. Vendor implements controls meeting a named standard such as the NIST Cybersecurity Framework or ISO 27001; undergoes annual third-party security audits such as SOC 2 Type II; reports security incidents affecting government data within [24] hours; maintains encryption of sensitive data in transit and at rest; and limits access to authorized personnel.
• Data rights and retention. Vendor uses government data only for contract purposes; does not use it for its own purposes or derivative products; retains it only for the contract duration plus a [30] day period; returns or destroys all government data at termination; and provides signed certification that deletion occurred.
• Data subject rights. Vendor fulfils access requests within [15] business days and deletion requests within [30] days; complies with the applicable privacy law; does not limit the agency's own ability to fulfil those rights; and documents and reports every request it receives.
• Subcontractor management. Vendor documents all subcontractors with access to government data; obtains agency approval before engaging them; requires them to meet the same security and data handling obligations; remains accountable for their compliance; and audits them at least annually.
• Transparency and auditability. Vendor provides access to system logs and audit trails; documents all data accessed by vendor personnel; provides regular compliance certification; permits the agency and agency-authorized auditors to audit its practices; and documents any system change affecting government data.
• Incident response and reporting. Vendor maintains documented incident response procedures; notifies the agency of any incident affecting government data within [24] hours; provides investigation details within [5] business days; cooperates with agency investigations; and provides a remediation plan.
• Service level agreements. Availability stated as a percentage measured monthly, such as [99.9]%; response time targets; critical issues addressed within [4] hours and non-critical within [24]; and service credits as the remedy when the vendor misses the committed levels.
• AI governance and fairness. Vendor tests for bias and documents results quarterly; maintains transparency documentation explaining how the system works; implements explainability mechanisms; documents model limitations and failure modes; and maintains model provenance documentation.

Subcontractors: the liability you never signed

Your vendor hires other vendors. You are liable for their failures even though they are not in your contract and you may never learn their names. That is the shape of David's problem: a module that decided who got housing assistance was written by a company his county had no relationship with, no leverage over, and no way to question. The remedy is not clever legal drafting after the fact. It is four steps applied at procurement, each of which is easy to ask for before signature and close to impossible to obtain afterwards.

First, document every subcontractor. Require the vendor to list each one with access to government data, what each does, what data and systems they can reach, and where they operate geographically. Second, flow the same requirements down: the contract should state that the vendor shall require subcontractors to meet the same security, data handling, compliance and AI governance obligations as the vendor. Third, require the vendor to audit subcontractors annually, and audit the vendor's subcontractor audits yourself, because an unexamined audit programme tends to become a filing exercise. Fourth, require notice and approval before the vendor changes subcontractors, so the chain does not quietly reshape itself mid-contract.

The geographic question in step one is not paperwork. Consider a vendor that outsources data processing to a subcontractor operating in a jurisdiction with weak privacy protections. The subcontractor has a breach. You are liable, you never contracted with them, and your remedies run only against the vendor who chose them. The deeper supply-chain questions, including how compromised components propagate through a chain of dependencies, are worked through in AI Supply Chain Risk; this lesson stays with the contractual relationship you can actually reach.

After you go live: continuous monitoring

The biggest myth in vendor management is that risk work ends at signature. Assessment at contract start is a snapshot of a moving object. AI drifts: populations change, the vendor updates the model, and a tool that was fair in January can degrade quietly by June without anyone doing anything wrong. Continuous monitoring means a standing process with owners and dates, not an annual review that gets rescheduled twice and then skipped. It also means monitoring the vendor as an organization, not only the system as a piece of software, because most of what will hurt you shows up first as a change in the company.

Six streams are worth running, each with a defined action when it goes wrong. Performance monitoring asks whether service levels are being met, whether outages are frequent, how fast tickets are resolved, and whether recurring issues point to something systemic; when a vendor consistently misses, escalate and demand an improvement plan. Security monitoring asks whether the vendor has had incidents, how they responded, whether certifications are current, and whether recent audits found anything; concerning findings should trigger a remediation plan or a serious look at switching. Compliance monitoring asks whether the vendor remains compliant, whether regulators have made findings, and whether certifications have lapsed.

The other three streams are the ones agencies skip. Financial monitoring tracks stability, corporate changes, pricing sustainability and signs of distress, and its action is a contingency plan for the day the vendor fails rather than a conversation about the vendor's balance sheet. Data handling monitoring tracks how much government data the vendor holds, how long it is retained, whether breaches occurred, whether deletion happens as required, and whether data subject requests are being fulfilled. Fairness and performance monitoring tracks whether system fairness is stable or degrading, whether complaints about bias are arriving, and whether accuracy and latency have shifted without explanation.

A workable cadence separates what you look at monthly, quarterly and annually. Monthly: performance metrics such as uptime and response time, security incidents, and service tickets. Quarterly: compliance certifications, security audit status, and fairness metrics, reviewed in a meeting where the vendor presents current performance data rather than a sales update. Annually: a comprehensive vendor reassessment, compliance audit, security audit and financial review, plus a focused rerun of your original AI assessment. Any major model change resets that annual clock, because the thing you assessed is no longer the thing you are running.

Be honest about what this buys you. Monitoring surfaces problems in the things you chose to measure, on the schedule you chose to measure them. It will not catch a harm you never defined a metric for, and a clean dashboard is evidence that your indicators are green, not proof that the system is behaving well for everyone it touches. Pair the numbers with a channel where caseworkers and citizens can report something that looks wrong, and treat a rise in complaints as data even when every measured metric is holding steady.

Two worked scenarios

Consider first a cloud provider used for AI development and deployment. Large commercial cloud providers arrive with established security programmes and mature governance, which lowers some risks and eliminates none: data exposure, service outages and compliance gaps remain live. The governance work is to establish boundaries. What can the provider access? How long may they retain data? Which compliance requirements apply? Concretely, require a current third-party security audit and controls mapped to a named framework, incident reporting within 24 hours, and encryption in transit and at rest. Specify that data may be used only for this project, must be deleted within 30 days of contract end, and may never train the provider's own models. Require audit logs showing who accessed what data when.

Where EU residents are served, that layer adds GDPR compliance and Standard Contractual Clauses for data transfers, plus annual third-party compliance audits and current certifications. Monitoring runs monthly on service metrics and incident reports, quarterly on the audit findings and on the fairness metrics of the systems running there, and annually on the full reassessment and renewal negotiation. The outcome you are aiming for is not a smaller cloud footprint. It is a cloud relationship where the governance boundary is written down, so that the provider stays the primary infrastructure and the agency can still say precisely what it has authorized.

The second scenario is a commercial machine-learning platform underpinning several agency AI systems, which is a higher-risk shape because failure propagates to everything built on it. The platform vendor sees training data, models and system logs, and pushes updates that reach all users at once, so a change intended for someone else can reduce your performance or introduce a vulnerability. Require current third-party security certification and demonstrated alignment to a recognized cybersecurity framework, and review incident reports quarterly for anything touching government systems. Establish that the agency owns all training data and models, prohibit use of that data for the vendor's own products, require tenant isolation from other customers, and require deletion at contract end.

Subcontractor management matters more here than anywhere else, because platform vendors commonly run on commercial cloud infrastructure, which means your requirements have to reach a party two steps away. Require the vendor to document subcontractors and to audit them. On availability, set service levels and check monthly whether they are met and what the impact was when they were not; persistent misses justify escalation and, if they continue, a switch. On AI governance, require the vendor to document the platform's known biases and performance across demographic groups and how interpretable the models trained on it actually are, then watch whether systems you train there develop biases nobody predicted.

A usable artifact: the third-party AI risk register

Bring the assessment, the contract and the monitoring together in one living document your governance board reviews. Each AI vendor gets a row; you score it and date it. This is the single page that answers an auditor, a legislator, or a judge, and its real value is that it makes absence visible. A blank cell under subcontractors is not a formatting problem, it is a finding. Nine fields carry the weight, and the last one is the field agencies most want to leave empty and most need to fill.

Had David kept this register, the hidden subcontractor and the missing audit right would have been visible on day one, in a single cell, long before a citizen's lawyer found them. The register also gives you the one thing agencies most often lack in a vendor dispute, which is a dated record of what you asked for and what you were told. That record is what turns "we had concerns" into a documented sequence a general counsel can work with, and it costs one meeting a quarter to maintain.

Anti-Patterns

• Engaging a vendor before the assessment is finished. Schedule pressure makes assessment feel like a delay, so the contract gets signed and the review is completed later, or never. What goes wrong is discovered in production: the vendor is not competent, the systems are insecure, the data is mishandled, the regulations are unmet, and it is now too late to switch. Never engage without a completed assessment covering security, data governance, compliance, technical capability and financial viability, and ask for references from other government agencies that use the vendor.
• Never asking who your vendor's vendors are. You contract with a company and never ask about subcontractors, so your vendor quietly uses parties you would have rejected, possibly in jurisdictions you would not have approved. When one of them has a breach you are liable to citizens for a relationship you did not know existed. Require documentation of all subcontractors, agency approval before engagement, flow-down of every obligation, and the right to audit.
• Treating assessment as a one-time gate. After contract start the vendor is not monitored, so a degrading security posture, an unreported breach, missed service levels and a compliance lapse all accumulate for months or years until something forces them into view. Implement monitoring on a monthly, quarterly and annual rhythm, review the results with someone who has authority to act, and escalate the first occurrence rather than the fifth.
• Reading a certification as a verdict. A current SOC 2 Type II or an ISO 27001 certificate becomes shorthand for "this vendor is safe," and the questions stop. A certification reports what an auditor tested against criteria that were largely the vendor's own, during a window that has already closed, and no security certification evaluates whether a model treats applicants unevenly. Read the scope and the exceptions, and keep the AI-specific questions on the list regardless of what certificates are held.
• Believing monitoring covers you. A monitoring programme reports on the metrics you defined, and it is easy to slide from "our indicators are green" to "the system is fine." Monitoring catches what you chose to watch, on the cadence you chose to watch it, and a harm nobody wrote a metric for will not appear on the dashboard. Pair metrics with a complaint channel and treat a pattern of complaints as a signal even when every measured indicator is holding.
• Staying with a failing vendor because leaving is hard. Switching costs are real and visible; the cost of continuing with a vendor who consistently misses obligations is diffuse and easy to defer. Agencies talk themselves into another year, then another. Decide in advance what pattern of failure triggers a switch decision, write it into the contract as termination rights, and make sure your data and configurations can actually leave, which is the subject of Vendor Lock-In Prevention.

Practice Prompts

• Assess a vendor. Take a vendor you currently use or are considering. Work through security (do they meet your standards, and what certifications do they hold), data governance (how do they handle government data, and can they delete it), compliance (do they meet the regulations that apply to you), technical capability, and financial viability. Rate each area from 1, meaning inadequate, to 5, meaning excellent. Then answer the only question that matters: would you contract with this vendor, and can you defend that answer in writing?
• Draft the contract requirements. For a critical vendor engagement, write out the security standards the vendor must meet, the data rights the agency retains, the regulations the vendor must comply with, how the vendor will manage its subcontractors, and how you will monitor them. Then hand it to your contracting officer and ask which of your requirements are actually achievable under your acquisition rules, and which need to be reframed as evaluation criteria instead.
• Build a monitoring plan. For one critical vendor, specify which metrics you will track, how often you will look at them, what range counts as acceptable for each, what triggers escalation, and who reviews the results. Name a person, not an office. Then check the plan against the six monitoring streams in this lesson and note which ones you left out, because the omissions are usually more revealing than the inclusions.
• Fill in the register. Take your three highest-stakes AI systems and complete a row for each in the third-party AI risk register. Leave blank anything you cannot answer today rather than guessing. The blanks are your work plan for the next quarter, in priority order.

Reflection

• Which vendors are critical to your AI systems, and what risk does each one create that the others do not?
• For your highest-risk vendor, how robust is your governance, and what specifically is missing from it?
• If a critical vendor failed today through a service outage, a data breach or a compliance violation, what would the impact be, and who would make the first decision?
• Are you monitoring vendors continuously, or did the monitoring effectively stop at contract start?
• Which of the five risk categories does your agency review well, and which one has no owner at all?

Glossary

• Third-party risk. Risk created by engagement with an external vendor or contractor.
• Vendor assessment. Comprehensive evaluation of a vendor's security, compliance, technical capability and financial viability before engagement.
• SOC 2 Type II. A third-party audit report describing whether specified security controls operated effectively over a stated period. Evidence about a control set during a window, not a guarantee of security and not a statement about model behavior.
• Data retention. How long a vendor holds government data, including after the contract completes.
• Data subject rights. Citizens' rights under privacy law, such as rights to access, deletion and explanation.
• Service level agreement. Contract terms specifying a vendor's performance commitments, such as availability and response time, together with the remedy when they are missed.
• Subcontractor. A vendor hired by your vendor to perform part of the work.
• Flow-down. Contract language requiring a vendor to impose its obligations to you on its own subcontractors.
• AI supply chain. The full chain of models, data, infrastructure and parties that a deployed AI system depends on.

Related Lessons

• AI Supply Chain Risk goes deeper into the chain behind your vendor, including compromised components and dependency propagation.
• AI Vendor Evaluation Methodology turns the assessment areas here into a scored, repeatable selection process.
• Evaluating AI Vendor Claims covers how to test what a vendor asserts about accuracy and capability before you rely on it.
• Working with AI Vendors and Contractors covers the day-to-day working relationship once the contract is in place.
• Managing AI Vendor Performance covers the mechanics of holding a vendor to the levels you negotiated.
• Vendor Lock-In Prevention covers the exit you need before switching becomes theoretically available and practically impossible.
• OMB M-24-18 and AI Procurement Governance places these obligations in the federal acquisition policy context.
• Enterprise AI Risk Management connects third-party risk to the agency-wide risk picture your governance board owns.

Closing

Third-party AI risk management is foundational to government AI governance, and the reason is structural rather than procedural. If you cannot govern third-party vendors, you cannot govern your AI systems, because the systems are theirs. The most sophisticated internal governance in the country is undermined by a vendor relationship nobody reviews after signature. Vendors are not simply service providers. When you engage one, you extend your governance boundary to include them, and poor vendor governance is therefore a governance failure on your part rather than on theirs.

Agencies that manage vendors well do four things consistently: comprehensive assessment before engagement, clear contractual requirements addressing the risks that matter, continuous monitoring with owners and dates, and a genuine willingness to switch when a vendor consistently fails. None of that guarantees a quiet year. It does mean that when something goes wrong you will find out earlier, you will have the contractual standing to respond, and you will be able to show a legislator or a judge what you asked, when you asked it, and what you were told. That is the difference between David's position and the position he wishes he had been in.

Key Takeaways

• Your AI risk lives in your vendors and their vendors. Since agencies rarely build their own AI, the supply chain, not your own code, is where most of the risk sits, and a vendor's failure becomes yours by default.
• Assess the model, not just the company. Financial stability and references say nothing about training data, hidden subcontractors, or error rates by group. Walk all five risk categories and all six assessment areas, and ask the AI-specific questions in writing before you sign.
• A certification is evidence, not a verdict. A security audit report covers controls the vendor defined, over a window that has closed, and evaluates nothing about how the model treats people. Read the scope and the exceptions.
• If it is not in the contract, you do not have it. Audit rights, subcontractor disclosure and flow-down, change notice, performance floors, data rights and exit, and incident reporting are clauses to negotiate, not assumptions to make.
• Vendors change the model after you sign. Require advance notice of retraining and version changes, re-test after every one, and reset your annual reassessment clock when a major change lands.
• You are liable for subcontractors you never chose. Require the list, require approval before changes, flow every obligation down, and audit the vendor's audits of them.
• Monitoring is continuous, and it only covers what you measure. Run the six streams on a monthly, quarterly and annual cadence, and keep a complaint channel open for the harms no metric was written for.
• Keep one risk register your governance board owns. One row per vendor, scored and dated, with residual risk stated plainly, is the artifact that answers an auditor or a judge in a single page.
• Be willing to switch. Staying with a vendor who consistently fails to meet requirements is usually riskier than the switch you are avoiding, and the ability to switch has to be built into the contract before you need it.

Frequently Asked Questions

We are a small county and the vendor will not negotiate. What do we actually get? Less than a federal agency, but more than nothing. Prioritize three clauses: data rights and exit, subcontractor disclosure, and change notification. Those three cost the vendor almost nothing to grant and give you the ability to leave, to see the chain, and to know when the system changed. Where you cannot get a clause, document that you asked and were refused, and record the gap in the residual risk field of your register.

Does a FedRAMP authorization mean the AI is safe to use? No. It addresses whether the cloud environment has been assessed for handling data at a given sensitivity level. It does not evaluate whether the model is accurate, whether it performs evenly across groups, or whether the vendor documented its training data. Hosting authorization and AI assessment are two separate reviews, and passing one tells you nothing about the other.

The vendor says its bias testing results are proprietary. Is that acceptable? It is a decision point, not a fact of life. If the system touches a consequential decision about a citizen, an inability to show performance by the affected groups is a red flag under the AI governance assessment area. Escalate it as a documented finding rather than absorbing it, and if you proceed anyway, the reason and the approver both belong in the register.

How do we monitor a vendor when we have no technical staff? Most of the six monitoring streams are contract administration rather than engineering. Service levels, incident reports, certification currency, financial news and data handling attestations are all documents a program analyst can track. The fairness stream is the one that needs help, and the usual route is to require the vendor to produce the performance-by-group report on a set schedule and to have someone outside the program office read it.

Who owns third-party AI risk in an agency? That is the question this lesson exists to force. It usually sits between procurement, the program office, the security office and the privacy office, which in practice means nobody. The register is the mechanism for fixing it, because every row has a named monitoring owner with a next review date, and a row without a name is an unowned risk in plain sight.