Priya Venkatesan ran risk management for a state benefits agency that had just launched an AI tool to help caseworkers screen applications for a food assistance program. The vendor's test results were spotless: 98 percent accuracy. Two weeks after launch, a community legal aid group filed a complaint. They had found that when an applicant wrote their address in a format common on tribal lands, the tool flagged the application as high risk of fraud far more often. Nobody had tested for that. The accuracy number was real, and it had completely hidden a discriminatory failure that only appeared when someone went looking for trouble on purpose.

Going looking for trouble on purpose is exactly what AI red-teaming is. A red team deliberately attacks your AI system before your adversaries do, or in government, before harmed residents and their lawyers find the weakness for you. It is the difference between asking "does this work?" and asking "how can I make this fail, and who gets hurt when it does?"

Why a 98 percent accuracy score lies to you

Standard testing measures average performance on typical inputs. Red-teaming measures worst case performance on hostile, unusual or edge case inputs. These are different questions with different answers. Priya's tool was 98 percent accurate on average and badly biased against a specific population, and both statements are true at once. Average accuracy is a single number that quietly averages away the failures that matter most, because the people most affected by edge cases are usually the smallest groups in the data.

It is worth being precise about what that number does not say. Accuracy is the overall share of cases the system got right. It is not precision, which asks what share of flagged cases were truly positive, and it is not recall, which asks what share of truly positive cases were flagged. None of the three tells you anything about how the error is distributed across groups, which is the only question the legal aid group was asking. A headline accuracy figure with no denominators and no strata behind it cannot answer a fairness question at all, no matter how high it is.

The federal AI risk framework from the National Institute of Standards and Technology, a voluntary and non binding standard that many agencies have adopted, explicitly calls for testing AI under adversarial and stress conditions rather than only normal ones. Red-teaming is how that gets done. It is the practical answer to the framework's expectation that AI systems be examined for robustness, security and fairness before they touch real decisions about real people.

What AI red-teaming is, and what it is not

Red-teaming began in Cold War military planning and entered cybersecurity in the 1990s as adversarial penetration testing. AI red-teaming borrows the adversarial posture and operates on a different attack surface: the model, its training data, its inference pipeline, and the sociotechnical system around all three. Federal guidance defines it as a structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with the developers of the AI. That definition is deliberately about finding flaws, not about certifying their absence.

It is distinct from three things it gets confused with. Traditional penetration testing targets networks, hosts and applications. Vulnerability scanning is largely automated and looks for known issues. User acceptance testing verifies that documented requirements are met. A program that runs any of those and reports it as a red team has not red-teamed. Running standard benchmarks and calling the result a red team exercise is the same mistake in a different costume.

AI red-teaming covers three tiers. Model level attacks include prompt injection, jailbreaks and adversarial examples. System level attacks include indirect prompt injection through retrieved context, tool abuse, and data exfiltration through agentic behavior. Sociotechnical attacks include disinformation generation, bias exploitation and misuse by authorized users. A federal red team that looks only at the model has covered one tier and left two untested, which is how Priya's program passed its security review and failed its first contact with the public.

Four ways to attack an AI system

Red-teaming a government AI system means probing four practical attack surfaces. Priya's food assistance tool is vulnerable on all four, and it is worth carrying it through each.

Bias and fairness attacks

You deliberately feed the system inputs that differ only in a demographic signal and look for unequal outcomes. Priya's team should have submitted applications identical except for address format, the origin of the applicant's name, or the language used. Testing of that kind is how the tribal address failure gets found, and only if somebody thinks to vary that particular field. That caveat is the whole discipline: a bias test finds the disparities you thought to look for, which is why the team composition below matters as much as the technique. This is the highest priority surface for rights impacting government systems.

Prompt and input manipulation

For systems that accept text, attackers craft inputs designed to confuse or hijack the model. An applicant or a fraudster might phrase an application so that the model approves something it should flag, or the reverse. The harder version is indirect prompt injection, where the hostile instruction is not typed by the user at all but sits inside a document the system retrieves and reads as context. Public exercises have found that prompt injection remains trivially successful against most models that lack hardened scaffolding, so assume it works against yours until your own testing says otherwise.

Data poisoning, extraction and edge cases

You probe what happens with strange but legitimate inputs: a blank field, an enormous number, a date in 1899, a name with characters the system has never seen. Real residents generate weird but valid data constantly, and if the system crashes or defaults to deny, people lose benefits over a formatting quirk. The adversarial versions of this surface are data poisoning, where an attacker plants training examples that trigger chosen behavior; model extraction, where repeated queries reconstruct a copy of the model; and supply chain attacks against the foundation model itself before you ever downloaded it.

Privacy and leakage

You test whether the system can be made to reveal information it should protect: another applicant's data, training records, or internal decision logic. The named techniques here are model inversion, which reconstructs training inputs from model behavior, and membership inference, which determines whether a specific record was in the training data. For a benefits system holding sensitive personal information, a model that can be coaxed into echoing someone else's details is a breach waiting to happen, and the fact that a probe did not succeed is not the same as the fact that no path exists.

Your test suite asks whether the AI works. Your red team asks who it fails, how, and whether that failure is something you could survive explaining in front of a judge.

The twelve generative risk categories

Federal guidance for generative AI names twelve risk categories that a red team is expected to probe, and running the list is a fast way to find the surfaces your team never considered. They are: CBRN information, meaning meaningful uplift to a non expert seeking chemical, biological, radiological or nuclear harm; confabulation, meaning plausible but false content, especially fabricated citations and numbers; dangerous, violent or hateful content; data privacy, covering leaked training data, memorized personal information and exposed context; environmental cost at scale; harmful bias or homogenization; human and AI configuration problems such as over reliance and misaligned mental models; information integrity, covering synthetic media and provenance; information security, covering prompt injection, jailbreaks and agentic misuse; intellectual property; obscene, degrading or abusive content, including non consensual intimate imagery and child sexual abuse material; and value chain and component integration.

Counting the categories in that source list gives twelve, which matches the number the guidance itself states. Not every category applies to every system, and the ones that obviously do not still deserve a recorded sentence saying so and why. A red team plan that silently omits a category is indistinguishable, six months later, from a red team plan that considered and dismissed it.

Planning attacks with a taxonomy instead of imagination

MITRE ATLAS, the Adversarial Threat Landscape for Artificial-Intelligence Systems, first published in 2021 and updated since, organizes adversarial machine learning into an ATT&CK style matrix of tactics and techniques. Reconnaissance covers gathering information about model architecture, training data and deployment environment. Resource development covers obtaining adversarial examples, building proxy models and acquiring data. Initial access covers supply chain compromise, valid accounts and exploitation of a public facing machine learning application. Attack staging covers creating proxy models, backdooring a model and verifying an attack before using it.

The matrix continues through persistence, which includes poisoning training data; defense evasion, which includes crafting adversarial data to evade a model; discovery of model ontology and model family; collection of artifacts and repository data; exfiltration through the inference interface or by conventional means; and impact, which covers evading the model, denial of service, eroding model integrity, cost harvesting, external harms and theft of model intellectual property. The point of adopting a taxonomy is that it converts red-teaming from a creative exercise whose coverage nobody can describe into a structured campaign whose coverage you can report. ATLAS also carries a case study library, which is free experience.

The federal directives behind all of this

Executive Order 14110, issued in 2023, directed the Secretary of Commerce through the Director of NIST to coordinate guidelines for AI red-teaming at Section 4.1(a)(ii), and its dual use foundation model provisions created reporting obligations on red team results for covered models. OMB Memorandum M-24-10, issued in 2024, requires pre deployment testing among its minimum practices for rights impacting and safety impacting AI and instructs agencies to conduct or require red-teaming where appropriate. A National Security Memorandum on AI issued in 2024 extended red-teaming expectations into national security systems, alongside NSM-10 on critical and emerging technologies.

Adjacent guidance fills in the technical side. Defense practice folds red team elements into test and evaluation, verification and validation. CISA published guidelines for secure AI system development with the United Kingdom's National Cyber Security Centre in November 2023 and has issued AI specific advisories since. Homeland security guidelines for critical infrastructure owners and operators, issued in 2024, reach across 16 critical infrastructure sectors. International coordination runs through the Bletchley Declaration in 2023, the Seoul Declaration in 2024 and the emerging network of national AI safety institutes. Agencies contracting for AI services should require red team evidence aligned to these instruments, because the alternative is explaining the omission to an auditor.

Designing an exercise you can actually run

You do not need a hacking lab. You need a structured, documented and authorized exercise. Federal practice organizes it into five phases: plan, authorize, execute, analyze and report. Here is what each demands, written as the exercise Priya should have run before launch and should run now.

1. Plan, starting from harm. Define what harm means for this system. For a benefits tool the worst harms are wrongful denial and discriminatory treatment. Name the harms first, because they tell you what to attack. Set objectives explicitly: find safety impacting vulnerabilities before deployment, validate the mitigations already in place, and generate regression tests.
2. Assemble a mixed team. Include security engineers, a data person, a caseworker who knows the real edge cases, a legal or civil rights advisor, and representatives of affected communities for the bias work. Federal practice puts domain experts on the team by design, meaning clinicians for health systems and field operators for enforcement systems. Homogeneous teams miss the failures that hurt people unlike themselves.
3. Authorize in writing. Obtain written authorization from the system owner and the authorizing official. Clarify what data the red team may use, whether live prompts against production are permitted, and how findings will be handled and disclosed. Federal coordinated vulnerability disclosure policy applies to vulnerabilities affecting federal systems, so the disclosure path belongs in the rules of engagement rather than in an argument afterwards.
4. Execute across the taxonomy. Work through each tactic and each applicable risk category. Aim for 20 to 40 concrete test cases, each naming an input and the harm it probes. Document the unsuccessful attempts as carefully as the successful ones, because the record of what you tried is what makes the coverage claim checkable.
5. Analyze and rate. Categorize by severity using an AI adapted vulnerability scoring approach, estimating exploitability and impact. Score each finding by severity, meaning how bad the harm, and likelihood, meaning how easily it happens in real life.
6. Report and route. Produce an executive summary, the technical findings, proposed mitigations, the residual risk you are accepting, and a regression test suite. Route it to the officials who can act: the Chief AI Officer, the governance board and the authorizing official. Then assign fixes and re-test, because a finding is not closed until the attack has been re-run against the fix.

A typical engagement on a production system runs two to six weeks, with continuous lightweight testing layered on top of periodic deeper engagements. The re-test step needs one caveat that people skip: re-running the original attack shows that the specific attack no longer works. It does not show that the underlying weakness is gone, because a small variation may still succeed. Turn every finding into a regression test that runs on every release, which is the discipline of making findings become tests.

What the large public exercises found

In 2023 at DEF CON 31, the AI Village, Humane Intelligence, SeedAI and the White House Office of Science and Technology Policy coordinated what was then the largest public AI red-teaming exercise, with approximately 2,200 participants probing production models across eight risk categories. The follow up exercise in 2024 extended it to agentic systems and federal use cases in coordination with the U.S. AI Safety Institute. Together they produced the first systematic public corpus of successful attacks against deployed models, which is why their findings are worth reading before you design your own tests.

The recurring findings were these: prompt injection succeeds easily against most models without hardened scaffolding; jailbreaks transfer across model families, so a technique developed against one system often works against another; role play and persona framing is highly effective against alignment; crafted unicode and non English language inputs evade filters; models leak training data when prompted with partial matches; and models will produce unsafe biological and cybersecurity content when the request is framed as fiction. Budget for these techniques in your own exercise rather than rediscovering them, and treat every one of them as a starting point rather than as the complete list, because the technique catalog changes faster than any curriculum.

Interpreting results without panicking or shrugging

Red-teaming will always find problems. The skill is sorting them. Use a severity by likelihood grid: a high severity, high likelihood finding such as the tribal address bias is a launch blocker, while a low severity, low likelihood finding such as an odd date format before 1900 goes on the backlog. The mistake leaders make is treating every finding as either a crisis or noise. The grid forces a proportionate response and produces a defensible record of which risks were accepted, by whom and why.

The harder interpretive discipline concerns the tests that pass. A red team result is evidence about what you looked for, using the techniques you used, on the system as it was configured that week. It is never evidence that a vulnerability does not exist. No number of clean runs converts into a guarantee, no completed exercise makes a system jailbreak proof, and a report with no findings is more likely to indicate a narrow scope or an inexperienced team than a secure system. Write your conclusions in that voice, because the sentence "we found no leakage in this round" survives later scrutiny and the sentence "the system does not leak data" does not.

A red-team finding log you can use Monday

This is the artifact that turns an exercise into accountability. Every finding gets a row, no rows are ever deleted, and the log feeds the risk register rather than living in a slide deck. Note the wording of the third row: it records what the team observed, not a conclusion about what is absent.

Tools, infrastructure and their limits

Open tooling has matured. PyRIT and garak are widely used for automated probing of language model systems, and evaluation suites including HELM, BIG-bench, TruthfulQA, ToxiGen and BOLD cover capability, truthfulness, toxicity and bias, with jailbreak prompt datasets available for adversarial work. Federal evaluation programs run by the U.S. AI Safety Institute assess risks and impacts on a shared basis. Commercial red-team platforms also exist; evaluate any of them against your own criteria rather than a capability claim, and do not treat a tool's clean report as a conclusion about your system.

Tools catch a fraction of the attack space. Automated scanners find known patterns quickly and miss the creative attacks that matter, which is why every serious framework insists on human adversarial thinking alongside the tooling. The infrastructure requirements are unglamorous and load bearing: isolated test environments, logging at prompt and response granularity, version control for attack artifacts, and chain of custody for evidence you may need later. For classified environments, cross domain considerations apply and should be resolved with your security office before the exercise rather than during it.

Cases worth internalizing

An early public chatbot was induced through coordinated prompting to produce racist content within 24 hours of release in 2016, a failure of never having tested against an adversarial user population. Clearview AI's scraping of social media photographs for a facial recognition database used by law enforcement agencies produced litigation under the Illinois Biometric Information Privacy Act and a settlement in 2022, illustrating that supply chain provenance is a legal exposure and not only a technical one. A widely used consumer AI service exposed other users' conversation titles and partial payment information in 2023 through a defect in a caching library, a reminder that AI systems inherit the entire infrastructure attack surface beneath them.

Employees at a large manufacturer leaked proprietary source code by pasting it into a public chat service in 2023, which is the social engineering surface around permitted employee AI use rather than an attack on a model. A 2024 tribunal ruling held an airline liable for incorrect refund information its chatbot gave a customer, establishing that an organization owns what its agent says. New York City's MyCity chatbot gave legally incorrect advice to small businesses in 2024, including guidance that encouraged unlawful action, and correcting it took several rounds of press scrutiny. The IRS ID.me episode in 2022 shows vendor and supply chain review failures around AI adjacent biometrics. Michigan MIDAS and the Dutch childcare benefits scandal remain the standing cautionary tales for fraud detection systems that were never adversarially tested against the people they would accuse.

Integrating red-team output with governance

Red team findings feed the MEASURE function of the AI Risk Management Framework by supplying empirical evidence about safety, security and resilience, robustness and managed bias. They feed the MANAGE function by forcing a decision on each risk: mitigate, avoid, transfer, or accept with documentation. Under M-24-10 minimum practices for safety impacting AI, agencies conduct impact assessments, perform testing including red-teaming where appropriate, monitor on an ongoing basis and provide human oversight, so red team output is not a separate workstream but an input to obligations that already exist.

Practically, that means findings are tracked in a risk register reviewed at governance board cadence and summarized in the annual AI use case inventory. For vendor supplied AI, require delivery of red team evidence as a contract deliverable, with the right to repeat red-teaming on the delivered system yourself. Read vendor evidence for what it is: a report on the tests the vendor chose to run. Independently spot checking a sample of it, on your own data, is what turns a deliverable into assurance. After an incident, the red team capability doubles as root cause analysis and remediation input. Over several years, mature programs build internal red team cadres with domain depth that external contractors cannot match, supplemented by periodic outside engagements for a fresh perspective.

Anti-patterns

• Red-team as a safety certificate. Treating a completed exercise, or a fixed number of clean runs, as proof that a system is safe or cannot be jailbroken. Findings are evidence of what you found. They are never proof of what is not there, and any report or briefing that implies otherwise should be sent back.
• Red-team theater. A one week contractor sprint that produces a glossy report with no regression tests, no remediation plan and no follow up.
• Scope collapse. Limiting testing to model level attacks and ignoring the sociotechnical tier, the vendor supply chain and agentic tool abuse.
• Findings without remediation. Documenting vulnerabilities that are never fixed because no one owns the model.
• Single shot engagement. One pre deployment exercise and nothing afterwards, while the technique landscape keeps moving.
• Homogeneous team. No domain experts, no affected community representation, no sociotechnical perspective, and therefore no chance of finding the failure that hits people unlike the testers.
• Tool only red-teaming. Running an automated scanner without human creativity, then reporting the scanner's coverage as your coverage.
• Over sharing. Publishing attack artifacts before coordinated disclosure, which arms the next attacker before the fix ships.
• Under sharing. Hoarding findings inside the originating program so no other agency learns anything from them.
• Metric tyranny. Counting attempts, hours or test cases instead of measuring actual risk reduction.
• Benchmark in a red team costume. Running standard evaluations and reporting the result as adversarial testing.
• Empty log as good news. Reading a report with no findings as a strong result rather than as a question about scope, technique and team composition.

Practice prompts

1. Write the harm statement for one deployed system: who is hurt, and how badly, when it fails. Everything else in the exercise design follows from that statement.
2. Take one form field in your system and construct a set of inputs that differ only in a demographic signal. Run them. Record what happened even if nothing did.
3. Draft the rules of engagement for an exercise on your highest risk system, including who authorizes it, whether production is in scope, and how a finding gets disclosed.
4. Walk the twelve generative risk categories against one system and write one sentence per category saying whether it applies and why. File the ones you dismissed.
5. Take your last vendor test report and list what it did not test. That list is the first draft of your independent verification plan.
6. Pick one closed finding from any system and write the regression test that would catch a variation of it, then check whether that test runs on every release.

Reflection

The uncomfortable fact in Priya's story is that everyone did their job. The vendor tested and reported accurately. The agency reviewed the results. The number was true. What nobody did was ask who the system would fail, and that omission was invisible until an outside group with different priorities went looking. Ask yourself who in your organization is paid to try to break your AI systems, and whether they have the standing to stop a launch. If the answer is that testing is owned entirely by the people whose deadline depends on the launch, then the tribal address failure in your own program already exists and simply has not been found yet.

Glossary

• AI red-teaming. A structured testing effort to find flaws and vulnerabilities in an AI system, often in a controlled environment and in collaboration with the system's developers.
• Prompt injection. Crafting input that causes a model to follow the attacker's instructions instead of the system's.
• Indirect prompt injection. The same attack delivered through content the system retrieves and reads, rather than through anything the user typed.
• Jailbreak. Input designed to bypass a model's safety behavior, frequently transferable between model families.
• Data poisoning. Planting training examples that cause chosen behavior on chosen inputs later.
• Model inversion. Reconstructing training inputs from a model's outputs or behavior.
• Membership inference. Determining whether a particular record was part of the training data.
• Model extraction. Reconstructing a functional copy of a model through repeated queries to its interface.
• Sociotechnical tier. The attack surface created by people, incentives and institutions around the model, including misuse by authorized users.
• Rules of engagement. The written scope, permissions and disclosure path agreed before an exercise begins.
• Coordinated vulnerability disclosure. The agreed process for reporting a vulnerability so it can be fixed before it is published.
• Regression test. A test derived from a past finding that runs on every release to confirm the failure has not returned.

Related lessons

• Bias Detection and Mitigation at Scale deepens the fairness thread: how to measure and mitigate bias systematically across protected classes and intersectional groups.
• Enterprise AI Risk Management is where red team findings land once they become risk register entries.
• Cybersecurity for AI Systems covers the infrastructure attack surface underneath the model.
• AI Supply Chain Risk covers the components you inherited before your testing ever began.
• Minimum Risk Management Practices covers the pre deployment testing obligation that red-teaming satisfies.
• Risk Classification: Safety-Impacting vs. Rights-Impacting determines which systems carry the heavier testing expectations.

Closing

Priya's agency now runs a red team exercise before every rights affecting deployment, requires red team evidence from vendors as a contract deliverable, and spot checks a sample of that evidence on its own data. None of that makes the screening tool safe. It makes the agency the party that finds its own failures, on its own schedule, with time to fix them and a written record of what it accepted and why. The cheapest red team is the one your contract required a vendor to run. The most expensive is the legal aid group that finds the failure for you two weeks after launch, and it charges in credibility rather than in dollars.

Key takeaways

• Red-teaming asks who the AI fails, not whether it works. Average accuracy hides the worst case failures that land on the smallest and most vulnerable groups.
• Findings are evidence, never a guarantee. A clean result describes what you tried on the configuration you tried it against, and no number of passes makes a system jailbreak proof.
• Cover three tiers. Model, system and sociotechnical attacks fail differently, and testing only the model leaves two of the three tiers untouched.
• Probe four practical surfaces. Bias and fairness, input manipulation, poisoning and edge cases, and privacy leakage each need their own scenarios.
• Use a taxonomy, not inspiration. A published attack taxonomy and the twelve generative risk categories turn coverage into something you can report and check.
• Authorize in writing before you touch anything. Scope, data permissions, production access and the disclosure path belong in rules of engagement agreed in advance.
• Mixed teams find what homogeneous teams cannot. Domain experts, civil rights advisors and affected community members are part of the method, not a courtesy.
• Turn every finding into a regression test. Re-running one attack proves that attack fails now, so protect against the variation with a test that runs on every release.
• Sort by severity and likelihood. The grid separates launch blockers from backlog items and produces a defensible record of accepted risk.
• Make red-teaming a contract deliverable and verify it yourself. Vendor evidence reports the tests the vendor chose to run, which is why independent spot checking is the part that matters.

Frequently Asked Questions

Does passing a red team exercise mean our system is safe to deploy?

No. It means that the techniques your team used, against the configuration in place that week, did not surface a blocking finding. That is genuinely useful evidence and it is not a safety certificate. Systems change, techniques change weekly, and scope always excludes something. Write the conclusion as what you tested and what you found, keep the residual risk statement explicit, and schedule the next engagement rather than treating the last one as a permanent result.

How is this different from the penetration test our security team already runs?

A penetration test targets networks, hosts and applications, and it is the right tool for those. AI red-teaming targets the model, the training data, the inference pipeline and the human system around them, where the failure mode is often a correct system doing an unjust thing rather than an unauthorized access. Both are needed. Neither substitutes for the other, and reporting one as the other is a documented anti-pattern that auditors specifically look for.

We have no in house expertise. Can we just buy this?

You can buy an engagement and you should still own the design. Contract for the exercise, require the finding log and regression tests as deliverables rather than a report, keep the right to repeat testing on the delivered system, and have someone inside who can read the findings critically. Over time the goal is an internal cadre with domain depth, supplemented by external engagements for fresh perspective, because outsiders bring technique and insiders bring knowledge of what actually harms your applicants.

How many test cases is enough?

Twenty to forty concrete scenarios is a reasonable first exercise for a single system, with each scenario naming an input and the harm it probes. Do not treat that as a completion criterion. Coverage is better described by which tiers, which attack surfaces and which risk categories you addressed, and by which ones you consciously excluded and why. A count of test cases is exactly the kind of metric that measures effort rather than risk reduction.

Should we publish our red team findings?

Publish what helps and withhold what arms an attacker before the fix ships. Coordinated disclosure practice exists for exactly this balance, and both extremes are named anti-patterns: publishing attack artifacts early enables downstream abuse, while hoarding findings prevents any other agency from learning from them. Summarizing findings for the use case inventory and sharing techniques through cross agency channels is usually the workable middle.

What do we do when the red team finds something we cannot fix before launch?

You make the acceptance explicit and you make somebody sign it. Record the finding, its severity and likelihood, the mitigation you are putting in place instead, the residual risk that remains, and the name of the official who accepted it. That record is what distinguishes an informed risk decision from an oversight, and it is the first document anyone will ask for if the risk materializes. A high severity, high likelihood finding on a rights affecting system is a launch blocker, not a signature.