←
AI-Ready Foundation
Visionary · M3 · lesson 3 of 5 · queued
Preview — browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll →
Continuous AI Assurance: Red Teaming, Model Risk Management, and Audit-Ready Evidence
📖
now learning

Continuous AI Assurance: Red Teaming, Model Risk Management, and Audit-Ready Evidence

15 min

Continuous AI assurance is how you answer the question internal audit will eventually ask about every AI system you run: is the thing in production today the thing you approved, and is it still safe? Too often the answer is last year's slide deck. This lesson replaces it with tests that run on every change, human red teaming where stakes are high, independent challenge that can block a release, and stored AI audit evidence an auditor can read in minutes. You leave with an Assurance Evidence Matrix and one number that shows whether your assurance keeps pace with your vendors.

Where you are on the ladder: this is Level 5, AI Native, lesson 4 of 5. The previous lesson, The Data Flywheel, gave you Eval Set v1, a versioned file of real cases with expected answers and graders, and Lesson 22 gave you the Agent Register. You may feel you have met continuous evaluation before. You have: Lesson 18 on AI observability gave your engineers traces, live quality signals and an incident runbook. That is the engineering view. What is new here is the independent view: a challenger with the standing to stop a release, evidence an auditor accepts without an engineer translating, and a clock set by your vendors' notice periods. The next lesson, AI Strategic Resilience, asks what happens when a model is withdrawn rather than upgraded.

The Problem: An AI Red Team Finds a Bug Nobody Was Looking For

In January 2025 Microsoft's AI red team published what it had learned from attacking more than 100 generative AI products, in a Microsoft Security blog post of 13 January 2025 and the white paper behind it. A red team is a group whose job is to attack your own systems the way a real adversary would, so you find the failures first. Microsoft formed its AI red team in 2018, and by October 2024 it had run more than 80 operations. The paper sets out eight lessons, five case studies, and a simple way of describing any attack: who the actor is, what tactics they use, which weakness they exploit, and what the impact is.

One case study should change how you staff your testing. Examining a video-processing AI application, the team did not find a clever prompt. It found a server-side request forgery flaw, a classic web vulnerability that lets an attacker make the server fetch things it should not. The cause was an outdated FFmpeg component, a widely used library for handling video files.

Nobody needed to trick a model. Somebody needed to check a library version, which is routine application security work. Now look at your own organization. A separate "AI risk" team that only looks at prompts would have missed it. An application security team that assumes AI products are someone else's job would have missed it too. The bug sat in the gap between the two.

The eight lessons, in the order the paper on arXiv (2501.07238, January 2025) lists them, are worth pinning to the wall of whoever runs your testing:

  1. Understand what the system can do and where it is applied.
  2. You don't have to compute gradients to break an AI system.
  3. AI red teaming is not safety benchmarking.
  4. Automation can help cover more of the risk landscape.
  5. The human element is crucial.
  6. Responsible-AI harms are pervasive but difficult to measure.
  7. LLMs amplify existing security risks and introduce new ones.
  8. The work of securing AI systems will never be complete.

The last one is why this lesson exists. If the work is never complete, one review before launch cannot be the answer. You need a loop.

What does the gap look like inside your firm?

Picture this, as an illustration. Internal audit's request arrives on a Monday: show us that the system running today is the system you approved. The system is a customer-facing assistant that passed a thorough review eleven months ago. Since then the vendor has swapped the underlying model version twice, the product team has edited the prompt nine times, and two tools have been connected so the assistant can look up order status and open a service desk ticket. Nobody retested, because nobody was asked to.

You cannot show it. The approval describes a system that no longer exists. Nothing has gone wrong yet, which is what makes it expensive: you are carrying risk you cannot see, and the first person to see it may be a customer.

What does weak AI assurance cost?

It costs you in two opposite directions, and a third pressure removes the safety net.

  • Assurance that is too slow blocks delivery. In ETR's survey of 200 firms, reported by Techstrong.ai on 22 September 2026 and examined in full in Lesson 25, firms piloting open-weight models cited incomplete security and compliance reviews (62%) far more often than model quality (10%) as the obstacle to production. The review queue, not the model, holds those pilots back.
  • Assurance that is too thin lets change through unseen. Anthropic's model deprecations page promises at least 60 days' notice before a public model is retired, and its log shows about one announcement every two months (Lesson 25 examines it). That is one vendor. Add the others you use and model change is routine, which an annual review cannot keep up with.
  • The regulators are not supplying the bar. On 17 April 2026 the Federal Reserve, OCC and FDIC replaced SR 11-7, the long-standing US bank model risk guidance, with SR 26-2, "Revised Guidance on Model Risk Management", which places generative and agentic AI outside its scope by footnote as "novel and rapidly evolving".

Who owns continuous AI assurance?

Ownership is split, and that is part of the problem. The chief risk officer or head of model risk owns the framework, the CISO owns security testing, internal audit consumes the evidence, and the AI platform lead owns the pipeline that produces it. Without a model risk function, the AI governance lead from Lesson 15 on AI governance and the model inventory chairs. Your job is to get those four around one shared artifact.

Why do organizations get AI assurance wrong?

Seven patterns recur; the failure-modes table below shows the fingerprint of each.

  1. Assurance as a gate. One review before launch, then nothing.
  2. Benchmarks mistaken for assurance. A leaderboard score says little about your deployed system.
  3. AI risk kept apart from application security. Classic bugs live inside AI products, as the FFmpeg case shows.
  4. Waiting for the rulebook. Waiting means having no bar at all.
  5. Manual evidence. Results live in emails and decks, so every audit is archaeology.
  6. Vendor models treated as someone else's problem. You cannot outsource accountability for a model you cannot see inside.
  7. Challenge without teeth. An independent reviewer exists on the org chart but has never stopped anything.

How Continuous AI Assurance Works: Smoke Detectors, Not an Opening-Day Certificate

Before any jargon, picture a building. On opening day it passes its fire inspection: the inspector walks every floor, checks every exit and signs the certificate.

Then the tenants move in. One knocks down a wall, another fits a kitchen. Six months later it is not the building the inspector saw.

So nobody relies on the certificate. They fit smoke detectors that test the air all day. Twice a year they run a fire drill, where real people try the exits and find one blocked by boxes. And the alarm panel keeps a log, so the returning inspector reads five minutes of history instead of interviewing everyone.

  • The smoke detectors are your automated test suites, running on every change and on a schedule.
  • The fire drill is human red teaming, scheduled for the systems where a failure would really hurt.
  • The panel log is your stored, versioned evidence, which turns testing into something an auditor can use.

Your AI systems are that building. Every model swap, prompt edit or new tool is a wall moved.

What is continuous AI assurance?

Continuous AI assurance means the same evaluation and attack tests run automatically on every change (model version, prompt, tool or data source) and on a schedule in production, with the results stored as evidence. Three parts of that sentence matter.

"The same tests" means like with like: if the test set changes every time, you cannot tell whether the system improved or the exam got easier. "On every change" means the trigger is the change, not the calendar. "Stored as evidence" means each result carries a version stamp: model, prompt version, tool list, test set version, date. Without the stamp, a pass is a rumour.

What is AI red teaming, and how is it different from benchmarking?

AI red teaming is deliberately attacking your own deployed system the way an adversary or a careless user would, to find failures before they do. Benchmarking scores a model in general against a public test. It is the difference between a car model's crash rating and a mechanic inspecting your actual car.

A benchmark says nothing about the tools you connected, the permissions the system runs under or the outdated library in the upload path. Quote Microsoft's lesson 3, "AI red teaming is not safety benchmarking", when someone offers a vendor's leaderboard score as assurance evidence.

Microsoft's lesson 2 is just as useful: attackers rarely need sophisticated mathematics, just patience, an understanding of what the system can do (lesson 1) and the habit of trying ordinary things. Lesson 7 adds that LLMs amplify existing security risks and introduce new ones. So your red team needs both an application security engineer and someone who understands AI-specific harms.

What is model risk management for generative AI after SR 26-2?

Model risk management is the discipline, long established in banking, of keeping an inventory of models, tiering them by how much damage a wrong answer could do, and having someone independent challenge them. For years the US reference text was SR 11-7, and some banks stretched it over their first large language model deployments as the nearest thing to a rulebook.

That stretch is over. SR 26-2 supersedes SR 11-7 and is "expected to be most relevant" to Fed-supervised banking organizations with over US$30 billion in total assets, according to the Federal Reserve's letter. Orrick's summary of 24 April 2026 adds that the new guidance:

  • is shorter and principles-based;
  • narrows the definition of "model", excluding simple arithmetic, deterministic rules and spreadsheets;
  • tiers models by inherent risk and materiality;
  • defines effective challenge by expertise, independence and standing;
  • keeps validation as conceptual soundness, outcomes analysis and ongoing monitoring;
  • adds a vendor section saying the principles still apply when third-party models cannot be fully validated;
  • raises the "most relevant" level from US$1 billion, which is a statement about relevance, not an exemption for smaller banks.

Be precise with your steering committee: SR 26-2 neither forbids nor permits any generative AI practice. It does not cover it, and banks must rely on broader risk management. The durable parts run through this lesson: one inventory, tiering by materiality, independent effective challenge, outcomes monitoring, and evidence about vendor models. If you are not a bank, those five are still a sound starting skeleton.

What does effective challenge mean?

Effective challenge means an independent reviewer with the expertise to understand the system, the independence to say no, and the standing to make that no stick. Standing is the part easiest to skip. A reviewer who can only write comments is an editor, not a challenger. Back in the building: an inspector who can write "exit blocked" in a report but cannot close the building is decoration.

The simplest test of standing: has your independent reviewer ever blocked or delayed a release? If never, either every system was perfect or the challenge is ceremonial.

Where does the EU AI Act stand after the Digital Omnibus?

The Digital Omnibus on AI was adopted as Regulation (EU) 2026/1744, published in the Official Journal on 24 July 2026 and in force from 27 July 2026, as White & Case reported on 4 August 2026. Stand-alone Annex III high-risk duties now apply from 2 December 2027 and Annex I product-embedded duties from 2 August 2028.

Three duties sit closer. Most Article 50 transparency duties have applied since 2 August 2026, including the deployer duty to label deepfakes. Providers of generative systems already on the market before that date have until 2 December 2026 to meet the Article 50(2) machine-readable marking (watermarking) duty; that extension is for providers only. Article 4 on AI literacy has applied since 2 February 2025 and now reads as a duty to take measures to support the development of AI literacy of staff. Confirm every date against the Official Journal text before it goes in a board paper; law-firm summaries are a starting point, not the law.

The dates moved; the direction did not. For generative and agentic systems your firm sets its own bar for now, so set it deliberately and write it down.

What does a break-fix cycle look like in practice?

A break-fix cycle is a repeated round of red teaming, measurement and mitigation, then red teaming again to check the fix held. Microsoft pairs it with two ideas. Automation gives coverage: in 2024 the team released an open-source tool, PyRIT, to automate and scale attacks. People give judgment: the human element is crucial and harms are hard to measure (lessons 5 and 6). And since no mitigation removes risk completely, the answer is defence in depth: several imperfect layers rather than one perfect wall.

Lesson 18 showed the other half: Morgan Stanley's expert-graded evaluations and daily re-run of sample questions, per OpenAI's own customer story. The re-run is the detector; break-fix is the drill.

The Workshop: Build Your AI Assurance Evidence Matrix

You make the decisions in one four-hour session and build the first pipeline in the following sprint. The output is the Assurance Evidence Matrix: one row per AI system, showing what evidence it must have, whether that evidence exists, where it is stored and who can stop a release.

Who should be in the room, and what should they bring?

Six roles, not all for the whole session. Assurance is a negotiation between people who want speed and people who must sign, so both sides attend.

  • Chair: head of model risk, or the AI governance lead if you have no model risk function.
  • Security: a CISO delegate and one application security engineer. The engineer is not optional; remember FFmpeg.
  • Platform: the AI platform lead, who will own the pipeline.
  • Audit: internal audit, to say what evidence they would accept. Audit advises; it does not design the controls it will later test.
  • Business: the owner of one high-tier system, for Steps 1 and 2, to keep tiering honest.
  • Vendors: procurement or vendor management, for Step 7, with the contracts.

Bring the AI system inventory, Agent Register, Eval Set v1, vendor list, last audit findings, release log and regulatory calendar. A rough clock: Steps 1 and 2 take the first 70 minutes, Steps 3 to 6 the next 80, Steps 7 to 10 the last 75, with a break. Steps 4 and 8 are designed in the room and built in the sprint.

Step 1. Merge one inventory and tier it

Merge AI systems, models (including those inside vendor products) and agents into one list, then tier each by materiality: the worst realistic outcome of a wrong or manipulated output. Three tiers are enough to start.

Ask the business owner: "If this system gave a confidently wrong answer to a thousand people tomorrow, what happens?" A customer told the wrong loan status is tier 1. A branch employee who gets a poor policy summary and checks the source is tier 2. A meeting summary nobody acts on is tier 3. Write the rationale next to the tier; that is what audit will challenge.

Step 2. Define the required evidence per tier

For each tier, state what must exist before release and in production. Lower tiers get less; that is how assurance stops blocking delivery. Get the chair and audit to agree the table. This template is illustrative.

Evidence itemTier 1 (high)Tier 2 (medium)Tier 3 (low)
Automated eval suite on every changeRequiredRequiredOptional
Automated attack suite on every changeRequiredStarter setNot required
Scheduled run in productionDailyWeeklyOn vendor change
Human red team exerciseScheduled, plus on major changeOn major changeNot required
Production monitors with thresholdsRequiredRequiredIncident log only
Vendor model provenance recordRequiredRequiredRequired
Independent challenge sign-offEvery material changeAnnual and on major changeAnnual sample

Every tier needs a provenance record: you cannot tier a system honestly without knowing which model is inside it.

Step 3. Fix the evidence record and who can read it

Agree the version stamp every test run must carry: system, model version, prompt version, tool list, eval and attack set versions, date, pass or fail per case, and who approved the release. Then agree where it lives, and give audit read access that does not depend on an engineer. This panel log is what makes evidence audit-ready rather than merely existing.

Step 4. Automate the suites in the release pipeline

Wire Eval Set v1 and a starter set of attack tests into the release pipeline so they run on every change to model, prompt, tool or data source, and on a schedule in production. This is LLM regression testing: the same cases, re-run, so a change that breaks yesterday's good answer is caught before customers see it.

The attack starter set need not be clever on day one. Begin with failures you already know: the prompt injection and oversharing probes from Lesson 13 on AI security, and the questions that once embarrassed you in a pilot. Sort them against the OWASP Top 10 for LLM Applications 2026 (OWASP GenAI Security Project, August 2026), which keeps Prompt Injection at LLM01 and lifts Excessive Agency to LLM03, so gaps in coverage are visible. The set only grows; a test is never deleted because it became inconvenient.

Step 5. Scope human red teaming for the top tier

Book human red-team exercises for tier 1 systems and scope each one to cover both classic security (libraries, access, network calls, injection paths) and AI-specific harms. Then run a break-fix cycle: findings, fixes, and a re-test that proves the fix held.

Write one line into every scoping note: "Review every third-party component in the data path, not just the prompts." That is the FFmpeg lesson made operational. Then turn every confirmed finding into an automated test, so the drill's discoveries become the detector's routine.

Step 6. Establish effective challenge with real standing

Name the independent function that challenges tier 1 and tier 2 systems, confirm it has the expertise to read the evidence, and write into the governance charter, not an email, that it can block a release. Record who holds the blocking authority and who can overrule it, because an overrule with no record quietly removes the standing.

Step 7. Bring vendors into third-party AI risk management

For each AI vendor, record model provenance, concentration, contract clauses and the requalification step that fires when the vendor changes a model. The Cloud Security Alliance research note of 11 March 2026 gives a practical list: an AI vendor inventory including model provenance inside SaaS products, workload concentration per provider, migration time for critical workloads, and contract reviews for continuity, data portability and notification of government action.

Include the agentic supply chain, which the OWASP Top 10 for Agentic Applications (December 2025) names as ASI04: the tools, plugins, connectors and other agents your agents call. Your Agent Register lists the tools; add who supplies each and what happens when it changes.

Step 8. Choose production monitors and the incident-to-test rule

Pick the signals that trigger a re-test: drift, spikes in human overrides, and incidents. When reviewers start correcting more outputs, something changed. Write the rule that every incident becomes a test case, and record, for each incident, whether monitoring caught it or a customer did.

Step 9. Map evidence to the rules, and say where you set your own bar

Line up the stored evidence against the EU AI Act dates and any sector guidance, and state plainly where the firm sets its own bar. Rows might read: "Article 50 deepfake labelling, deployer duty since 2 August 2026, evidence: disclosure text in release record"; "Article 50(2) marking for our image generator placed on the market in 2025, provider deadline 2 December 2026, evidence: marking test in the attack suite"; "Generative AI at a US bank, outside SR 26-2 scope, own bar: tier 1 evidence set approved by the risk committee." Writing "own bar" down is not a weakness; it shows you decided on purpose.

Step 10. Estimate requalification time against the notice window

Pick one tier 1 system. In the room, estimate from the last model or prompt change in your release log how many days it took from the change notice to the independent challenger signing the pass. Then measure it for real on the next change and record both numbers. Compare them with a 60-day notice window.

Count the waiting, not just the testing: the red-team queue, the sign-off slot, the week someone was on holiday. The target is yours to set, but if the number is longer than the notice period, you have found your first project.

Worked example: what one afternoon reveals

Illustrative only. Picture a regional bank with about 4,000 staff and eleven AI systems, below the US$30 billion level at which SR 26-2 is described as most relevant (no exemption), with generative AI outside that guidance anyway. Three rows of its first matrix:

SystemTierRequired evidenceAutomated today?Last human red teamRequalification (estimated)
Customer-facing loan-status assistant1 (high)Evals each release, attack suite, quarterly red team, monitoringEvals yes, attacks no11 months before the session9 weeks
Internal policy search for branch staff2Evals each release, monitoringYesNever (not required)4 days
Meeting-notes summarizer inside a SaaS suite3Vendor provenance record, annual checkNo, and model provenance unknownNot requiredUnknown

Three findings fall out. First, the tier 1 assistant needs about nine weeks to requalify, longer than a 60-day notice: if its vendor announced a retirement, the bank would have to run an unqualified model or switch the assistant off. Second, its attack suite is manual, so running it is a project and nobody starts it. Third, the bank cannot say which model sits inside the SaaS summarizer, so it cannot even tell whether a vendor change has happened.

The first sprint's backlog writes itself: automate the attack suite, add "check every third-party library in the document-upload path" to the assistant's red-team scope, pre-book the sign-off slot so requalification starts the day a notice arrives, and send the SaaS vendor a provenance question.

The artifact: your Assurance Evidence Matrix

One row per AI system with the fields below, in a shared table the pipeline updates wherever it can; humans fill in the fields that need judgment.

FieldWhat to record
System and ownerName and a named accountable person
Tier and rationaleTier plus the worst realistic outcome
Model and vendorModel version, vendor, provenance for embedded models
Required evidenceThe tier's evidence set from Step 2
Test suites and triggersEval set version, attack set version, on change and on schedule
Results locationWhere version-stamped results are stored, and audit's access
Human red teamLast date, open findings, fix dates
Independent challengerNamed function and person, with blocking authority confirmed
Production monitorsMonitors and thresholds you chose
Incident-to-test logEach incident and the test it became
Regulatory mappingRule or guidance, applicable date, evidence item, or "own bar"
Requalification timeEstimated days, measured days and your target

Once the pipeline writes to this table, the auditor reads the panel log.

Failure Modes and the Metrics That Catch Them

Each of the seven patterns from the opening leaves a fingerprint in your own records. Look for the fingerprint, not a confession.

Failure modeHow to spot it
Assurance as a gateThe latest test date is older than the latest model, prompt or tool change
Benchmarks as assuranceThe evidence file cites a public leaderboard or vendor score and no test of your own deployment
AI risk outside application securityRed-team scope covers prompts only, with no library, access or network review
Waiting for the rulebook"Not yet required" appears as the reason a control is missing
Manual evidenceAnswering an audit request needs an engineer to search email or rebuild a run
Vendor models as someone else's problemA system's model field reads "vendor managed" or is blank
Challenge without teethThe independent reviewer has never blocked or delayed a release

What is the fastest way to find out if your assurance is a one-off gate?

Compare two dates for each tier 1 system: the last change to model, prompt or tools, and the last full test. If the test is older, the approval describes a system that no longer exists. It takes ten minutes with a release log.

How do you avoid assurance that blocks delivery?

Tier ruthlessly and automate the heavy lifting. Tiering gives low-risk systems a light path, automation removes the waiting from the high-risk path, and pre-booked sign-off slots remove the calendar delay. Slow assurance is not safer; teams route around it.

Which metrics should you track for continuous AI assurance?

Track six metrics, report them to the same committee each month, and set your own targets. No public benchmark for "good" exists, so your baseline is your first measurement.

MetricHow to measure itWhat it tells you
Evaluation coverage of the inventorySystems with an automated suite divided by systems in the inventory, per tierHow much of the building has detectors
Time to requalify a model changeDays from change notice to signed-off pass, against the 60-day windowWhether assurance keeps pace with vendors
Open red-team findings and time to fixCount by severity and median days openWhether break-fix cycles actually close
Vendors with provenance documentedVendors with a model-provenance record divided by AI vendorsHow much of your risk you can see
Incidents caught by monitoring versus by customersSource of first detection for each incidentWhether the detectors are wired to the right rooms
Releases challengedReleases the challenger delayed or blocked, with reasonsWhether challenge has standing

When customers find problems first, your detectors are in the wrong rooms. And a challenge count of zero for a year is a finding, not a success.

Frequently Asked Questions

Does SR 11-7 still apply to generative AI at US banks?

No. On 17 April 2026 the OCC, Federal Reserve and FDIC issued revised interagency guidance (the Federal Reserve's SR 26-2) that rescinds SR 11-7 and leaves generative and agentic AI out of scope by footnote, so banks must rely on broader risk management and set their own bar. The durable parts to keep are inventory, tiering, effective challenge, outcomes monitoring and vendor evidence.

When do the EU AI Act high-risk obligations apply, and what falls due on 2 December 2026?

Under Regulation (EU) 2026/1744, stand-alone Annex III high-risk duties apply from 2 December 2027 and Annex I product-embedded duties from 2 August 2028. On 2 December 2026, providers of generative systems placed on the market before 2 August 2026 must meet the Article 50(2) marking duty; other Article 50 duties, including deployer deepfake labelling, have applied since 2 August 2026. Confirm every date against the Official Journal text.

How often should we retest an AI system?

On every change and on a fixed schedule, not once a year. Anthropic's published policy gives at least 60 days' notice before retiring a public model, so your pipeline must be able to requalify a replacement inside that window; estimate how long it takes you today, then measure it on the next change.

Can we rely on our AI vendor's own testing?

No, not on its own. A vendor tests its model in general, not your deployment with your prompts, tools and data, and the April 2026 US guidance keeps the principles in force when third-party models cannot be fully validated. Ask for model provenance and change notice, and run your own suites on every vendor change.

How do I red team an AI system without AI security specialists?

Start with the application security engineers you already have, paired with someone who knows the business harms. Microsoft's red team says you don't have to compute gradients to break an AI system, and its FFmpeg finding was classic security work. Scope libraries, access and injection paths as well as prompts, fix, re-test, and turn each finding into an automated test.

Key Takeaways

  • An approval describes a moment, not a system. Every model, prompt, tool or data change makes it older. Continuous AI assurance re-runs the same tests on every change and stores the results as evidence.
  • Detectors, drills and a log. Automated suites give coverage, human red teaming gives judgment, version-stamped results give audit evidence.
  • Test the whole deployed system. Microsoft's red team found a classic server-side request forgery bug from an outdated FFmpeg library inside an AI app, and a benchmark score would never have shown it. Put application security in the room.
  • Nobody is handing you the bar. SR 26-2 leaves generative and agentic AI out of scope, and EU high-risk dates moved to December 2027 and August 2028, while Article 50 marking falls due for existing generative systems on 2 December 2026. Set your own bar deliberately and write it down.
  • Challenge needs standing. An independent reviewer who has never blocked a release is not yet effective challenge.
  • Know your requalification time. Estimate it from your release log, measure it on the next change, and if it is longer than a 60-day notice window, that is your first project and the bridge to Lesson 25.