←
AI Readiness & Process Transformation
Strategic · M16 · lesson 16 of 25 · queued
Preview — browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll →
📖
in this lesson

Security and Privacy Architecture for AI Programs

15 min

It is 5:50 on a Thursday and the eleventh team is nearly done. Their pilot goes to the steering committee on Monday, the analysis needs one clean pass over a customer export, and the sanctioned AI tenant will not accept a file that size without an access request that they have been told, by someone who heard it from someone, takes about two weeks. So a bright, conscientious, entirely well-meaning analyst opens a personal account on a consumer AI tool, drags in a spreadsheet with 8,400 customer records, and gets a beautiful answer in ninety seconds. Nobody is careless here. Ten other teams in the same portfolio ran the privacy pre-flight properly, and this team would have too, with another week. The exposure that follows is modest. The disclosure obligations are not. And when the review happens, the question that lands is not about the analyst at all. It is the sentence this lesson exists to answer: what controls prevented this?

Eleven Checklists, One Incident

Level 2 gave you the Data Pre-Flight Checklist: five flag families (personal data and special categories, confidentiality and contract boundaries, tool and tenancy boundaries, retention and deletion, access and authorization) run before a pilot touches a dataset. It is a good instrument, it has saved careers, and at pilot scale, one team running one experiment, it is exactly right.

Now scale it. Your portfolio has eleven items across three waves, carried by three platforms, touched by seven vendors, owned by six functions. Run the pre-flight eleven times independently, with eleven people interpreting five flag families against no shared definitions, and you get eleven different answers. Not because anyone is negligent, but because "is this confidential?" and "is this tenant approved for that?" are not questions a checklist answers on its own. They are questions it is supposed to look up, and when there is nothing to look up, every team invents an answer.

Then arithmetic does the rest. An organization relying on eleven-for-eleven human carefulness has quietly designed for a failure rate it cannot state out loud. It takes one team, once, under a deadline, choosing a shortcut that looked reasonable from inside their week. The manufacturer in the opening scene had ten immaculate pre-flights and one shortcut, and the ten did not protect them, because controls do not average.

What makes the review painful is the defense: "each team was individually responsible for checking." That is not a defense, it is the finding. The safe path existed only as an instruction, compliance depended on eleven separate acts of discipline, and nothing in the environment made the unsafe action harder than the safe one. The environment made it easier: two weeks for a sanctioned upload versus ninety seconds for a personal account. The architecture, such as it was, rewarded the incident.

Distributed diligence is not architecture. If the safe path depends on eleven teams each remembering, you have designed a failure rate you cannot state.

Your job as an enterprise strategist is to move these decisions up a level: from per-pilot checking to standing boundaries every use case inherits by default. Decided once. Provided as infrastructure. Published where anyone can find them. So the eleventh team on a Thursday evening needs no heroism, judgment, or two-week wait, just a rule that is already true and a tool that already works.

The reframe: it is not about the model

Most security and privacy conversations about AI go wrong in the first ten minutes because everyone stares at the model. Is it safe? Does it hallucinate? Could it be jailbroken? Real questions, mostly not yours, and mostly not where the incidents come from. Incidents come from two things an operations strategist can reason about with total precision:

  • Data movement. What leaves which boundary, into whose custody, held how long, in what derived forms. Every AI incident that ends in a disclosure letter is underneath a data-movement event: somebody's information went somewhere it was not permitted to go.
  • Identity. Who or what is acting, under whose authority, with which permissions, and whether that authority ends when the person does. Every AI incident that ends in an access finding is underneath an identity event: something acted with more authority than it should have had.

Governing either requires no model expertise. It requires knowing how work flows, where data lives, who may see what, and how to write a rule that survives contact with a deadline. That is your day job. This lesson turns it into four architectural decisions and one artifact.

Decision One: The AI Data Boundary Map

The artifact you leave with is the AI Data Boundary Map: one page, data classes across the top, sanctioned destinations down the side (orientation matters less than completeness), every cell carrying a verdict of permitted, permitted with control, or prohibited. Where it says permitted with control, it names the control, because "with appropriate safeguards" is not a control, it is a hope with a suit on.

Its value is being one artifact governing all use cases. Eleven teams do not each derive an answer; they read the same cell. A new use case at gate 0 does not open a debate; it locates itself on the grid. The map answers questions before they are asked, the only form of governance that survives a Thursday evening. Gartner found 63 percent of organizations lack or are unsure of AI-ready data practices, and this page is among the cheapest pieces of that practice to install.

Data classes, in business terms with your own examples

Classification schemes fail when they are abstract. "Confidential" means nothing until someone writes down that supplier pricing, the unsigned contract folder, and the site closure analysis are confidential here. Five classes are usually enough, and more than six is a scheme people stop using:

  • Public. Already published: the website, the price book, filed accounts, published standards.
  • Internal. Not secret, not for outsiders: SOPs (standard operating procedures), memos, org charts, routine operational reporting.
  • Confidential. Commercially damaging if leaked: contracts and their terms, supplier pricing, margin analysis, unreleased plans, source code, security documentation.
  • Personal. Data about identifiable people: customer records, employee files, anything meeting your jurisdiction's definition of personal data, including the PII (personally identifiable information) buried in free-text fields where nobody expects it.
  • Special category and regulated. Health, biometric, children's data, payment card data, anything sector-specific your regulator names. This class gives "it depends, ask privacy and legal" a home rather than letting it contaminate the whole map.

Write two to four real examples beside each class, drawn from your own systems. The examples make the map usable by someone who was not in the room when you built it, and that person is the actual user.

Destinations, enumerated honestly

The second axis is where data can go, and the discipline is honesty: list every destination that exists, not every destination you approve of. Four cover most enterprises:

  • The sanctioned enterprise tenant. Your contracted AI environment with negotiated terms, configured retention, and administrative visibility.
  • Embedded capability inside an existing system. AI features arriving inside your ERP, CRM, service desk, and document suite, often switched on by a vendor release rather than a procurement decision. This one surprises people because nobody bought it deliberately.
  • A partner or vendor environment. The implementer, outsourcer, agency, or consultancy analysing your data in their tenant.
  • A personal or unmanaged account. Yes, list it. MIT's research found a thriving shadow AI economy of employees using personal tools for real work while official pilots stalled, and your own census will confirm it. A map that omits this destination is a map of a company you do not run.

The control vocabulary

Keep the named controls few enough to memorize so a cell verdict is unambiguous. Five do most of the work: redaction (strip the sensitive fields before the data moves), pseudonymization (replace identifiers with tokens reversible only inside your boundary), tenancy restriction (contracted environment only, training on your data disabled, retention window defined, hosting region pinned), contractual control (only where the vendor lesson's clause set is signed and the vendor sits at the assessed tier), and prohibition. A sixth verdict is a route rather than a control: named approval required, meaning a specific human owner signs for this specific use, which keeps the hardest cells from collapsing into a blanket yes or no.

The storyline's map, rendered

Here is the 2,400-person storyline organization's map, illustrative but realistic in shape. Five classes, four destinations, twenty cells, one page.

Data classSanctioned enterprise tenantEmbedded AI in an existing systemPartner or vendor environmentPersonal or unmanaged account
PublicPermittedPermittedPermittedPermitted (still counted in the shadow census)
InternalPermittedPermitted where the platform decision record is on fileWith control: NDA plus named-project scopeProhibited
ConfidentialWith control: tenancy restriction (no training on our data, retention 30 days, region pinned)With control: vendor at assessed tier, AI features reviewed rather than defaulted onWith control: signed clause set, data returned or destroyed at closeProhibited
PersonalWith control: redaction or pseudonymization, purpose logged, retention setWith control: processor terms, no secondary use, no cross-tenant trainingWith control: processor agreement plus documented transfer basis, counsel firstProhibited
Special category and regulatedNamed approval: privacy owner signs per use caseNamed approval: route to privacy and legal before enablementProhibitedProhibited

Look at what this page does on a Thursday evening. The analyst holds personal data and is eyeing an unmanaged account: one cell, verdict prohibited, permitted route named on the same page (the tenant, with redaction), provided (this is Decision Four) the redaction tooling and tenant access actually exist and work.

The map and the governance catalog are the same decision

Earlier in this chapter you built a governance service catalog promising standing rules and a fast path: pre-approved patterns that let a team proceed without bespoke review. The boundary map is that promise seen from the data side rather than the rule side. Every "permitted" cell is a standing rule, every "permitted with control" cell is a fast path with a named prerequisite, and every "prohibited" or "named approval required" cell is a routing instruction to a human owner. You cannot publish a credible fast path without the map, because the fast path's whole claim is that somebody already decided, and the map is where those decisions live. Build them as twins, review them in one quarterly cycle, one owner accountable for both.

Decision Two: Identity and Authority as Infrastructure

The second decision generalizes a rule you met in Level 3 with agents, worth restating because it is the most-violated principle in enterprise AI: every AI system acts under a named identity with scoped permissions, never a borrowed human credential.

Named identity, scoped permissions

When a tool runs under a person's login, three bad things happen at once. It inherits everything that person can reach, which is almost never what it needs. The audit log records the person doing things they did not do, destroying the reconstruction standard your Level 3 audit trail depends on. And its access follows that person's career, growing on promotion and vanishing on exit, both at the worst moment. Named service identities fix all three: visible in the directory, permissions enumerated and reviewable, actions attributable, revocable without firing anybody.

At enterprise scale, add one thing pilots never need: an identity register listing every AI service identity with its owner, permission scope, and review date. An unowned service identity with broad permissions is the most quietly dangerous object in your estate.

The Friday question

Now the unglamorous control that prevents more incidents than any policy document. Human access to AI tools must flow through the organization's normal identity system: the same single sign-on, the same joiners-movers-leavers process that provisions a new hire's email and deprovisions a leaver's building pass. Not a separate vendor console with its own user list maintained by whoever set the platform up.

To find out whether that is true where you work, ask this out loud in your next platform meeting: "When someone leaves on Friday, what happens to their AI access?"

Then say nothing and watch the room. You will usually get a confident answer about the main tenant and a slow silence about the other two platforms, the vendor portal, the pilot workspace somebody stood up in March, and the contractors onboarded by email invitation because that was faster. The silence is the finding: a gap with an owner and a date, and the cheapest serious risk you will close all year.

In the storyline organization, the Friday question surfaced 14 former contractors still holding active access to a sanctioned AI tenant, several months past their end date, because that tenant had been provisioned by invitation rather than through the identity system. Closing it took a week: an access review against the leaver list, revocation, and a change so all three platforms provision through single sign-on with group membership that expires. Illustrative numbers, entirely typical shape. Notice that none of this is an AI problem. It is identity hygiene that AI tools joined, and it is exactly the standing infrastructure a strategist can insist on without knowing a thing about model weights.

Permission-aware retrieval, stated before procurement

The third identity requirement changes purchasing decisions, so state it precisely. If an assistant retrieves from your document store to ground its answers (the RAG pattern, retrieval-augmented generation, from Level 3), retrieval must respect the asker's permissions, not the system's. Otherwise any employee can ask a question whose honest answer requires a document they cannot open and receive a fluent paraphrase of it. That is not an assistant. It is an exfiltration tool with a friendly interface and no audit trail that looks like one, because from the log's perspective a legitimate user asked a legitimate question.

Two things make this architectural. First, it disqualifies a surprising number of products: many retrieval systems index everything into one pool under a single set of credentials with no concept of who is asking, a design that is efficient, common, and unusable above the Internal class. Second, finding out late is brutal. In the requirements document it is a vendor filter; in acceptance testing it is a re-architecture and a delayed launch; in production it is an incident with a disclosure obligation.

In the storyline, this blocked a use case at gate 1: a popular, well-scoped policy assistant on a platform whose retrieval could not enforce per-user rights over a document set containing personal and confidential material. Stopped at design, not in production. That is not the architecture failing, that is the architecture doing its job at the cheapest possible moment.

So it goes in writing, in the standard requirements pack, before procurement: retrieval must enforce the requesting user's existing access rights at query time, permission changes must propagate within a stated interval, and the vendor must demonstrate this in a test we design, using a low-privilege account against a document that account cannot open. Demand a demonstration, not an assurance. It takes twenty minutes and settles the question permanently.

Decision Three: Retention, Custody, and the Copies Nobody Classified

The third decision is the one organizations most often make eleven times by accident: how long anything is kept, and by whom. Retention is a governance question with a legal shadow, and your job is not to invent the answer but to ensure a single answer exists and applies everywhere.

What your organization retains

An AI program generates four new record types, each needing a retention answer aligned to your existing records policy rather than invented separately: prompts (frequently containing the sensitive content itself), outputs (which may have been acted upon), system logs (who used what, when, from where), and decision records (the Level 3 audit trail showing what an AI-touched decision was, who approved it, on what evidence).

Keeping everything forever and keeping nothing both fail, in opposite directions: an archive of your most sensitive conversations, discoverable in litigation and attractive to attackers, versus an inability to reconstruct a decision when a regulator or auditor asks. The resolution is boring and correct. Prompts and outputs follow the retention rule of the process they serve, so an AI draft of a customer letter is kept like a customer letter; logs follow normal security log retention; decision records follow the decisions they document, usually the longest of the four. Write it once, apply it to all eleven items, stop debating it per pilot.

What your vendors retain

All of that is irrelevant if the vendor keeps a copy on different terms. Vendor-side retention is set by contract, and the previous lesson gave you the clause set: no training on your data, retention window in days, deletion on termination with confirmation, sub-processor disclosure, notification of material change. Three additions belong at architecture level. Configured state must match contracted state, because a contract permitting zero retention alongside a tenant left on a ninety-day default is a control that exists only on paper. "Where are all the copies?" now includes prompts, logs, and vendor storage. And verification is scheduled rather than assumed: once a year, per vendor, someone confirms the setting survived two releases and a migration.

The blind spot: derived data is still your data

Now the finding most organizations miss entirely. A grounded assistant does not merely read your documents; it creates copies of them in other forms: an index of their contents, embeddings (numerical representations of text passages, stored so the system can find relevant material fast), chunks of original text held for retrieval, and caches of recent results. These can reproduce substantial parts of the originals, and they live wherever the system lives, frequently not where the sources live.

The architectural rule is one sentence: derived data inherits the classification of its source. An index over the contract repository is confidential. Embeddings of employee records are personal data. This bites in three places: hosting (a confidential index may not sit in a region the confidential class prohibits), access (the index must be permission-aware, where Decisions Two and Three shake hands), and deletion (removing a source document while its embeddings remain does not satisfy a deletion obligation).

In the storyline, this surfaced at architecture review for a contract-lookup assistant. The source repository sat in the contracted region under the confidential class; the proposed grounding index was to be built in the vendor's default region, which the map prohibits for that class. Caught at review, the fix was a hosting-region change agreed before build: a configuration decision and a short delay. Caught after go-live, the same finding costs an index rebuild, a re-verification cycle, and a disclosure conversation with lawyers who will reasonably ask how long the data was there. Cheap before, expensive after, and the only difference is whether anyone wrote down that derived data has a class.

Decision Four: Provide What You Require, and Name the Line

The fourth decision separates an architecture from a policy document, and it is where you carry an argument security leadership has usually not heard put this way.

A hard control with no provided alternative buys negative security

The principle: if policy requires a control, the organization must supply the means of complying with it. If policy requires redaction before personal data enters a tenant, supply redaction tooling an analyst can run in five minutes, not an instruction to "remove personal data first" and a hope that eleven teams each invent an approach. If policy requires a sanctioned tenant, that tenant must be at least as good as the alternative people would otherwise use: equivalent capability, sane file limits, access in hours rather than the fortnight of rumor. If policy prohibits personal accounts, provide an approved path for the need those accounts were meeting, because the need does not evaporate when the policy is published.

The argument is worth memorizing in this form: every control that makes the compliant path harder than the non-compliant one buys negative security. Not zero. Negative. It converts a visible, loggable, governable activity into an invisible one while producing a document that lets everyone believe the risk was addressed. The organization ends up less safe and more confident, the worst pairing in risk management.

And it is measurable, which makes this an argument rather than an opinion. The shadow-AI census is the instrument. In the storyline organization it found 71 percent of surveyed staff using unsanctioned AI tools for real work at least weekly; after the governance fast path and the provided capabilities landed, the follow-up census read 21 percent. The policy did not soften and the prohibited cells stayed prohibited. The permitted path became reachable. Fifty points of shadow use moved inside the boundary where it can be logged and improved, and the residual 21 percent is a small, investigable population rather than a majority the organization was pretending not to see.

The capability that paid for itself fastest there was a central redaction service: roughly 12 IT-days to stand up (hosted field-masking and pseudonymization with an upload interface and a log). Four use cases then used it instead of improvising four different approaches to one problem, each with its own gaps and eventual audit finding. That is the arithmetic of provided capability: build the control once as infrastructure, or fund it eleven times as heroism and pay for the variance.

The regulatory frame, kept accurate and non-advisory

Two things are true at once, and holding both is the mark of a competent strategist. First, data-protection law already applies to everything in this lesson, today, independent of AI: personal data in a prompt is personal data, a vendor processing it is a processor, a cross-border transfer is a transfer. None of that waits for an AI-specific regime.

Second, AI-specific obligations arrive on a published calendar. Under the EU AI Act (post-Digital-Omnibus calendar agreed May 2026, pending formal adoption), GPAI (general-purpose AI) obligations have applied since August 2, 2025; transparency for AI-generated content applies from December 2, 2026; high-risk Annex III obligations from December 2, 2027; and Annex I obligations for AI embedded in regulated products from August 2, 2028. Jurisdictions and sectors vary, and applicability depends on facts this lesson cannot know.

Which brings the crucial boundary: you are not counsel and this is not legal advice. Your job is not to interpret whether a use case is high-risk. It is to maintain the inventory and the artifacts these regimes will ask for, and to route interpretation to people qualified to give it. Notice what this chapter's artifacts already are: an inventory of AI systems and owners, a boundary map showing what data goes where under what control, retention decisions with evidence, vendor terms with verification records, decision records showing human accountability. That is most of a compliance file, assembled as a byproduct of running the program well. Chapter 4.5's regulatory-preparation lesson maps obligations to dates and Level 5 turns it into a standing compliance program; this lesson supplies the architecture those programs document, which is why building it now beats reconstructing it under a deadline.

The incident perimeter: which events leave the quality playbook

One short but load-bearing decision remains. Level 3 gave you a quality playbook for wrong output: contain, correct, review, adjust the guardrail. Your organization also has a security and privacy incident process with legal timelines and people whose job this is. During an event is the worst moment to work out which applies, so decide now and publish the routing.

EventRoute
Confidential or personal data reached a prohibited destinationSecurity and privacy incident process (clock starts immediately)
Someone obtained through an AI tool what they could not access directlySecurity incident: unauthorized access, regardless of intent
Injected content caused a system action (send, change, pay, delete)Security incident first, quality review afterward
Data kept past policy, or a vendor retained after a deletion requestPrivacy incident and a control failure to log
An agent or service identity acted outside its permission scopeSecurity incident, plus identity-register review
Output was inaccurate, fabricated, poorly reasoned, or off-toneQuality playbook: contain, correct, adjust the guardrail

One rule handles the ambiguous cases: if data went somewhere it should not, an identity acted beyond its authority, or an action happened without authorization, it is a security or privacy incident even when it also looks like a quality problem. Route it to the security process and let that process decide it was minor. The reverse mistake, treating a data exposure as a quality issue for three days, is how notification deadlines get missed.

What to Do Monday Morning

Five moves, each small enough to start this week and each producing something you can put in front of a steering committee.

  1. Draft the boundary map in ninety minutes. Five classes, four destinations, twenty cells, filled fast and imperfectly with the people who actually know: a privacy owner, a security representative, an IT platform lead, and one operations lead who knows what teams are really doing. Then mark the three cells people ask about most and get those three right before polishing anything else, because that is where your review time goes anyway.
  2. Ask the Friday question out loud. "When someone leaves on Friday, what happens to their AI access?" Ask it about every platform and vendor portal, not just the main tenant, then reconcile user lists against the leaver list. Whatever number comes back is a gap with an owner and a date, closable in about a week.
  3. Put permission-aware retrieval in writing before your next assistant procurement. One paragraph in the standard requirements pack: retrieval enforces the asker's rights at query time, permission changes propagate within a stated interval, and the vendor demonstrates it in a test you design with a low-privilege account. Demonstration, not assurance.
  4. Classify your derived indexes. List every grounding index, embedding store, and cache built or planned, write the source class beside each, then check hosting, access, and deletion against the map. Expect at least one mismatch; that is the point of looking.
  5. Publish the incident routing on one page. Six event types, two destinations, one tie-breaking rule, circulated to AI owners and the security duty team together so both have read it before the day they need it.

That closes the chapter. The Data and Platform Foundation has given you five artifacts that outlive any single use case: the data program charter, the governance service catalog with its fast path, the platform decision records, the vendor risk register, and now the AI Data Boundary Map. Together they are the substrate your eleven-item portfolio runs on, the reason use case twelve costs less to launch than use case three, and the reason the eleventh team's Thursday evening has a permitted answer that takes ninety seconds. Which leaves the part BCG's 10-20-70 rule says holds 70 percent of the value and all of the difficulty: the people. Chapter 4.4 is the change plan.

Key Takeaways

  • Replace per-pilot privacy checking with standing architecture: eleven independent runs of the Level 2 pre-flight produce eleven different answers, and the eleventh team's shortcut becomes the organization's incident, because controls do not average.
  • Reframe AI security away from the model and onto what you can reason about precisely: data movement (what leaves which boundary, in whose custody, retained how long) and identity (who or what acts, under whose authority).
  • Build the AI Data Boundary Map as one page: data classes with your own examples, destinations enumerated honestly including personal accounts, and every cell marked permitted, permitted with control, or prohibited, with the control named.
  • Insist that every AI system act under a named, scoped, revocable identity, route human AI access through the normal joiners-movers-leavers process, and ask the Friday question until the silence turns into a closed gap.
  • Require permission-aware retrieval in writing before procurement and demand a live demonstration with a low-privilege account, because an assistant that ignores the asker's rights is an exfiltration tool with a friendly interface.
  • Classify derived data at the level of its source: indexes, embeddings, chunks, and caches are copies of your documents in another form, and they govern hosting, access, and whether a deletion obligation is actually satisfied.
  • Provide what you require and measure it: any control that makes the compliant path harder than the non-compliant one buys negative security, and the shadow-AI census (71 percent falling to 21 percent in the storyline) settles the argument either way.