Adaeze Lindqvist-Moreau was the Senior Agency Official for Privacy at a federal health agency, and she had completed dozens of Privacy Impact Assessments (PIAs) over her career. A PIA, the analysis federal agencies conduct under the E-Government Act of 2002 to evaluate how a system handles personally identifiable information, was familiar terrain. Then the agency proposed an AI system to predict which Medicaid beneficiaries were at risk of a preventable hospitalization, and Adaeze pulled out her standard PIA template and realized within an hour that it asked all the wrong questions. Her template assumed the system did something fixed and knowable with data. This system learned from data, changed behavior as it learned, made probabilistic inferences about people from information they never directly provided, and could perpetuate patterns no one had intended. The familiar PIA was necessary but nowhere near sufficient. AI demanded a different and harder analysis.

This lesson is about conducting Privacy Impact Assessments for AI systems: when they are required, how AI changes the analysis, what a complete assessment actually asks, and how to produce a PIA that protects people rather than merely documenting a system. The PIA is one of the oldest privacy tools in federal government. Applied carelessly to AI, it becomes a box-checking ritual. Applied well, it is the single best opportunity to catch privacy harms before a system touches a real person.

When a PIA Is Required

Under the E-Government Act, a federal agency must conduct a PIA whenever it develops or procures a system that collects, maintains, or disseminates personally identifiable information about members of the public, or whenever it initiates a new collection of such information. For AI systems the trigger is easily met: training data, input data, and the model's inferences about individuals are all PII or are derived from it. A PIA is also required, and should be revisited, when a system materially changes, and AI systems change as they are retrained, a point Adaeze flagged in her first hour with the proposal.

Almost all government AI systems fall inside that trigger, which means almost all of them should have a PIA. Some agencies nonetheless treat the PIA as something reserved for systems they consider sensitive, skipping it for anything they consider routine. That is wrong, and it is wrong in a predictable direction, because "routine" is a judgment made before the analysis rather than after it. An email filtering system sounds routine. If it analyzes message content, it is processing personal information and deserves an assessment. The working rule is simpler than the exception-hunting: if an AI system processes any personal information, whether names, identifiers, demographics, location, behavior, transactions, or inferences about any of those, a PIA is required.

The PIA also sits alongside, and sometimes overlaps with, other instruments. A System of Records Notice (SORN) is required under the Privacy Act of 1974 when records are retrieved by an individual identifier. An AI impact assessment is required under OMB Memorandum M-24-10 for rights-impacting AI. A well-run agency aligns these so they reinforce rather than duplicate each other, with one data inventory feeding all three rather than three teams interviewing the same system owner. Adaeze's risk-prediction model triggered all three.

Why the Stakes Are Higher in Government

Government agencies hold vast amounts of personal information about the public: tax data, health information, benefit eligibility, criminal history, employment records. When an agency deploys AI, it is usually combining and analyzing that information in ways nobody contemplated when it was collected. Data that seemed safe in one context, such as employment records inside an HR system, becomes something else entirely when combined with other data and used to make predictions about who might leave, who might be a risk, or who deserves scrutiny.

There is a genuine tension here, and a PIA is where it has to be named rather than avoided. Data minimization requires using only the data necessary for a stated purpose. AI development pulls the other way, because more data can improve accuracy and the marginal cost of adding a field is close to zero. Nobody resolves that tension by principle alone. It is resolved case by case, in writing, by someone who has to justify each field against the purpose it serves. That is precisely what the assessment is for, and it is why a PIA written after the data pipeline is built arrives too late to do its job.

How AI Changes the Privacy Analysis

A traditional PIA asks what data you collect, why, how it is stored, who can access it, and how long you keep it. Those questions remain essential. But a traditional system processes what it is given; a database stores what you tell it. AI systems generate. They create new information about people from behavior, characteristics, and other data, and that shift introduces six privacy dynamics a traditional template never had to consider.

Inference: data the system creates about people

The most important shift is that AI generates new sensitive information by inference. A model might infer from browsing history that someone has a health condition, or from a postal code and income that they belong to a particular ethnic group. Adaeze's model would infer a health-risk score about each beneficiary, information the beneficiary never provided and may not want known. That inferred score is itself sensitive PII, even though no one "collected" it. A PIA for AI must treat model outputs as new personal data, with their own collection justification, retention limit, access controls, and disclosure rules. This is the question her old template never asked.

Re-identification: the limits of removing identifiers

Stripping direct identifiers from a dataset is de-identification, and it is worth doing. It is not the same as making people unidentifiable. Re-identification through data linkage and pattern matching is a well-documented risk, and models trained on de-identified data can sometimes support it. The honest framing in a PIA is that removing names and identifiers reduces the risk of re-identification and raises its cost; it does not eliminate it. Treat any claim that a dataset is anonymous as a claim requiring evidence about what other data could be linked to it, not as a conclusion that follows from deleting a column.

Combination: safe pieces, unsafe whole

Data that looks unremarkable in isolation becomes sensitive when joined with other data. An agency may hold several databases that are each reasonably protected on their own terms, and the AI project that draws on all of them at once creates an exposure that none of the individual system assessments ever examined. The PIA has to be scoped to the combination rather than to each source, because the combination is the system, and the privacy risk lives in the join rather than in any of the tables.

Repurposing and secondary use

AI systems are hungry for data, and the temptation is to feed them whatever the agency already holds. But the Privacy Act and the principle of purpose limitation restrict using data collected for one purpose to serve another. The pattern to watch for is data gathered to improve a service being redirected toward screening job applicants, targeting enforcement, or identifying fraud. Adaeze's model would draw on claims data collected to pay providers, not to profile beneficiaries' future health. Using it for prediction was a new purpose requiring its own legal basis and possibly a revised SORN. The PIA had to surface that, not bury it.

Training data persistence and leakage

Personal data used to train a model does not simply disappear after training. It can be memorized by the model and, under some conditions, extracted from it. Once data is inside a deployed model it is hard to know what is in there and hard to remove. The PIA must address what happens to training data, whether the model could leak it, and how the agency would honor a Privacy Act amendment or deletion request when the underlying data has been absorbed into trained parameters. "We deleted the record but the model still encodes it" is a real and underappreciated gap, and it is best answered before deployment with a retraining commitment rather than afterward with an apology.

Disparate privacy impact

AI privacy harms are not evenly distributed. A model performing inference about health risk may be more confident, or more wrong, about some populations than others, and inference errors about vulnerable groups carry heightened harm. The PIA should examine whether privacy risks fall disproportionately on particular communities, an analysis that connects the assessment to the agency's civil rights and equity obligations. This is also where privacy and fairness are most often confused, and they are distinct: a system can distribute its errors evenly and still collect far more about people than it has any business holding.

What a Complete Assessment Asks

A comprehensive PIA for an AI system works through eight areas. The questions look mundane written down. They are not mundane to answer honestly, and the answers are what an oversight reviewer will read.

A Worked AI-PIA Structure

Adaeze rebuilt her template around those areas and added the AI-specific ones her old form lacked. The document her team now completes for every AI system answers, in order:

1. System and purpose: what the AI does, the decision it informs, and the population affected.
2. Data inventory: every data source, its original collection purpose, its legal authority, and whether this use is consistent with that purpose.
3. Inferences produced: what new personal information the model creates, how sensitive it is, and how it is protected, retained, and disclosed, treated as data in its own right.
4. Training data lifecycle: how training data is sourced, secured, and retained, and how deletion and amendment rights are honored against a trained model.
5. Access, retention, and disclosure: who can see inputs and outputs, for how long, and under what sharing agreements.
6. Disparate impact and fairness of privacy risk: whether privacy harms fall unevenly across populations.
7. Individual rights: how beneficiaries are notified, how they can access or correct data, and how they can contest an inference that affects them.
8. Mitigations and residual risk: what controls reduce each identified risk, and which risks the agency knowingly accepts and why.

For the risk-prediction model, this structure caught a specific, concrete problem before launch. The inferred risk scores were going to be retained indefinitely and made visible to case-management staff, with no expiration and no notice to beneficiaries. The PIA forced three changes: a defined retention period for the scores, access limited to staff with a service-delivery need, and a notice to beneficiaries explaining that a risk model informed outreach and how to opt out of predictive outreach. None of those protections existed in the original design. The PIA is what surfaced them.

Privacy-Preserving Techniques and What Each One Actually Buys

Assessment identifies risk; technique reduces it. Seven approaches come up repeatedly, and each is worth understanding for its limits as much as its benefits. Data minimization means collecting and using only what the purpose requires, and removing unnecessary variables before training. It is the least glamorous technique and usually the most effective, because a field you never collected cannot leak, cannot be repurposed, and cannot be subpoenaed.

Anonymization means removing identifying information, and it needs the caveat attached every time it is named: some anonymization can be reversed through data linkage. Differential privacy adds carefully calculated noise to data or to model outputs, which makes it harder to infer information about any specific individual while preserving usefulness in aggregate. Both are real protections. Neither is a switch that moves a dataset from identifiable to safe, and the strength of differential privacy depends on parameter choices that trade utility against protection, which is a decision the PIA should record rather than leave to whoever configured the library.

Federated learning trains models across decentralized data without moving personal records to a central location, which reduces the concentration of risk. Homomorphic encryption allows analysis to run on encrypted data without decrypting it. Purpose limitation is a policy control rather than a technical one: use data only for the stated purpose, and do not train a model for a new purpose on data gathered for an old one. Retention limits mean deleting data that is no longer necessary rather than keeping it indefinitely in case it becomes useful, and for AI they have to extend to the model, not just the source table.

Worked Example: A Benefit Eligibility System

An agency implements an AI system to help determine eligibility for unemployment benefits. The proposal arrives with five data categories and a justification for each. Employment history and income history are necessary on their face. Criminal history is argued for because past fraud is relevant. Credit reports are argued for as evidence of financial need. Healthcare records are argued for on the theory that health status affects employment prospects. Every one of those arguments was made in good faith by someone who wanted the model to work well.

The privacy analysis takes them one at a time against the stated purpose. Employment and income data are necessary. Criminal history is relevant, but collecting the full history when only fraud convictions bear on the purpose is broader than necessary. Credit reports are tangential at best, because benefit eligibility should not depend on creditworthiness. Healthcare records are not necessary, because health status does not determine eligibility for the benefit. The recommendations follow directly: keep employment and income, narrow criminal history to relevant fraud convictions, and discontinue collection of credit reports and health records entirely.

The result is a system that holds substantially less sensitive data about applicants, which reduces breach exposure, narrows the population of staff who need access, and removes two categories of information the agency had no purpose-based right to hold. What that outcome does not tell you is whether model accuracy changed. Removing fields can affect performance, and the honest way to close a data minimization decision is to measure the model with and without the disputed inputs and record the result in the PIA, rather than asserting that privacy was improved at no cost.

Worked Example: A Law Enforcement System

An agency is deploying predictive policing AI. The PIA examines what is in the model: crime reports, which are legitimate inputs; 911 call locations, also legitimate; neighborhood demographics, which raise immediate fairness concerns; prior arrests by neighborhood, which raise a deeper concern because arrests reflect policing patterns as much as they reflect crime; and social media data about identifiable individuals, which is personal expression rather than evidence of an offense.

The analysis is uncomfortable and has to be written down anyway. The system infers which individuals and which neighborhoods are risky. Those inferences are not necessarily grounded in crimes; they are grounded in demographics and arrest patterns. A model trained on arrest data learns where police have historically gone, and its predictions send police back there, which generates the arrests that confirm the prediction. The system also processes extensive personal information and makes inferences about specific people, which places it squarely in the inference and disparate-impact categories described above.

The recommendations are correspondingly firm. Use crime and 911 data and remove demographic proxies. Use victim crime reports rather than arrest data. Remove social media data entirely. Implement fairness monitoring to track whether the system concentrates attention on particular neighborhoods, understanding that monitoring produces evidence over time rather than assurance on day one. Be transparent about how the system works. Establish an appeal process for someone who believes they are being unjustly targeted. The redesign narrows the privacy intrusion and the fairness exposure substantially; whether it preserves any crime prevention benefit is an empirical question the agency should answer with evaluation rather than assume in the PIA.

Making the PIA Protective, Not Performative

A PIA written after a system is built, to satisfy a checklist, protects no one. Adaeze's rule was that the PIA began at design, not at deployment, so its findings could still change the system. She also insisted the PIA name a residual-risk owner and a re-assessment trigger: specifically, that it be revisited each time the model was retrained, because retraining can change what the system infers and from what data. A PIA that is current at launch and stale forever after is an accurate description of a system that no longer exists.

The second discipline is transparency toward the people in the data. Notice is not a courtesy appended at the end; it is one of the few controls that lets an individual exercise any of the other rights, because nobody contests an inference they were never told about. People should know what data is collected about them and how it is used, and for an AI system that means saying plainly that a model informs the decision or the outreach they are receiving. Adaeze's beneficiary notice cost almost nothing to write and was the only reason an opt-out could exist at all.

The third discipline is treating recommendations as requirements. A finding that leadership chooses not to implement should be overridden explicitly, in writing, with a documented justification and a named official, rather than quietly dropped between the assessment and the deployment. That is the difference between a governance decision and an evaporation. Track implementation status for each recommendation the way you would track any other commitment, and expect someone to ask for the evidence.

Anti-Patterns

• PIA without action. The assessment is completed, serious concerns are identified, and none of the recommendations are implemented. The document becomes a compliance artifact with no effect on system behavior. Treat PIA recommendations as requirements unless explicitly overridden by leadership with a documented justification, and track implementation status to closure.
• Privacy theater. The PIA is conducted and filed, and no meaningful protections change. It is the same failure one step earlier: the process ran, the risk did not move. Require evidence that each recommendation was implemented, not a signature confirming the assessment was performed.
• "We removed the names, so it is anonymous." Removing direct identifiers is de-identification. It reduces re-identification risk and raises its cost; it does not make individuals unidentifiable, because linkage against other datasets can undo it. Never record in a PIA that a dataset carries no privacy risk because identifiers were stripped, and never describe a technique such as differential privacy as ensuring individual records cannot be identified. State what the control reduces and what it leaves open.
• Conflating privacy and fairness. Treating a privacy assessment as a fairness assessment, or the reverse. They are related and distinct. A system can be fair and still violate privacy, and it can be privacy-minimal and still distribute its errors unjustly. Conduct both, because they answer different questions and neither substitutes for the other.
• Ignoring inference risk. Focusing only on explicit collection while ignoring what the model derives from what it was given. Document the inferences the model produces, and test whether it is inferring sensitive attributes that were never in the input data.
• Skipping the routine system. Deciding a system is too ordinary to warrant assessment, before the assessment that would have revealed what it processes. If it touches personal information, it gets a PIA, and the judgment about sensitivity is an output of the analysis rather than a precondition for it.
• The PIA that never expires. Assessing once at launch and never again, while the model is retrained on new data and its inferences shift. Name a re-assessment trigger, make retraining one of them, and treat a materially changed system as a system requiring a fresh look.

Practice Prompts

• Map the data. For one AI system in your area, list every category of personal data it touches, including data that arrives through a joined table rather than a direct collection. For each category, write one sentence justifying it against the system's stated purpose. Which entries were hard to justify, and what would break if you removed them?
• Outline the PIA. Draft the section outline for a PIA for that system. Which of the eight assessment areas would be quick to answer, and which would require you to go and ask someone? The areas requiring a phone call are usually where the unexamined risk is.
• Analyze the inferences. What does your system infer that nobody explicitly provided? Could any of those inferences amount to a sensitive attribute, such as health status, family circumstance, or group membership? How would you detect that it was happening, and who would you tell?
• Redesign for privacy. If you were rebuilding the system to be more privacy-protective, what would you change? Which of the seven techniques would apply, what would each one cost in accuracy or engineering effort, and which change would you make first?
• Assess individual rights. Document what privacy rights individuals actually have with respect to your system. Can they find out it exists? See what it holds? Request correction or deletion? Contest an inference that affected a decision about them? Where the answer is no, note whether that is a legal position or simply an unbuilt feature.

Reflection

Take three minutes to assess privacy in your own context. What personal data do your organization's AI systems process today? Is all of it necessary for the purpose each system serves, and could any category be removed without changing what the system is for? What inferences do those systems make, and are any of them inferences about sensitive attributes that nobody set out to collect?

Then the harder questions. Do the people in your data know it is being used this way? Can they access it, correct it, or ask for deletion, and would a front-line staff member know how to help them do so? And if you had to explain your system's privacy posture to a civil liberties advocate rather than to an auditor, how would that conversation go? The places where you would want to change the subject are the places to start.

Glossary

• Privacy Impact Assessment (PIA): formal documentation of what personal information a system uses, how it is protected, and what privacy risks exist.
• Personally identifiable information (PII): information that identifies or can be linked to a specific individual.
• System of Records Notice (SORN): the notice required under the Privacy Act of 1974 when an agency maintains records retrieved by an individual identifier.
• Inference: information about a person derived from data analysis that the person never explicitly provided.
• Re-identification: determining the identity of individuals in a de-identified dataset through data linkage or pattern matching.
• De-identification: removing direct identifiers from records, which reduces but does not eliminate the risk of re-identification.
• Differential privacy: a technique that adds calibrated noise to data or outputs to make inference about any specific individual harder, at some cost to utility.
• Federated learning: training a model across decentralized data sources without moving personal records to a central location.
• Homomorphic encryption: performing computation on encrypted data without decrypting it.
• Data minimization: collecting and using only the data necessary for a stated purpose.
• Purpose limitation: the principle that data collected for one purpose may not be freely used for another.
• Residual risk: the risk that remains after mitigations, which a named official knowingly accepts and documents.

Related Lessons

• PII and AI: The Bright Red Lines covers the handling rules for personal information that a PIA assumes are already understood, and the specific things never to put into a general-purpose AI tool.
• Minimum Risk Management Practices situates the PIA among the documentation and transparency practices required for higher-risk AI, of which this is one instantiation.
• Data Governance for AI establishes the standards for data quality, provenance, and appropriate use that make the PIA's data inventory answerable rather than speculative.
• Risk Classification: Safety-Impacting vs. Rights-Impacting determines whether a system also needs the AI impact assessment required under M-24-10 alongside its PIA.
• NIST AI RMF: The GOVERN Function covers the governance body that receives PIA findings and the decision rights that let a privacy objection stop a deployment.
• Data Sensitivity and Classification provides the vocabulary for describing how sensitive each data category in your inventory actually is.

Closing

PIAs sit at the intersection of privacy law, which governs federal information systems, civil rights law, which prohibits discrimination, and AI governance, which requires thoughtful risk management. That intersection is why the AI-era PIA cannot be a form. The questions that matter most, what the model creates about people, whether the data was collected for this, what happens to a deletion request once the data has shaped a model's parameters, and who bears the residual risk, are questions of judgment that a template can prompt but cannot answer.

Adaeze's model shipped, with a retention limit on the risk scores, access confined to staff with a service-delivery need, a notice to beneficiaries, and an opt-out from predictive outreach. None of that was in the original design, and none of it was discovered by a technical review. It came from someone asking, early enough to matter, what this system would create about people and who would be able to see it. That is the whole discipline. The document is just where the answers get written down.

Key Takeaways

• A PIA is required when a system handles the public's PII, and almost every government AI system does through its training data, inputs, and inferences. It should be re-assessed whenever the system materially changes, including on retraining.
• Do not decide a system is too routine to assess. Sensitivity is a conclusion of the analysis, not a precondition for doing it. If it processes personal information in any form, it gets an assessment.
• Treat the model's inferences as new personal data. A risk score or profile the system generates is sensitive PII with its own retention, access, and disclosure rules, even though nobody collected it.
• De-identification is not anonymity. Removing identifiers reduces re-identification risk and raises its cost. Linkage against other data can undo it, and no PIA should claim otherwise.
• Watch for repurposing and combination. Data gathered for one program cannot be freely used to train a model for another, and datasets that are each safe alone can create real exposure when joined.
• Address training-data persistence. Models can memorize and leak training data, which complicates deletion and amendment rights. Plan how those rights are honored against a trained model before deployment, not after a request arrives.
• Examine disparate privacy impact. Privacy harms fall unevenly, and inference errors about vulnerable groups carry heightened harm. Connect the PIA to the agency's equity and civil rights obligations, while keeping privacy and fairness as separate assessments.
• Minimization is the strongest technique available. Differential privacy, federated learning, and encryption all help, each with limits worth recording. A field you never collected has no limits to record.
• Recommendations are requirements until someone overrides them in writing. An unimplemented finding that quietly disappears is the difference between an assessment and a ritual.
• Start the PIA at design and keep it alive. One written after deployment protects no one; one that begins at design and is revisited on retraining changes the system for the better.

Frequently Asked Questions

Our vendor operates the AI system and holds the data. Do we still need a PIA? Yes. The E-Government Act trigger turns on the agency developing or procuring a system that handles the public's personal information, and procurement is explicitly included. A vendor arrangement changes who answers some of the questions, not whether they must be answered, and it adds an entire section: what the third-party sharing agreement permits, what safeguards apply to the data in the vendor's hands, and what happens to the model and the data at the end of the contract.

Does a PIA replace the SORN, or the M-24-10 impact assessment? No. They are distinct instruments with distinct triggers. The SORN is required under the Privacy Act when records are retrieved by an individual identifier. The AI impact assessment is required under M-24-10 for rights-impacting AI. The PIA is required under the E-Government Act for systems handling the public's PII. A system can trigger all three, as Adaeze's did. What a well-run agency does is align them so a single data inventory and a single set of interviews feed all three documents.

Someone requests deletion of their record, but the model was trained on it. What do we do? Answer this before it happens, in the PIA. Deleting the source record does not remove whatever the model absorbed during training, so the practical options are commitments made in advance: a retraining schedule that periodically rebuilds the model from the current, post-deletion dataset, limits on how long training snapshots are retained, and a documented position on what the agency can and cannot undo. Discovering the question after a request arrives leaves you choosing between an inaccurate answer and an expensive one.

We de-identified the training data. Can the PIA say there is no privacy risk? No, and that specific sentence is one of the most common defects in AI privacy documentation. De-identification reduces risk; linkage against other datasets can reverse it, and models trained on de-identified data can still support inference about individuals. Write what is true: which identifiers were removed, what linkage risk remains given what else exists, and what controls address it. A PIA claiming zero residual risk tells a reviewer that the analysis was not done.

When exactly should the PIA start? At design, while the data pipeline is still a diagram. That is the only point at which a finding can change a data source rather than merely annotate one. Adaeze's three protections, a retention limit, narrowed access, and a beneficiary notice, were all cheap to add at design and would each have been a change request afterward. A PIA whose first draft appears alongside the deployment checklist has been scheduled to fail.

Who should own the residual risk? A named official with the authority to accept it, not a team and not a role that exists only on an org chart. The point of naming an owner is that a later reviewer, an Inspector General, or a successor can identify who made the judgment and on what evidence. Pair the name with the date the acceptance gets revisited, because a residual risk accepted three retrainings ago was accepted about a different system.