Priya Nadkarni, a program director at a state department of labor, had a winning idea: an AI model to flag unemployment claims likely to involve errors, so caseworkers could review them faster. The pilot results were strong. Then the agency's privacy officer asked one question in a planning meeting: "What's the Privacy Impact Assessment, and which of the eleven years of claimant data is this model actually touching?" Priya did not have an answer. The project stalled for four months while the team retrofitted privacy controls that should have been designed in from day one.

That four-month delay is the price of treating privacy as a compliance checkbox at the end instead of an engineering requirement at the start. Privacy engineering means building privacy protections into an AI system's design, data flows and operations, the same way you build in security or accuracy. For government, where you handle citizen data under legal mandates and cannot simply ask people to opt out, getting this right is not optional. This lesson gives you the legal frame, the techniques, and a working artifact you can lead with.

Why AI Raises the Privacy Stakes

Government AI systems routinely handle sensitive personal information: health data, financial information, benefit histories, and in some agencies national security information. Privacy protection here is not a preference. It is legally mandated and ethically essential, and it is also the foundation of public trust, because citizens have to believe the government protects their information responsibly before they will give it accurately in the first place.

AI changes the risk profile rather than simply inheriting it. A traditional database holds what you put in it. A model amplifies privacy risk in three specific ways: it makes personal information queryable, because outputs can be probed; it makes it inferrable, because a model can derive sensitive attributes that were never supplied; and it makes it compromisable in new ways, because training data, model weights and outputs are all now assets an adversary can target. Those are three distinct attack surfaces, and a control designed for one does not cover the others.

The consequence is a sequencing rule that this lesson keeps returning to. Privacy must be engineered from the beginning, not bolted on afterward. A system that was not designed for privacy generally cannot be made private without fundamental redesign, because the data flows, the retention behaviour and the access model are already set in the architecture. Priya's four months were not spent writing a document. They were spent unpicking design decisions that were cheap in week one and expensive in month six.

The Legal Floor You Are Building On

A few obligations shape almost every government AI project that touches personal data. You do not need to be a lawyer, but as a strategist you must know what they require and when they trigger, and you must know which office in your agency owns each of them.

• The Privacy Act of 1974 governs how federal agencies handle personal information about citizens. Its key principles are practical: collect only necessary data, use it for the intended purpose only, protect it from unauthorized disclosure, and provide transparency to citizens. Its core ideas reach far beyond federal use. An AI model that quietly repurposes data collected for one program into a new predictive use runs directly at the purpose limitation this statute is built on.
• FISMA, the Federal Information Security Modernization Act, requires federal agencies to implement information security protections adequate to the risk level of the data handled. Note the boundary carefully: FISMA does not specifically address privacy. It requires protecting confidentiality as one part of confidentiality, integrity and availability. Security controls are necessary for privacy and are not the same thing as privacy.
• Office of Management and Budget guidance issued in 2024, memorandum M-24-10, requires a Privacy Impact Assessment for any AI system that uses personal information. Treat that as the trigger question for your project: if personal information is anywhere in the pipeline, the assessment is in scope.
• Sector-specific regimes add requirements on top. If your agency handles health data or education data, statutes such as HIPAA and FERPA bring their own obligations, and they apply to the AI use as much as to the database it draws from.

The Privacy Impact Assessment, or PIA, is the document that ties these together. It is a structured analysis of what personal data a system uses, why, where it flows, who can see it, and what could go wrong. Priya's project did not fail a law. It failed to do the analysis that the law assumes you have already done, which is why the privacy officer's question was so hard to answer and so fair to ask.

Data Minimization: The Cheapest Privacy Control You Have

The most powerful privacy move is also the simplest: do not collect or feed the model data it does not need. Data you never ingest cannot leak, cannot be exposed in a breach you did not anticipate, and cannot bias the model on an attribute it should never have seen. Minimization also shrinks your compliance burden, because every element you hold is an element you must justify, secure, retain correctly and eventually dispose of.

Priya's first model design fed in the entire claimant record: name, Social Security number, address, full work history, and demographic fields. When the team applied data minimization, they asked one question per field: does the model's accuracy actually depend on this? Name, Social Security number and address came out entirely, because the model predicts error likelihood from claim patterns rather than identity, and caseworkers re-attach identity from the case system after a claim is flagged. Demographic fields such as race and gender came out too, both because they were not predictive and because including them risked encoding bias into who gets extra scrutiny. Claim and employment patterns stayed, because they genuinely drove the prediction.

The minimized model performed within one percentage point of the original and carried a fraction of the privacy risk. That trade is almost always worth taking. In practice, minimization is four habits rather than one decision: limit what you collect to what the decision actually requires; use aggregate statistics instead of individual-level records wherever the purpose allows; purge data regularly once it is no longer needed; and do not retain training data longer than the purpose and your records schedule require. The last one is the most commonly skipped, because training sets feel like assets rather than liabilities.

Every field you feed a model is a field you must protect, justify and defend in an audit, a Freedom of Information Act response or a breach notification. The safest data is the data you chose not to collect.

Aggregation and De-identification, Honestly Described

Where individual identities are not needed for the purpose, use aggregated or de-identified data. The techniques are familiar. Remove or separate identifying information from the data used for analysis. Use aggregate statistics such as average income instead of individual values. Generalize precision, using income bands instead of exact amounts. De-identify before training when identities are not needed downstream.

Be precise about what this buys you. De-identification and generalization reduce re-identification risk. They do not eliminate it. Records that have had direct identifiers stripped can often be re-identified by combining the remaining fields with other available data, and a small number of generalized attributes can be enough to single out a person in a sparse population. Treat de-identified data as lower risk, not as no risk, and keep the access controls, agreements and monitoring that its actual sensitivity warrants. A dataset described as anonymized in a project document is still your responsibility if it can be re-linked.

This matters most at the moment of release. The safest-looking dataset in your agency is usually the one someone is about to share with a partner or publish, precisely because it has been through a de-identification step that everyone now treats as final. Ask what other data an outsider could join to it, and ask it before the file leaves the building rather than after.

Differential Privacy in Plain Terms

Sometimes you need to publish statistics or share a dataset, and even aggregated numbers can leak information about individuals. Differential privacy adds carefully calibrated random noise to data or to the algorithm so that the presence or absence of any single person's record does not measurably change what gets published. The underlying test is intuitive: if someone queries the system repeatedly across overlapping datasets, can they tell whether one particular record was in there? If the calibrated noise makes that determination unreliable, the system provides differential privacy.

The intuition for a public agency is easy to picture. Imagine you publish the average benefit amount by ZIP code. In a ZIP code with only three claimants, that average can expose individuals. Differential privacy injects a small, controlled amount of noise so the published figure is still useful for planning but no longer reverse-engineers any one person's data. The United States Census Bureau adopted this approach for the 2020 census for exactly this reason.

The benefit is that the protection is mathematical rather than a matter of judgment: you can state and defend a bound on how much any one person's data can influence the output. Describe it that way rather than as a blanket guarantee of privacy. The bound holds at the privacy parameter you choose and for the queries the mechanism actually covers; weaker settings and repeated release across many outputs erode the protection, and nothing here prevents a correct population-level inference about a group that a person belongs to. The trade-off is accuracy, since noise is noise, so the technique fits situations where you need a strong, statable privacy property and can tolerate a slight loss of precision.

You will rarely implement the mathematics yourself. Your job as a strategist is to recognise when it is needed: whenever you release aggregates, share datasets with partners, or expose model outputs that could be probed to reconstruct individual records. Flag those moments and bring in the technical expertise, and make sure whoever implements it writes down the parameter chosen and why.

Federated Learning, Encryption and Access

Two more techniques belong in a government privacy engineer's vocabulary because they change what has to move rather than what has to be masked.

Federated learning trains models on distributed data instead of centralising it. Suppose agency A and agency B each hold relevant data. Rather than combining the data into one repository, each trains locally and the results are combined. Personal data never leaves either agency, which sidesteps a whole class of problems around sharing agreements, joint custody and the expanded breach surface a merged dataset creates. It is not a privacy panacea, since what is shared can still carry information about the underlying records, but as an architectural choice it removes the single most dangerous artefact: the combined pile.

Encryption protects data from unauthorized access, and it needs to be specified rather than asserted. Encrypt data in transit using TLS. Encrypt data at rest, whether by full-disk or database encryption. Then answer the question that actually determines whether any of it works: who holds the keys, and how are they protected? Key management is where encryption programs succeed or quietly fail. Finally, consider search capability early, because the ability to query encrypted data efficiently shapes the architecture, and retrofitting it is the kind of fundamental redesign this lesson keeps warning about.

A Privacy Impact Assessment Template for AI Systems

A standard PIA answers a familiar set of questions: what personal data will the system handle, how will it be used, who will have access, how will you protect it, what are the privacy risks, how will citizens be notified, and what are their rights of access, correction and deletion. Those questions remain necessary. They were written for traditional databases, and for AI they are no longer sufficient.

The AI-specific additions are the ones that catch teams out. Can the system infer sensitive attributes that were never explicitly provided? Could the system be queried to extract information about individuals? Does training on historical data perpetuate past discrimination that is itself recorded in the data? And what is the privacy impact of transparency and explainability, given that a detailed explanation of a decision can reveal more about the underlying record than the decision itself does? Use the template below as a working artifact. Run it at design time, and revisit it whenever the data or the use changes.

1. Purpose and authority. What decision does this AI support? Under what legal authority was the data originally collected, and does this AI use fall within that authority? If not, stop.
2. Data inventory. List every data element the model ingests. For each, record source, sensitivity, and a one-line justification of why the model needs it. Any field without a justification gets removed.
3. Data flow map. Trace the data from collection through training, inference, storage and output. Note every system and person who can access it at each stage.
4. Minimization check. Confirm direct identifiers and non-predictive sensitive attributes have been removed or separated. Document what was dropped and why.
5. Inference risk. Assess whether the system can derive sensitive attributes that were never supplied, and whether that inference is itself a use the authority covers.
6. Memorization and leakage risk. Can the model reproduce individual training records in its outputs? Can outputs be probed to reconstruct personal data? Note mitigations such as output filtering or differential privacy.
7. Access and retention. Who can query the model, see its outputs and access the training data? How long is each retained, under which schedule, and what is the deletion plan?
8. Individual rights. How can a person find out the system used their data, see the relevant record, and request correction? An AI that makes this impossible is a Privacy Act problem.
9. Security categorization. Under FISMA, what is the impact level of this system, and do its controls match? Confirm with your security officer.
10. Residual risk and sign-off. State the privacy risks that remain after mitigation, and name the official who accepts them. Privacy risk is owned, not erased.

The process around the document matters as much as the document. A credible PIA involves stakeholder consultation, including the civil rights office, the privacy office and, where the system affects them directly, the communities on the receiving end. It carries a risk assessment and named mitigation strategies rather than a list of concerns. And it is reviewed at least annually, because a PIA describes a system as it was on the day it was written, and models, data sources and uses all move.

Data Governance, Citizen Rights and Transparency

Compliance becomes real through ordinary operational machinery. Data governance for an AI system means an inventory of all personal data in use, a stated purpose for each data element, a retention schedule that says when it is deleted, access controls that say who can reach what, and audit logging that tracks all data access. If you cannot produce those five artefacts on request, you do not yet know what your system holds, and neither does anyone who would have to answer for it.

Citizen rights need implementation, not just acknowledgement. Citizens can request a copy of their data, request correction of inaccurate data, and request deletion where retention requirements permit it. That last qualification is load-bearing in government, because records schedules and statutory retention obligations frequently prevent deletion, and telling someone their data will be erased when it legally cannot be is worse than explaining the constraint. What matters operationally is that you have written procedures for responding to each type of request, and that someone owns them.

Transparency has three components in the public sector. A privacy notice informs citizens what data is collected and why. Explanation helps them understand how their data is used, in language that a person affected by the decision can act on. And the Freedom of Information Act makes government data accessible to the public with appropriate redactions, which means your AI system's records may be read by people you did not anticipate. Design the records with that in mind. Finally, workforce training closes the loop: all staff handling personal data need training on privacy requirements, refreshed when requirements change, with clear consequences for mishandling personal information.

Leading the Work, Not Just Documenting It

The difference between Priya's stalled pilot and a smooth launch was not more paperwork. It was sequence. Privacy engineering done at design time costs days. Done as a retrofit, it costs months and erodes trust with the privacy and security colleagues whose sign-off you will need again on the next project. Bring them into the first design meeting, run the data inventory before you train anything, and treat the PIA as a living design document rather than a launch-day formality.

Your role as a strategist is to make privacy a property of the system rather than a gate the system has to squeeze through later. That means asking the minimization question in the room where the data schema is decided, insisting that de-identification claims are stated in terms of reduced risk, and making sure the person who accepts the residual risk knows they are accepting it. None of that requires you to be a cryptographer. It requires you to ask the questions early enough that the answers can still change the design.

Anti-Patterns

• Privacy designed after the system is built. It happens because privacy is less visible than functionality, and the result is a system with no privacy protections that needs expensive redesign. Avoid it by setting privacy requirements on day one of system design.
• Collecting everything just in case. More data feels safer. It is not: sensitive data increases breach impact and creates transparency and consent problems. Collect only necessary data and justify each element individually.
• Skipping the PIA because it feels bureaucratic. Privacy risks that were never identified surface after deployment, as citizen complaints and oversight findings. Run a rigorous PIA with stakeholder input and update it annually.
• Leaving citizens unaware their data is being used. Agencies avoid notification because of the overhead and the fear of a negative reaction. Citizens then learn from the press, feel violated, and trust is damaged. Notify proactively about data usage and about the rights people hold.
• Calling de-identified data anonymous and treating it as risk-free. Stripping direct identifiers reduces re-identification risk; it does not eliminate it, and remaining fields can often be linked to outside data. Keep the controls proportionate to what re-identification would expose.
• Describing differential privacy as a guarantee that nobody can learn anything. It provides a statable bound at a chosen parameter, for the queries the mechanism covers. Repeated release and weak parameters erode it, and it does not prevent correct inferences about groups.
• Treating security controls as privacy compliance. FISMA-grade encryption and access control protect confidentiality. They say nothing about whether you should have collected the data, whether the use matches the authority, or whether people can see and correct their records.
• Retaining training data indefinitely. A training set is a standing collection of personal data with all the obligations that implies. Give it a retention schedule like any other holding.

Practice Prompts

1. Identify privacy risks for your system. What personal data will it handle? Could it infer additional private information? Could it be queried to extract information about individuals? What are the data protection and breach risks? For each, write the risk, the impact and the mitigation.
2. Design a minimization approach. Separate what you absolutely need from what would merely be nice to have. For each element that survives, record a retention period and access controls, and describe how you will purge what is no longer needed.
3. Draft the PIA. Work through data inventory, uses, risks, mitigations, citizen rights of access, correction and deletion, and how citizens will be notified. Note every question you cannot answer yet; that list is your real project plan.
4. Evaluate the privacy-preserving techniques. Could minimization alone reduce the risk enough? Could aggregation replace individual-level data? Would differential privacy be appropriate? Is federated learning feasible? For each, assess feasibility, privacy benefit and accuracy trade-off.
5. Build the compliance procedures. Specify data governance including inventory, access control and audit logging; how citizens exercise their rights; what privacy notice they receive; and the content, frequency and tracking of workforce training.
6. Stress-test a de-identification claim. Take a dataset your team calls anonymized and list what an outsider could join it to. Decide whether the claim survives that exercise, and rewrite it in terms of reduced risk if it does not.

Reflection

Think about the AI system closest to you right now. If a privacy officer walked in today and asked Priya's question, which of the years and fields of personal data is this model actually touching, how long would it take you to answer with evidence rather than recollection? The gap between those two answers is a fair measure of how much of your privacy work is engineering and how much is documentation written after the fact.

Then consider the harder one. Which data element in your system would be hardest to justify if you had to defend it line by line in front of the people it describes? Teams usually know the answer immediately, and it is usually a field that was included because it was available rather than because the model needed it. Removing it before launch costs an afternoon. Removing it after a breach notification costs considerably more.

Glossary

• Privacy engineering: building privacy protections into a system's architecture, data flows and operations from the beginning rather than adding them later.
• Data minimization: collecting and using only the data necessary for the system's purpose.
• Privacy Impact Assessment (PIA): a systematic evaluation of a system's privacy risks and the mitigations applied to them.
• Differential privacy: adding calibrated noise to data or algorithms so that any single individual's presence in the dataset does not measurably change published outputs, within a chosen privacy parameter.
• Federated learning: training models on distributed data held by separate parties without centralising the data.
• De-identification: removing or separating identifying information from data, which reduces but does not eliminate the risk of re-identification.
• Generalization: reducing the precision of a value, such as reporting an income band instead of an exact figure, to make individuals harder to single out.
• Purpose limitation: the principle that data collected for one stated purpose should be used only for that purpose.
• Retention schedule: the documented rule that determines how long each category of data is kept and when it is disposed of.
• Audit logging: a record of who accessed which data and when, kept so that access can be reviewed after the fact.

Related Lessons

• Privacy Impact Assessments for AI Systems goes deeper into the assessment itself and how agencies review and approve one.
• Privacy Policy Engineering covers turning privacy policy into enforceable system behaviour.
• Data Governance for AI builds out the inventory, retention and access machinery this lesson depends on.
• Enterprise AI Risk Management places privacy risk alongside the other risks your agency has to manage together.
• AI Red-Teaming Fundamentals is where you learn to probe a model for the leakage and extraction risks named here.
• Bias Detection and Mitigation at Scale addresses the discrimination that historical training data can carry.
• Rights-Impacting and Safety-Impacting AI Safeguards explains which uses attract additional required practices.
• Transparency: Citizens' Right to Know extends the notice and explanation obligations into practice.

Closing

Privacy engineering is not a document, a sign-off or an office you route things past. It is a set of design decisions about what data enters the system, where it travels, who can reach it, how long it stays, and what a person can find out and correct about themselves. Every one of those decisions is cheap while the architecture is still on a whiteboard and expensive once it is running in production against eleven years of claimant records.

Priya's project shipped in the end, with a smaller data footprint, a documented assessment and a privacy officer who had been in the room from the start. That is the realistic goal. Not a system with zero privacy risk, which does not exist, but a system whose remaining risk is known, proportionate to the purpose, accepted by a named official and visible to the people it affects. Engineer it from the beginning, and the compliance takes care of itself. Bolt it on, and you will pay for it twice.

Key Takeaways

• Privacy is a legal requirement, not a nice-to-have. FISMA, the Privacy Act of 1974, OMB guidance issued in 2024 and agency-specific regimes such as HIPAA and FERPA all mandate protection, and non-compliance creates liability.
• Engineer privacy from the beginning. A system not designed for privacy generally cannot be made private without fundamental redesign, and retrofitting is both expensive and often insufficient.
• AI amplifies risk in three specific ways. It makes personal information queryable, inferrable and compromisable, and a control for one of those does not cover the others.
• Minimization is the first principle. Data you never ingest cannot leak or bias the model, and removing identifiers and non-predictive fields often costs very little accuracy.
• De-identification reduces risk; it does not erase it. Records stripped of direct identifiers can often be re-linked using the fields that remain, so keep controls proportionate.
• Differential privacy provides a statable bound, not a blanket promise. It holds at the parameter you choose and for the queries covered, and it trades a little accuracy for that property.
• Use an AI-specific PIA and review it annually. Standard assessments miss inference, training-data provenance, memorization and output reconstruction, and a PIA describes the system only as it was on the day it was written.
• Transparency and citizen rights build trust. Notice, explanation, access, correction and deletion where retention rules permit are what make the protection real to the person on the other end.
• Own the residual risk. Privacy risk is never fully erased, so name the official who accepts what remains and keep the assessment alive as the system changes.

Frequently Asked Questions

Does every AI project need a Privacy Impact Assessment? The trigger stated in federal guidance is the use of personal information: OMB memorandum M-24-10 requires a PIA for any AI system that uses personal information. In practice, the question worth asking early is whether personal data appears anywhere in the pipeline, including training data, prompts, logs and outputs, because teams often remember the database and forget the logs. Confirm scope with your privacy office rather than deciding it inside the project team.

If we anonymize the training data, are we outside privacy requirements? Not automatically, and you should be careful about the word. De-identification reduces re-identification risk rather than eliminating it, and whether a dataset is genuinely outside scope is a determination your privacy office makes on the specific data, not a conclusion a project team reaches because it ran a script. Document exactly what was removed, what remains and what external data could be joined to it.

Can we satisfy privacy requirements with strong security controls? No. FISMA requires security protections adequate to the risk of the data handled, and it does not specifically address privacy; it requires protecting confidentiality as one part of the security triad. Encryption and access control do not answer whether you should have collected the data, whether the use matches the original authority, or whether people can see and correct their records.

Where does differential privacy actually fit in a government workflow? Wherever you release something derived from individual records: published aggregate statistics, datasets shared with partners, and model outputs that could be probed to reconstruct individuals. The Census Bureau's use for the 2020 census is the canonical public-sector example. You will not implement the mathematics yourself, but you should be the person who spots the release moment and calls in the expertise.

Citizens are asking us to delete their data. Must we? The right to request deletion exists, and in government it is bounded by retention requirements. Records schedules and statutory obligations often require you to keep material you would otherwise remove. The correct response is a documented procedure that handles the request, applies the retention rules, and explains the constraint to the person clearly rather than promising an erasure you cannot perform.

Is federated learning a way to avoid data sharing agreements? It is a way to avoid centralising the data, which removes the most dangerous artefact and a great deal of risk. It is not a blanket exemption from your legal obligations, since what is exchanged between parties can still carry information derived from personal records. Treat it as an architecture that reduces exposure, and involve the privacy and legal offices in confirming what the exchange actually contains.