AI Policy Impact Assessment
Catalina Vargas had been Deputy Secretary for Policy at a large state health and human services agency for three years when the new AI fairness policy landed on her desk. The policy, drafted by a federal advisory body, required that all AI systems used in benefit eligibility determinations undergo a bias impact assessment before deployment. On paper it was a straightforward requirement. In practice, Catalina had fourteen legacy systems touching eligibility workflows, a technology team of twelve, an annual IT budget of $8.4 million, a Medicaid enrollment workload of 2.1 million beneficiaries, and exactly zero staff with prior experience running an AI bias impact assessment. The policy was well intentioned. Its authors had not modeled what implementing it would cost, who would bear that cost, or what would happen to agencies like hers that lacked the internal expertise to comply without outside help. This lesson is about how to model those things before a policy goes final, so that a well-intentioned requirement does not become an operational disaster.
Why Assess Impacts Before Adoption
Every AI policy has impacts. Some are intended, some are not. Some are visible to the drafters and many are not. A policy mandating impact assessments for all AI systems might reduce algorithmic bias, which is the intended effect, while also creating a compliance burden smaller agencies cannot absorb, slowing deployments that would have benefited constituents, or pushing agencies to avoid AI entirely rather than bear the assessment cost. All of those are plausible. None of them appear in a policy document without systematic analysis, because nothing in a well-drafted requirement announces what it will cost the people who have to meet it.
Policy impact assessment is the practice of modeling those effects before a policy takes effect, rather than afterward when the damage is visible but the policy is already embedded in regulation and practice. It works like an environmental impact review for governance decisions: a structured process for asking who is affected, how, whether the benefits justify the costs, and whether design changes could achieve the same goal with fewer adverse effects. The honest framing is that poor policy has severe consequences. It can harm the people it was meant to help, create problems that did not exist before, and fail at its own objective while consuming the resources that might have achieved it.
The examples are ordinary rather than exotic. A requirement for impact assessments on every AI system is simple in concept and expensive in implementation. A transparency requirement may build public trust and slow deployments. A human oversight requirement may protect civil rights and create a bottleneck in decision-making. None of those trade-offs is an argument against the policy. They are arguments for knowing about them while the text can still change.
The Dimensions of Policy Impact
Policy impacts are multi-dimensional, and the failure mode is assessing the one or two dimensions the drafting team happens to be fluent in. Between them, the two source treatments behind this lesson name seven dimensions, and running all seven is what turns an opinion about a policy into an assessment of it. For each dimension, describe the current state and the state after the policy, model the difference, and write down the assumptions you had to make and how uncertain each one is.
| Dimension | The question it answers | What it typically surfaces |
|---|---|---|
| Stakeholder | Who is affected, how, and do different groups experience it differently? | Groups the drafters did not have in the room |
| Operational | What changes in how agencies work: new processes, roles, systems? | The transition cost nobody budgeted |
| Financial | What does it cost to implement and to keep complying, and who pays? | Unfunded obligations landing on the least resourced |
| Compliance | What new obligations arise, and do they conflict with existing ones? | Requirements that contradict a rule already in force |
| Innovation | Does the policy enable or restrict beneficial innovation, and can it adapt? | Blanket scope that deters low-risk useful work |
| Risk | What risks does the policy mitigate, and what new risks does it create? | The trade-off being made implicitly rather than deliberately |
| Distributional | Are costs and benefits fairly spread, or regressive across agencies and constituents? | A policy that is affordable centrally and punitive locally |
Two of these deserve particular attention because they are the ones drafters skip. Compliance impact asks whether the new obligation collides with an obligation already in force, which is a question nobody can answer from inside a single policy shop. Risk impact asks what the policy creates as well as what it prevents, and it is the discipline that keeps an assessment from becoming an advocacy document for a requirement the team already wants.
Stakeholder and Distributional Impact
AI policy in government typically affects at least four groups: the constituents whose decisions are made or shaped by AI systems, the frontline staff who use those systems, the agency leadership who must implement and fund compliance, and the vendors who build and maintain the tools. Each experiences the policy differently. A transparency requirement might help a constituent understand why a benefit was denied, and might require a significant vendor software update costing the agency $200,000 in the first year of compliance. Both statements are true at once, and only a stakeholder map shows them side by side.
Distributional analysis then asks the harder question: who bears the costs and who receives the benefits? A policy requiring impact assessments for all AI systems distributes cost unevenly by design. Large agencies can absorb it and can employ or contract assessment expertise. Small agencies cannot afford that expertise at all. The result is a regressive policy that hurts small organizations more than large ones, which is rarely what the drafters intended and almost always what they produced. The same logic runs to constituents: a policy protecting people who interact with sophisticated AI systems may deliver nothing to a rural constituent whose agency decisions are still made entirely by human caseworkers.
Finding a regressive effect is not a finding that the policy is wrong. It is a finding that the design needs an adjustment, and the source material names four that work: tiered requirements, so that larger or higher-risk organizations face higher bars; technical assistance programs for those without internal capability; templates that make compliance cheaper for everyone; and direct funding to support smaller agencies. Each of those preserves the policy goal. None of them requires weakening the protection at the top end, which is what makes them the right answer rather than a compromise.
Operational and Financial Impact
Operational impact asks what changes in how the work gets done. A bias assessment requirement changes procurement, because the assessment has to happen before deployment. It changes contracting, because vendors must supply assessment-ready documentation. It changes the IT review process, which gains a step. And it may change the staffing model, because someone has to run or commission the assessment. Operational modeling identifies every one of those changes and estimates the time each takes, which is how a requirement that reads as one sentence turns out to be four process changes.
For Catalina's agency, a straightforward estimate put full compliance in the first year at approximately 2,400 staff hours across fourteen systems. Dividing the agency's own two figures gives roughly 170 hours per system, which is the number worth carrying into a conversation with a program director, because it is the unit at which the work actually appears on someone's calendar. Against a technology team of twelve who already have a full portfolio, that total could not be absorbed without additional resources or a longer timeline, and saying so with the arithmetic attached is far more persuasive than saying the team is stretched.
Financial impact asks what implementation costs and who pays. The source material sorts costs into four categories: implementation costs, meaning the one-time expense of building assessment capacity, updating contracts and completing initial assessments; ongoing compliance costs, meaning the annual expense of maintaining the program across covered systems; opportunity costs, meaning the value foregone because the policy restricts certain activities, including deployments delayed while assessments run; and indirect costs borne by third parties such as vendors and states. Benefits sort into four categories as well: risk reduction, rights protection, trust building, and enabling responsible innovation.
Catalina's agency estimated external consultant support for the fourteen initial assessments at $1.1 million, an unbudgeted expenditure in a fiscal year when the contingency reserve was already exhausted. Two divisions make that number land. Against the fourteen systems it is roughly $79,000 per assessment. Against the agency's $8.4 million annual IT budget it is about 13 percent of everything the agency spends on technology in a year, redirected to a compliance activity that produces no new capability. Both figures come from dividing numbers the agency itself supplied, and both are the kind of thing a policy drafter needs to see before the comment period closes.
Compliance, Innovation and Risk Impact
Compliance impact asks what new obligations the policy creates, whether they conflict with requirements already in force, how heavy the resulting burden is, and how compliance will be monitored and enforced. That last clause is the one policy shops treat as somebody else's problem, and it is where a requirement quietly becomes unenforceable. A rule with no defined monitoring mechanism produces exactly the outcome its drafters feared: agencies reporting compliance nobody has checked. Assess it during drafting, and read AI Policy Monitoring and Enforcement alongside this lesson, because the enforceability of a requirement is a design property rather than an afterthought.
Conflicts are the other half. An agency subject to a new AI transparency obligation may already be operating under records, privacy or security requirements that pull the other way, and the agency, not the policy author, is the one who discovers the collision. Nobody inside a single policy shop can enumerate those conflicts from memory. The practical method is to ask the implementers directly which existing obligations the draft appears to touch, and to treat every conflict they name as a finding requiring resolution rather than as a complaint.
Innovation impact asks whether the policy enables or restricts beneficial work, and whether it is flexible enough to remain sensible as the technology changes. Governance policies with overly broad coverage or prescriptive technical requirements create burdens that deter agencies from adopting tools that would genuinely help constituents. Requiring a full bias assessment for every AI tool, including a scheduling assistant that books conference rooms, deters useful work without meaningfully reducing risk, and it also teaches staff that the assessment process is a formality, which damages it for the cases that matter.
The remedy is precision rather than leniency. Scope definitions, risk thresholds and tiered requirements preserve pathways for lower-risk applications while keeping rigorous review for high-impact systems. A policy pinned to a specific technique or architecture will also age badly, since a rule written around one generation of models can end up either regulating nothing or blocking everything as the technology moves. Write to the decision and its consequences for a person, rather than to the method.
Risk impact closes the frame by asking both directions at once: which risks the policy mitigates, which risks it creates, and whether that trade is one you would make deliberately. Every AI policy makes this trade, and the failure is making it implicitly. A requirement that delays deployment reduces the risk of fielding a biased system and increases the risk of leaving a manual process in place that was already producing inconsistent decisions. Neither risk is zero, both fall on the same constituents, and the assessment's job is to put them on the same page so the choice is visible to whoever signs it.
Cost and Benefit When You Cannot Quantify
The central difficulty in policy cost-benefit analysis is that many of the costs and most of the benefits are non-monetary and genuinely hard to quantify. Preventing discrimination and building public trust do not have prices, and pretending otherwise produces a number that looks rigorous and is not. The temptation is to quantify anyway, because a figure survives a meeting better than a paragraph does, and a fabricated figure is worse than no figure because it will be quoted for years by people who never saw the assumptions.
The workable approach is ranges and explicit uncertainty. The source material's illustration is a policy whose implementation cost is estimated somewhere between $50 million and $100 million, against benefits consisting of preventing discrimination and building trust. The honest presentation states the cost range, states the benefits in words, and is transparent about what could not be quantified and why. A range with its assumptions written down is more useful to a decision-maker than a point estimate whose derivation nobody can reconstruct, and it is far more robust when someone eventually checks.
Assumptions deserve their own treatment. Base an analysis on assumptions that are never written down and the analysis becomes meaningless the moment one of them is wrong, usually without anyone noticing that it was the assumption rather than the policy that failed. An analysis assuming agencies have the internal resources to implement a requirement, when they do not, produces a cost estimate that is not merely low but structurally wrong. Document every assumption, identify which ones the conclusion actually depends on, and plan how you will verify those before the policy is final.
Modeling Unintended Consequences
Unintended consequences are the hardest part of impact assessment, because finding them means going beyond what the policy text says to ask how real agencies, vendors and constituents will actually respond. The sources between them describe a consistent set. Regulatory burden causes organizations to avoid AI entirely, which defeats the policy goal, since a policy intended to govern AI has instead eliminated it. Strict requirements on one stakeholder create burden for another who was never considered. Requirements interact with existing rules in ways nobody modeled. Vendors exit the government market because compliance costs exceed their margins, which hits small vendors first. Development moves to jurisdictions with less demanding rules. And assessment requirements produce paper compliance without risk reduction, because the methodology was never defined well enough to make the assessment mean anything.
Four modeling approaches help, and they are complementary rather than competing. Scenario analysis imagines different implementation approaches, models the outcome of each, and identifies where unintended consequences are likely so mitigations can be designed. Stakeholder interviews ask the affected parties directly what consequences they foresee, which taps experience the drafting team does not have. System dynamics modeling maps how policy elements interact and where feedback loops amplify effects. And pilot testing implements the policy at limited scale and observes what actually happens, which the source material calls the best way to discover unintended consequences, because it substitutes evidence for imagination.
Catalina's office used the second of those. It convened a two-day working session with technology directors from eight peer agencies before the federal comment period closed. The finding was blunt: six of the eight said they would pause all new AI deployments for at least twelve months, working through assessments of existing systems before adding any new ones. That pause appeared nowhere in the policy analysis, and its downstream effect on constituent services was larger than any effect the policy was designed to produce. Two days of other people's time surfaced the dominant consequence of a national policy, which is the best return available anywhere in this discipline.
Stakeholder Engagement in Impact Assessment
Effective impact assessment requires input from the people the policy lands on, and the source material sorts them into four categories: government, meaning federal agencies and state and local governments; regulated entities, meaning vendors and industry; affected individuals, meaning the citizens the policy is ultimately for; and civil society, meaning advocacy groups and academics. Each category knows something the others do not, and a process that consults only the first will produce a policy that is administratively convenient and substantively thin.
Five engagement mechanisms are available, and they are not interchangeable. Surveys gather quantitative data on expected impacts across many respondents. Focus groups produce depth on impacts and concerns that a survey instrument would never have thought to ask about. Public comment periods are the formal route and create a record. Steering committees keep stakeholders involved over the life of the work rather than at a single moment. And feedback from pilot participants is the only source that reports on the policy as implemented rather than as imagined.
Then do the part that determines whether any of it was worth doing. Document what stakeholders said, and show how their input changed the policy. A consultation whose visible effect is zero teaches every participant that the next one is not worth their time, and the reputational cost is paid by whoever runs the next consultation. Showing the trace from input to change is what builds the trust and the buy-in that make the eventual requirement enforceable.
Pilot Before You Mandate
The most reliable way to discover unintended consequences is to implement the policy on a limited basis before requiring full compliance. The sequence is straightforward: select a subset of agencies or systems, implement the policy for them, monitor the impacts, gather feedback from the participants, refine the policy on what was learned, and only then implement at full scale. A pilot involving three to five volunteer agencies with active monitoring of compliance costs and operational effects produces empirical data that no amount of modeling substitutes for.
The benefits compound. Unintended consequences surface in the pilot rather than at national scale. Impacts are measured rather than argued about. Stakeholders are engaged early and are invested in the outcome. The policy is refined before rollout. And confidence in the final requirement is higher precisely because it has been tested, which matters when you later need agencies to comply with it rather than litigate it.
Size and duration are where pilots fail. A policy piloted for three months on a handful of systems, showing no problems, tells you very little, and the full rollout then reveals issues the pilot was structurally incapable of finding. Compliance costs accumulate over a cycle. Vendor responses take a procurement cycle to appear. Behavioral adaptation, such as agencies quietly reclassifying systems out of scope, takes longer still. Pilot at sufficient scale and for long enough that the effects you care about have time to appear, monitor deliberately, and be willing to extend rather than declaring success on schedule.
Catalina argued exactly this in her public comment. The federal agency ultimately adopted a phased implementation schedule, requiring compliance in the first year only for high-risk systems, meaning those affecting benefit eligibility and enforcement, with lower-risk systems following in year two. It also established a $15 million technical assistance fund to support agencies without internal assessment capacity. The timeline extension and the funding were direct results of the stakeholder analysis her office commissioned and submitted, which is worth noting: the assessment did not weaken the policy, it made the policy something agencies could actually do.
Using What the Assessment Finds
The final discipline is the one that makes all the previous work count. An impact assessment that is conducted rigorously and then filed unchanged alongside an unmodified policy has consumed real money to produce nothing. If the assessment shows the policy will harm small agencies, the policy changes, or the assessment was theater. That sounds obvious and it is the most common failure in the field, because by the time the assessment reports, the policy has sponsors, a timeline and a communications plan, and modifying it costs someone visible credibility.
Build the off-ramp in advance. Agree, before the assessment starts, what kinds of finding would trigger a design change and who has the authority to make it. Agree what the assessment is empowered to recommend. That agreement is cheap while nobody knows what the finding will be, and impossible to negotiate afterward. A requirement adopted without the resources to meet it is not a requirement; it is a mandate that produces paper compliance and erodes the credibility of the next policy the same body issues.
Anti-Patterns
- Assessing and then not acting. The team runs a thorough impact assessment, it shows the policy will harm small agencies, and the policy is adopted unchanged. Time and money were spent on analysis and the harmful impacts arrived anyway. Decide up front which findings would trigger a design change and who owns that decision, because after the report lands, changing course costs someone credibility and the assessment quietly becomes documentation.
- Reporting aggregates and ignoring distribution. Net cost and net benefit look acceptable, so the policy proceeds, and the aggregate conceals that large agencies gain while small ones absorb the cost. Policies that reinforce existing inequities usually got there through an average. Analyze who bears costs and who receives benefits explicitly, and treat a regressive result as a design problem to fix rather than a finding to note.
- Piloting too small or too short. A three-month pilot on a few systems shows no problems, and the full rollout reveals everything the pilot could not have caught: cumulative compliance cost, vendor response, behavioral adaptation. Scale and duration are what make a pilot informative. Monitor deliberately, and extend the pilot when the effects you care about have not had time to appear.
- Undocumented assumptions. The analysis assumes agencies have the staff to implement the requirement. They do not. Every downstream number is wrong and nobody can tell, because the assumption was never written down to be checked. Document all assumptions, mark the ones the conclusion depends on, and plan how each will be verified.
- Consulting without showing the trace. Stakeholders are surveyed, comments are collected, and the final policy is identical to the draft with no explanation. Participants learn that engagement is decorative and the next consultation gets thinner responses. Record what was said, show what changed because of it, and say plainly why you declined the input you did not take.
- Quantifying what cannot be quantified. Preventing discrimination gets a dollar value so the analysis balances, and that invented figure outlives every caveat attached to it. Use ranges, state the benefits in words, and be explicit about what could not be quantified. A transparent gap is more defensible than a confident fabrication.
- Assessing only the dimensions you understand. A team of economists produces a strong financial analysis and never asks whether the requirement conflicts with an obligation already in force or what new risks it creates. Run every dimension even where the answer is short, and bring in someone from outside the drafting shop for the ones your team cannot answer.
Practice Prompts
- Select a proposed AI policy and conduct a full impact assessment across all seven dimensions in this lesson: stakeholder, operational, financial, compliance, innovation, risk and distributional. Note which dimensions your team could not answer credibly and who you would need in the room to answer them.
- Build a cost-benefit analysis for a specific AI policy. Separate implementation, ongoing compliance, opportunity and indirect costs, and set them against risk reduction, rights protection, trust and innovation benefits. Mark each item quantifiable or not, and use ranges rather than point estimates where the uncertainty is real.
- Use scenario analysis to model unintended consequences of a policy you are developing. Write at least three implementation scenarios, model the outcome of each, and design a mitigation for every consequence you would not accept.
- Analyze how one policy's costs and benefits are distributed across agencies of different sizes and constituents in different circumstances. Identify the disparities, then design mitigations using tiered requirements, technical assistance, templates or funding.
- Design a pilot for a policy: scope, duration, which agencies, what you will monitor, success metrics, and the criteria that would justify full rollout, an extension, or abandonment. Then argue against your own duration and see whether it survives.
- Take an impact assessment your organization has already produced and list every assumption it rests on. Mark the ones the conclusion actually depends on, and write down how each could be verified.
Reflection
Think about the most recent AI policy or requirement your organization adopted, whether you wrote it or received it. Who modeled what it would cost the units expected to comply, and in what units did they express that cost: dollars, staff hours, or delayed deployments? If nobody did, what would you now expect that assessment to have found, and would the requirement have survived it unchanged? Then ask the question Catalina had to answer: if a requirement lands on your organization tomorrow with no funding attached, what is the evidence you could assemble in two weeks to show what it would actually take, and who would listen to it?
Glossary
- Policy impact assessment. A structured analysis of who a policy affects and how, conducted before adoption so that the design can still change.
- Distributional analysis. Examination of who bears a policy's costs and who receives its benefits, which is the only way a regressive effect becomes visible.
- Regressive policy. A requirement that imposes proportionally greater burden on smaller or less resourced organizations than on large ones.
- Tiered requirements. Scaling obligations by organization size or system risk so the burden matches capacity and the stakes.
- Opportunity cost. Value foregone because a policy restricts an activity, including benefits from deployments delayed while compliance work is completed.
- Indirect cost. Cost borne by a third party such as a vendor or a state government rather than by the agency the policy addresses.
- Unintended consequence. An effect not sought by the policy's authors, which may be beneficial, harmful, or destructive of the policy's own goal.
- Scenario analysis. Modeling several implementation approaches and their outcomes to locate where unintended consequences are most likely.
- System dynamics modeling. Mapping how policy elements interact, including feedback loops, to understand effects that a linear analysis would miss.
- Technical assistance program. Support provided to organizations lacking internal capability, used to prevent a requirement from becoming regressive.
Related Lessons
- Drafting Agency AI Policies is where the policy text this lesson assesses actually gets written.
- AI Policy Monitoring and Enforcement is the downstream half: what happens after a policy is adopted and someone has to check that it is followed.
- Navigating the Federal AI Landscape covers the wider set of requirements a new policy has to fit alongside without contradicting.
- State and Local Government AI Policy covers the sub-federal layer where a national requirement most often turns out to be regressive.
- Public Consultation on AI Policy works the engagement mechanisms in depth.
- Algorithmic Impact Assessments is the system-level assessment that this policy-level assessment is often mandating.
- Federal/State/Local AI Alignment addresses the conflicts between obligations that compliance impact analysis is designed to surface.
- Measuring AI Impact covers the measurement discipline behind any credible before-and-after estimate.
- Stakeholder Management and Communication covers the relationship work that makes a consultation produce useful answers.
- Rights-Impacting and Safety-Impacting AI Safeguards defines the risk tiering that makes tiered requirements possible.
Closing
Policy impact assessment is not optional and it is not a delaying tactic. It is the foundation of policy that works, and the whole discipline reduces to a short sequence: understand the impacts, engage the people who will bear them, and use what you find to improve the design before adoption rather than after harm. The alternative is not a faster policy. It is the same policy arriving with its problems intact and its credibility spent.
Catalina's story ends better than it started, and it is worth being precise about why. Her office did not defeat the policy or water it down. It produced evidence, in staff hours and dollars and named consequences, about what the requirement would do to agencies like hers, and it delivered that evidence while the text could still change. What came back was a phased schedule and a funding mechanism, which meant the fairness protection the policy was written to deliver actually reached beneficiaries instead of stalling in fourteen unassessed legacy systems. A policy that cannot be implemented is not a policy. It is an aspiration that generates paperwork and erodes the credibility of every policy that follows it.
Key Takeaways
- Model impacts before adoption, not after. Stakeholder burden, operational change, financial cost, compliance conflicts, innovation effects, new risks and distribution should all be estimated during drafting, while the text can still change.
- Run every dimension, not the ones you are fluent in. Compliance conflicts and newly created risks are the two dimensions drafting teams skip, and both require someone from outside the policy shop to answer.
- Distributional analysis is not optional. A policy large agencies absorb easily may be prohibitive for small counties. Mapping who pays and who benefits is how a regressive design gets caught while it can still be tiered or funded.
- Estimate cost in staff hours as well as dollars. A consultant bill is visible; the hours diverted from program work are not, until program outcomes decline. Divide the totals into per-system figures, because that is the unit at which the work lands on someone.
- Use ranges and be transparent about what you cannot quantify. Preventing discrimination has no price, and inventing one produces a figure that outlives its caveats and gets quoted for years.
- Document your assumptions and verify the load-bearing ones. An analysis assuming capacity that does not exist is not merely inaccurate, it is structurally wrong, and nobody can tell unless the assumption was written down.
- Convene implementers before the comment period closes. Two days with peer agency technology directors is cheap and routinely surfaces the dominant unintended consequence, because they know what they will actually do if the policy passes.
- Tiered requirements and technical assistance preserve the goal. Scope precision is not a loophole. It is what lets rigorous review apply where the stakes are highest without making the policy impossible everywhere else.
- Pilot at real scale and duration. A short pilot on a few systems that shows no problems has demonstrated nothing, and cumulative cost, vendor response and behavioral adaptation all take time to appear.
- Use the findings or do not run the assessment. Agree in advance which findings trigger a design change and who has authority to make it, because that agreement is impossible to negotiate once the report exists.
Frequently Asked Questions
How is this different from an algorithmic impact assessment?
They operate at different levels and the confusion is common. An algorithmic impact assessment examines one AI system: what it does, who it affects, how it might fail, what safeguards apply. A policy impact assessment examines a rule that governs many systems across many organizations, and asks what happens when every covered body has to comply. The relationship runs one way: a policy impact assessment frequently concludes that a requirement to run algorithmic impact assessments is affordable for some agencies and not others, which is a question the system-level assessment cannot ask about itself.
We do not have the resources for a full seven-dimension assessment. What is the minimum?
Run all seven, briefly, rather than three of them thoroughly, because the value of the framework is that it forces a question into the room rather than that each answer is deep. A single honest paragraph per dimension, with the assumptions marked, will catch more than an elaborate financial model that never asked about conflicting obligations. Then invest the remaining effort where the first pass showed the most uncertainty, and say explicitly in the writeup which dimensions received a light touch.
Does impact assessment just slow policy down?
It adds time before adoption and usually removes more time after it, though nobody gets credit for the delay that did not happen. The comparison to make is not assessment against speed, it is assessment against the cost of a policy that agencies cannot implement: revised guidance, extended deadlines, waiver processes and a body whose next requirement is taken less seriously. If the schedule genuinely will not allow a full assessment, run the abbreviated version and convene the implementers, because the working session is the cheapest and fastest part of the whole discipline.
What if the assessment shows the policy is not worth doing?
Then it has done exactly what it was funded to do, and the difficulty is organizational rather than analytical. This is why the trigger criteria and the decision authority need to be agreed before the assessment starts, while nobody knows what the answer will be. Note also that this outcome is rarer than teams fear. Assessments far more often show that a policy is worth doing in a modified form, with a phased timeline, tiered scope or support funding, than that the goal itself was misconceived.
How do we assess distributional impact without data on smaller agencies?
Ask them. The data usually does not exist in any central system, and the fastest route is a small number of structured conversations with organizations at the bottom of the resource range, asking what they have today and what the requirement would displace. Catalina's office got the dominant finding of its analysis from two days with eight peer agencies. If even that is impossible, state the gap explicitly in the assessment rather than extrapolating from large-agency figures, because that extrapolation is precisely how a policy becomes regressive without anyone deciding that it should.
Skill.re