Marcus Bell runs the records desk at a county clerk's office. Last spring his supervisor handed him a new tool and said, "It reads documents and tells you what's in them." Marcus typed in a deed and asked, "Is this property in a flood zone?" The tool answered, confidently, "Yes, this property is in FEMA Zone AE." It was wrong. The deed never mentioned flood zones at all. The tool had not "read" anything in the way Marcus assumed. Understanding why it answered that way, and why it sounded so sure, is the difference between a useful tool and a quiet liability.

This lesson explains how modern AI actually works, in plain terms, with no math. We are not doing calculus. We are working at the conceptual level: what is happening when a system "learns," what it is actually doing when it makes a prediction, and why that mechanism matters for government decision-making. The goal is not to make you an engineer. It is to give you an accurate mental model so you can predict where these tools help and where they break, before that gap costs your agency a mistake on the public record.

It predicts, it does not think

The systems people call "AI" today are mostly large language models, or LLMs. A useful way to picture one is an extraordinarily well-read autocomplete. Your phone's keyboard guesses the next word when you text. An LLM does the same thing, but trained on a staggering amount of text, so its guesses are far better and can run for paragraphs.

Here is the part that surprises most people. The model has no idea what is true. It was trained to answer one question over and over: given the words so far, what word probably comes next? When Marcus asked about a flood zone, the model produced the kind of sentence that usually follows that kind of question. "Zone AE" is a common, plausible-sounding answer. The model reached for plausible, not for true. This is why experts say LLMs are designed to be fluent, not factual. Fluency is the product. Accuracy is something you have to add on top.

The same point applies to every kind of AI system, not just chatbots. An AI system does not understand; it correlates. It does not reason; it interpolates. It does not think; it calculates probabilities. These are not limitations of today's technology that a better version will fix next year. They are what the technology is. An LLM is a confidence machine that has never been taught the difference between knowing and guessing.

Why the mechanism matters more in government

Government requires explainability, accountability, and consistency. When a decision affects a citizen, we need to know why, and citizens have a right to know. But an AI system cannot explain itself the way a human analyst can. It cannot say "I reviewed the evidence and here is my reasoning." At best it can say something like "the pattern in my training data suggests 73%." Understanding that difference is foundational to responsible government use, because the explanation a citizen is owed is not the kind of explanation the system is capable of producing.

It also unifies a confusing vocabulary. Machine learning, large language models, natural language processing, and computer vision get introduced as four different technologies, and in a procurement document they look like four different products with four different price tags. Under the hood they run the same engine: examples in, parameters adjusted, patterns applied to new cases. What differs is the kind of input, the kind of output, and the kind of mistake each one makes. Once you can see the shared mechanism, a vendor's category label stops doing your thinking for you, and you start asking about the training data instead of the brochure.

This is also why understanding the mechanism is a form of protection. If you know what the system is doing, you will never be fooled by one that sounds authoritative. You will know its real strengths and its real limits, which is exactly the knowledge you need when someone asks whether your agency should trust it.

What "learning" actually means

When we say an AI system "learns," we are using shorthand. The system is not learning the way you are learning this lesson, storing concepts and building understanding. It is doing something far more mechanical: adjusting thousands, or millions, of numerical parameters based on patterns in data.

Picture a vast spreadsheet with millions of columns. Each column is a tiny knob the system can turn. Initially every knob sits at a random position. You show the system an example: a document labeled "fraud" or "not fraud." The system makes a prediction. It is probably wrong. Then it asks itself the only question it knows how to ask: if I turn up column 47 a little and turn down column 892 a lot, will I get this one right?

It does this for every example, thousands of times over. Each example nudges thousands of knobs. After millions of adjustments the knobs sit in positions where new examples tend to get predicted correctly. That is learning. It is not understanding. It is optimization. But it works, and it works well enough to be worth deploying, which is precisely why the misunderstanding is dangerous.

Large language models are the same process at a different scale. During training the model read enormous volumes of text and adjusted billions of internal settings to get better at predicting the next word. It was never handed a database of facts to look up. It absorbed patterns: that "Dear" is often followed by a name, that "the capital of France is" is usually followed by "Paris," that legal documents have a certain rhythm.

Three consequences of learning patterns instead of facts

Three practical consequences follow from that design, and each one has caused a documented problem in a government setting. They are worth memorizing as a set, because they explain most of the surprises people report in their first month with these tools.

• It has a knowledge cutoff. The model only saw text up to a certain date. Ask it about a regulation passed last month and it may not know, yet it will still answer. It cannot tell you what it does not know.
• It reflects its training data. If the text it learned from carried bias, gaps, or errors, those patterns come along too. A hiring tool trained on past resumes can quietly reproduce past discrimination.
• It does not retrieve, it reconstructs. When it cites a case number or a statute, it is often generating something that looks like a citation rather than copying a real one. These invented details are called hallucinations.

Training data is everything

An AI system's behavior is completely determined by its training data. Feed it one dataset and you get one system. Feed it a different dataset and you get a different system. There is no third ingredient that rescues a bad one.

The implications are serious. If your training data contains biases, your system will perpetuate them. If your training data is missing important examples, your system will fail on those cases. If your training data comes from 2020 and the world has moved on, your system will make decisions using outdated patterns while sounding exactly as certain as it did when the patterns held.

The clearest example in public administration is recidivism prediction. A widely used tool for predicting reoffending was trained on historical criminal justice data. That data reflected decades of biased policing: Black defendants were overrepresented in the high-risk category not because they were more likely to reoffend, but because they were more likely to have been arrested and incarcerated. The system, learning from that data, perpetuated the bias. It had "learned" a false pattern, and it delivered that false pattern with the flat consistency of arithmetic.

This is why the NIST AI Risk Management Framework and OMB guidance require agencies to audit their training data. You need to know what patterns you are asking a system to learn, because whatever is in the data is what the system will treat as normal.

Correlation is not causation

AI systems find correlations. Correlation means these things tend to occur together. It does not mean one caused the other, and the system has no way to tell the difference.

Consider a school district that trains a system to predict which students will graduate. It looks at 100,000 student records and finds that students wearing expensive shoes are more likely to graduate. If the system then recommends interventions for at-risk students, it may flag students wearing cheap shoes.

The real pattern is not that cheap shoes cause graduation failure. It is that poverty, which correlates with cheap shoes, correlates with graduation failure. By using shoe price as a proxy, the system makes a decision based on socioeconomic status, which may violate equity and anti-discrimination laws. The system did nothing wrong by its own logic. Its own logic was never the safeguard.

This is why government use of AI requires domain expertise alongside technical skill. A statistician can build a model that finds all kinds of patterns. A subject matter expert has to ask the questions the model cannot: does this pattern make sense, is it a cause or a correlation, and is using this pattern ethical even if it is statistically real?

Confidence and accuracy are separate things

Here is what confuses people most: an AI system can be very confident and very wrong, and the confidence tells you nothing about the wrongness.

Language models are trained to produce grammatically perfect, coherent text. That has a side effect: they sound authoritative whether they are right or wrong. Ask a general-purpose chatbot about the ruling in a Supreme Court case and it will give you a detailed, well-written answer. If the case does not exist, it will invent one, and it will sound just as sure.

The reason is in the training. The system learned from text scraped broadly from the internet, where correct-sounding answers tend to be grammatically perfect and detailed. It learned to generate plausible text. It was never trained to generate only factual text, because its training data was itself a mix of accurate and inaccurate information. So it generalized to "coherent and detailed equals good answer" and applies that pattern regardless of truth.

For government this is not an abstract concern. A citizen may see an AI-assisted letter from an agency, take it seriously because it looks official and detailed, and make a real decision about their housing, their benefits, or their filing deadline on the strength of it. If the system made up the facts, the citizen carries the consequence.

Overfitting: when a system memorizes instead of learns

Imagine training a system to predict which government employees will resign within two years. You feed it 50 years of agency data: hire dates, positions, salaries, performance reviews, and whether each person left. Your system achieves 99% accuracy on the training data. Excellent news, surely.

Not necessarily. It may be overfitting: memorizing the training data rather than learning generalizable patterns. Applied to next year's employees, it fails. Last year's data was specific to those people under those conditions. The system did not learn that people with certain characteristics tend to leave. It learned that these specific people had these specific trajectories. Shown new people, it has nothing useful to say, and it says it confidently anyway.

This is why rigorous development requires testing on separate data the system has never seen, and why government use requires continuous monitoring of performance in real-world conditions rather than a single approving glance at the test results.

Walking through Marcus's mistake

Replay the flood-zone moment with the right mental model and the failure becomes predictable rather than surprising.

1. Marcus pasted the deed and asked a factual question.
2. The model scanned for patterns. Deeds and flood zones appear together often in its training text, so the topic felt familiar.
3. It generated the most statistically likely answer to "is this in a flood zone," which tends to be a definite yes or no with an official-sounding zone code attached.
4. It had no flood map, no parcel database, and no actual lookup. It produced a confident sentence shaped like an answer.

The tool was not broken. It did exactly what it was built to do. Marcus's error was treating a text-prediction engine like a records database. The fix is not better typing. It is a habit: treat every factual claim as a draft to verify, never a fact to trust.

Three cases: the same mechanism, three outcomes

Document classification that works. A revenue agency receives millions of documents annually. A language system classifies them as a return, an estimated payment, correspondence, an amendment, and so on. It learns from past documents labeled with their type. Shown a new one, it effectively says "this looks 94% similar to returns in my training data." That confidence is about similarity to training examples, not about being correct, and for this task that is fine. If it gets 90% right, human staff review the rest. The system speeds up work; it does not replace judgment.

Fraud detection done the wrong way. A benefits agency trains a system on years of claims: those approved and those later found fraudulent. The flaw is upstream. Not all fraud was ever caught, so undetected fraudulent claims sit in the training data labeled "approved." The system learns a pattern that quietly includes the fraud nobody found. It reaches 85% accuracy on test data. In production it misses fraud that looks different from past cases and flags legitimate claims that resemble past cases which were never actually fraud. The quality of the training data determined the outcome before anyone wrote a line of code.

Scheduling that expires. A transit agency uses AI to optimize staffing across dozens of stations. The system learns that peak demand happens at 8 AM and 5 PM, that winter months are busier than summer, and that weekends differ from weekdays. Those are real patterns in historical data. Then a new office building opens beside one station, or the agency runs late-night service for the first time. The system will be wrong, because it is extrapolating from a pattern that no longer describes the world.

A five-minute reliability check for any AI tool

Before you trust a new tool with real work, run this quick diagnostic. It takes about five minutes and tells you a great deal about how the tool will behave under pressure. Read the third column as what a reassuring answer looks like, not as a certificate. Five prompts going well is evidence about the tool's character, not a guarantee about the next answer it gives you.

If a tool fails the citation trap or the made-up-fact test, that is not necessarily a deal-breaker. It is a signal that this tool must be paired with human verification on anything factual. Document that finding and move on. The point is to know the tool's character before it touches the public record, not after, and to record what you found so the next person does not have to rediscover it.

Where this design is a genuine strength

None of this means the tool is useless. The same machinery that invents fake citations is excellent at language tasks where you supply the facts. Strong uses include rewriting a dense policy memo into plain language, drafting a first version of a constituent reply you will edit, summarizing a long document you already have, or turning bullet points into prose. In each case the truth comes from your input and the model just reshapes it. That is when fluency without facts is exactly what you want.

Weak uses are the mirror image: anything where the tool must supply facts from its own memory. Legal lookups, statistics, current events, specific case numbers, medical or benefits eligibility rules. If you need authoritative information, go to an authoritative source: the actual law, the actual regulation, the official policy. Use AI to help you understand or organize what you found there, not to substitute for it.

Anti-patterns

Trusting a model because its accuracy number is high. Accuracy is a number, and numbers seem objective, so 90% sounds precise and settled. What goes wrong is that a system reaching 90% accuracy on historical data may reach 72% on real-world data, because real-world conditions differ. Or accuracy is high on average and very low for a subgroup: 92% for men and 68% for women, with decisions harming the group the system serves worse. One hiring prediction system was 87% accurate overall and considerably less accurate for women, because the training data came from a male-dominated industry, and in production it systematically recommended fewer women for certain roles.

Look beyond the headline number, measure performance separately for different demographic groups, test on data matching real conditions, and keep measuring after deployment.

Assuming the system knows things it does not. The system seems smart and answered your other questions well, so surely it knows this too. What goes wrong is quiet: you ask about local regulations in your state, get a confident answer, and pass it to citizens as guidance. In one case a city employee asked a general-purpose chatbot about local zoning rules, received an authoritative-sounding but inaccurate answer, and shared it with constituents, who were then confused about what they were allowed to build. Recognize the boundary of the training domain, and treat anything outside it as a guess wearing a suit.

Deploying without monitoring real-world performance. Testing is expensive and inconvenient, and once a system is deployed and automated it feels finished. What goes wrong is drift. A system works fine, then something in the world changes: seasonal patterns, new case types, a shifting population. Accuracy degrades, and nobody notices because nobody is measuring. One fraud detection system worked well for its first year, then criminals adapted their tactics, and the system, still hunting the old patterns, missed the new fraud for eight months.

The fix is unglamorous and it is the difference between a controlled system and an unattended one. Build monitoring into the deployment rather than bolting it on later. Define the metrics that would reveal degradation, check them on a set schedule, and give someone the explicit job of looking. Have a process ready to retrain or adjust when performance slips, and keep a human override available for the periods when the system looks unreliable and nobody has yet worked out why.

Assuming AI removes bias. Algorithms look mathematical and neutral, and people have biases while math supposedly does not. What goes wrong is that the algorithm was trained on biased data, or the designers chose variables that proxy for protected characteristics such as zip code standing in for race, or the algorithm found a correlation that is statistically true and unfair as a decision criterion. The bias is perpetuated and sometimes amplified, now hidden under a veneer of objectivity. Predictive policing is the standing example: trained on historical police data reflecting biased patrol patterns, the algorithms recommended deploying police to already over-policed neighborhoods, more arrests followed, those arrests became training data, and the loop tightened.

Bias was amplified, not removed, and the veneer of objectivity made it harder to challenge than an openly biased policy would have been. Audit training data for bias before training. Test fairness outcomes across demographic groups after deployment. Keep human review on high-stakes decisions where an unfair pattern would do the most damage. The operating rule is short: do not assume an algorithm is fair, verify that it is, and write down how you verified it.

Practice prompts

• Data quality exercise. Identify a dataset your agency uses for an important process. What biases, gaps, or limitations might be in it, and how would each affect a system trained on it?
• Correlation versus causation. Find an article claiming an AI system discovered a causal relationship. Rewrite the claim to say what the evidence actually supports.
• Accuracy interrogation. If a vendor tells you a system achieves 90% accuracy, write down the questions you would ask. Accurate for whom? On what type of case? Tested against what data? Compared to what baseline?
• Model explanation challenge. Describe a decision your agency makes, such as hiring, approving benefits, or issuing permits. Could a system make it? What contextual judgment would it lack?
• Real-world monitoring. What metrics would you monitor if your agency deployed a system tomorrow? How would you know it was failing, and who would be responsible for checking?
• Run the five-minute check. Take a tool you already have access to and run all five tests. Write down what it did, and keep the note.

Reflection

Take three minutes with a decision your agency makes repeatedly: approving something, denying something, prioritizing something. Ask whether historical data contains the patterns you actually want the system to recognize, which points toward a task AI could support, or whether the decision requires contextual judgment that was never captured in any dataset, which points away from it.

Then follow it through. What would happen if you automated that decision tomorrow? Who would it affect first, and who would notice if it started going wrong? What safeguard would have to exist before you would be willing to put your own name on the output? If you cannot answer the last question, that is the work to do before the tool arrives, not after.

Glossary

• Training data. The historical examples used to teach a system patterns. The system's behavior is completely determined by this data.
• Pattern recognition. What AI systems actually do: finding statistical correlations in data. Not the same as understanding.
• Overfitting. When a system memorizes its training data rather than learning generalizable patterns, so it fails on new data.
• Correlation. When two things tend to occur together. It does not mean one caused the other.
• Statistical inference. Making predictions about new cases based on patterns found in historical data.
• Accuracy. The percentage of predictions a system gets right in testing. It does not tell the whole story; you need accuracy by subgroup and under real-world conditions.
• Bias in training data. When historical data reflects discriminatory patterns, such as police data reflecting biased policing or hiring data reflecting past discrimination. Systems trained on it perpetuate it.
• Domain. The specific area or type of data a system was trained on. Systems do not generalize well outside their domain.
• Hallucination. An invented detail, citation, or fact produced fluently and confidently by a language model.

Related lessons

• What AI Is and Is Not establishes the boundary this lesson explains the machinery behind.
• Types of AI Systems classifies the landscape so you can name which kind of system you are looking at before you assess it.
• What AI Does Well and Where It Fails turns this mechanism into a practical test for matching tasks to capabilities.
• Understanding AI Bias goes deeper on how training data carries discrimination forward and what auditing for it involves.
• AI Confidence and Hallucination takes the confidence-versus-accuracy problem further with techniques for catching invented content.

Closing

Now you understand the mechanism. An AI system is not magic and it is not thinking. It is finding patterns in historical data and applying them to new cases. That mechanism has real strengths: it is fast, it works at scale, and it can surface subtle patterns humans would miss. It also has real limits: it can only find patterns that existed in the data it saw, it does not understand context, it does not adapt when the world changes, and it carries forward whatever bias the data contained.

Keep this mechanism in mind, because it is the foundation for every evaluation decision you will make about AI in your agency. When a vendor demonstrates something impressive, you now have the right question ready: what was it trained on, how would you know if it was wrong, and who checks? Marcus asks that question now. It takes him about a minute and it has already saved him from putting a second invented fact on the public record.

Key Takeaways

• Predict, not think. An AI system correlates rather than understands, interpolates rather than reasons, and calculates probabilities rather than thinking. That is what the technology is, not a temporary limitation.
• Learning means optimization. Training adjusts vast numbers of numerical parameters until predictions improve. There is no concept, no comprehension, and no fact database inside.
• Training data determines everything. Biased data yields biased systems, incomplete data yields systems that fail on new cases, and stale data yields systems confidently applying a world that no longer exists.
• Correlation is not causation. A system can find that students in expensive shoes graduate more often; acting on that pattern means deciding by socioeconomic status, which is a legal and ethical problem, not a technical one.
• Confidence is not accuracy. A polished, certain answer carries no guarantee of correctness, and hallucinated citations and statistics are built into how the model works.
• Test accuracy is not real-world accuracy. Headline accuracy hides subgroup failures and degrades as conditions drift, which is why measurement continues after launch.
• You are the verification layer. Treat every factual claim as a draft to confirm against a real source before it reaches a citizen or a record, and run the five-minute check before trusting a new tool.

Frequently Asked Questions

If the model does not know facts, why does it sound so certain?

Because certainty is a property of the writing style it learned, not of the knowledge behind the answer. It trained on text where correct-sounding answers tended to be grammatically perfect and detailed, so it generalized that coherent and detailed equals good. It applies that style whether the underlying claim is true or invented. Treat tone and confidence as formatting, never as evidence.

Does a high accuracy score mean a system is safe to deploy?

No. A single overall number hides two failure modes that matter most in government. Accuracy measured on historical test data can fall substantially on real-world data, and accuracy can be high on average while being far lower for one demographic group. Ask for performance broken out by subgroup and measured under production conditions, and keep measuring after launch, because performance drifts as the world changes.

Can we just remove the biased fields from the training data?

Not reliably, and assuming so is one of the standing traps. Systems find proxies: zip code stands in for race, shoe price stands in for household income. Removing an explicit field does not remove the pattern if something correlated with it remains. The work is auditing the data for what patterns it actually contains and testing fairness outcomes across groups after the fact, not just editing the input columns.

What does a fair explanation to a citizen look like if the system cannot explain itself?

It comes from the human, not the model. The system can only report something like a statistical similarity to past cases, which is not the reasoning a person is owed for a decision affecting them. That is precisely why a human stays accountable for consequential determinations: the accountable person can state the reasons, cite the rule applied, and answer an appeal. The model's output is an input to that reasoning, never a substitute for it.

Is a rules-based system safer than a learning-based one?

Safer in one specific way and not in others. A rules-based system is interpretable, because you can read the rules, and it applies the same logic every time. That makes it predictable, which is not the same as correct: a wrong rule is wrong consistently and at scale. It is also brittle, failing whenever a situation arises that nobody wrote a rule for. Choosing between them is a governance decision about which failure mode you can detect and live with.