AI FinOps: Token Economics, Model Routing, and Controlling the AI Bill
"This is up 40%. What did we get for it?" That is the whole email. Finance forwards last month's AI invoice to the CIO with that one line, and nobody can answer it, because the bill arrives as a few totals: a model API charge, a block of assistant seat licences, maybe a capacity commitment, none of it split by team or feature. AI FinOps is the practice that fixes this. It makes AI spend visible, attributable to a team and a feature, and tied to what the business got for the money. In this lesson you will learn how the AI bill is really built, why it climbs while token prices fall, and how to run a six-step workshop that ends with a one-page unit cost for one use case, a seat count finance can act on, and a hard ceiling on anything that can loop.
The AI FinOps Problem: One Email, No Unit, and What It Costs
Follow the email. The CIO forwards it to the AI product owner, who answers, truthfully, "Usage is up." Finance asks which usage, then asks why there are 400 assistant licences on the software line when about half the company uses the assistant. Silence.
Nobody in that thread is being lazy. They simply have no unit to argue about. A service desk lead can say what a ticket costs to close. The AI team has two invoice lines and a hunch.
The accounting agent that looped
Mandiant's special report "AI risk and resilience in 2026" (Google Cloud, September 2026) describes an agent at a global financial services firm, built to reconcile anomalies in accounting ledgers, with read and write access to internal billing databases. A corrupted null value broke one of its tools. The agent did what agents do when something fails: it tried again, in what Mandiant calls an unconstrained, recursive reasoning loop to brute force a fix. In under an hour it made over 15,000 high-cost reasoning API calls, ran up a cloud bill spike of about $50,000, and locked the database so badly that live business transactions stopped.
One caveat before you repeat this in a steering committee: the firm is not named, so treat it as a reported incident, not an audited case study.
The agent was not attacked. It met bad data, retried, and found nothing in its way. The controls Mandiant recommends are mostly a few lines of configuration:
- Cost-cap thresholds, with observability and monitoring requirements defined before the agent goes live.
- Automated financial circuit breakers that halt the agent after a set number of consecutive task failures.
- Real-time financial caps, bounded recursion limits and rate limits, set per service identity and per project.
- Identity and role-based access controls for the agent itself, so it holds only the access its job needs.
If you took Lesson 13 on defending against prompt injection and oversharing with the OWASP LLM Top 10, you will recognise this as LLM06:2026, Unbounded Consumption, arriving by accident instead of by attack. The bill does not care which.
How common is the AI cost problem?
- Worry is near universal, control is not. In EY's US AI Pulse Survey (press release 28 July 2026; 534 senior US leaders, fieldwork April to May 2026), 82% of leaders at AI-investing organisations said their organisation is concerned about AI token usage and related costs. Only 64% of the same AI-investing leaders said their organisation actively monitors AI token usage and has clear budgetary guardrails.
- Ownership is what separates the firms that can answer finance. Proving value was the most-named challenge at 43%, and organisations with defined ownership of AI economics were 3.7 times more likely to show value to the CFO (State of Tokenomics survey, September 2026, as cited on the FinOps Foundation's "FinOps for AI Overview" page, updated 28 October 2026).
- It has landed on the FinOps team's desk. 98% of FinOps respondents now manage AI spend, up from 31% two years earlier (State of FinOps 2026, as cited on the same FinOps Foundation page).
Read the second bullet twice. The strongest predictor is not a tool. It is a name on a line that says "owns AI economics".
Where this lesson sits on the ladder
This is Level 3, AI Operational, lesson 4 of 5. Lesson 11 gave each AI system a release record and its model signals. Lesson 12 defined its data. Lesson 13's AI security workshop gave it a threat map, and one row on that map, unbounded consumption, is where today starts. Today you add the fourth record: a unit cost. Next, Lesson 15 on model inventory, NIST AI RMF, ISO 42001 and the EU AI Act timeline gathers all four into one row per system, the view an auditor or board asks for first.
How Token Economics Works: The Courier That Charges by Weight
Your company uses a courier that charges by weight. Someone in the business writes a one-page letter. Before it goes out, the mailroom staples the entire company handbook to it, plus every earlier letter in the same correspondence, plus a few pages pulled from the filing cabinet in case they are relevant. Nobody decided this; it is how the pilot's envelope template was set up.
The courier's price per gram has fallen every year. Your postage bill has tripled. Finance sees the second.
The mailroom has a second cost too. Every desk in the building has a reserved courier account with a monthly fee, paid whether that desk sends anything or not. Every idea below is one part of that mailroom.
What is a token, and what is token economics?
A token is the small chunk of text a model reads or writes, and the unit API usage is billed in. Token economics is the study of how many tokens each piece of work consumes, at what price, and whether the result was worth it. Tokens are the grams.
Why is my AI bill going up when token prices are falling?
Because the price per token is falling slower than the number of tokens you send is rising.
The price drop is real. a16z's "LLMflation" analysis, 12 November 2024, found that for equal capability, inference cost had fallen about 10 times per year. GPT-3-level quality went from $60 per million tokens in November 2021 to $0.06 in November 2024, roughly a thousandfold in three years. That is 2024 data, and the author cautions that the method is imperfect and the rate may slow. Use it as direction, not as a budget forecast.
So why does the bill rise? Three reasons:
- The envelope is mostly staples. You are billed for far more than the user typed.
- One task is many parcels. An agent calls the model many times to finish one job.
- Nothing stops a loop. A retry with no ceiling is 15,000 parcels in an hour.
What are hidden tokens?
Hidden tokens are everything in a model call that the user did not type. The FinOps Foundation's guidance is plain about it: billed tokens include system prompts, conversation history and retrieved content, not only the user's words.
In a chat assistant, the system prompt (instructions, policies and tool definitions) is re-sent on every call. The conversation so far is re-sent on every turn, so the tenth message carries the first nine. Whatever retrieval pulled from the knowledge base rides along too. The user's sentence is often the smallest item on the scale.
This is why a harmless-looking prompt edit can move the invoice: add two pages of policy to the system prompt and you have added two pages to every envelope.
What else is on the AI bill besides tokens?
Tokens are only one way AI is priced. The FinOps Foundation's guidance lists the pricing models side by side, each with its own way of wasting money:
| Pricing model | How it charges | What to watch for |
|---|---|---|
| On-demand | Per token, call or hour used | Runaway usage with no limits or monitoring |
| Subscription or seat | A recurring fee per user or platform, including AI add-ons inside software you already buy | Paying for seats that are not used |
| Commitment discount | A lower rate in exchange for committed usage or spend | Underused commitments when workloads change |
| Provisioned capacity | A fixed block of throughput, billed whether used or not | Low utilisation; it may cost more per token than on-demand |
| Batch | Discounted, asynchronous processing | Only suits work that can wait |
Seat licences are the reserved courier accounts. Each looks small, renews quietly, and goes idle without tripping any alert. A token bill shouts when it goes wrong. A seat bill whispers.
How should AI seat licence chargeback work?
Showback means showing each team what its AI use costs. Chargeback means actually billing that cost to the team's budget. The FinOps Foundation's advice is to report through showback first, broken down by project and workload, before charging anyone, because visibility alone often changes behaviour. It also flags chargeback for shared models and platforms as one of the places AI differs from ordinary cloud cost.
That gives you a sequence. Allocate first: the Foundation says to map each API key to one team, application and cost centre; for seats, assign each licence to a cost centre. Show back for a quarter or two so business units can correct their numbers. Then charge back, in the budget cycle, once the figures stop surprising people.
In the same EY survey, 82% of senior leaders said they expect the traditional per-seat SaaS pricing model to become less relevant in their industry within five years. That is an expectation, not a fact, but it is a reason to allocate by usage rather than headcount.
What is cost per successful outcome?
Cost per successful outcome is the total cost of all runs divided by the number of runs that met your quality bar. It is the headline unit in the FinOps Foundation's guidance, and it is the one number that lets finance and the product owner have a real conversation.
Why not cost per call? Because cost per call flatters failure. If a cheap answer is wrong and the user asks again, you paid twice and delivered once. A courier that loses a third of your parcels is not cheap because its per-parcel rate is low.
The Foundation defines several companion measures with formulas, and you will use these in the workshop:
- Cost per inference and cost per million tokens: the raw rates.
- Cache hit rate: the share of input tokens served from cache, calculated as cache reads divided by cache reads plus cache writes plus uncached input.
- Anomaly cost percentage: the cost of anomalous spikes divided by total AI spend. Anomalies include loops, prompt changes that double token use, and model switches that change the price.
- Model fit: the share of requests sent to a more capable, more expensive model than the task needs.
Notice that uncached input sits in the cache hit rate's denominator, so a feature that caches its system prompt but re-sends long history and retrieved pages will show a modest rate. The formula is telling you most of the envelope still goes at full postage.
One more piece of its advice is worth adopting on day one: report unit costs as a range or distribution, not only an average. An average hides the one ticket type that costs seven times the others.
How does prompt caching reduce LLM costs?
Prompt caching means the vendor stores a repeated block of your prompt so that re-reading it is billed at a fraction of the normal input price. In the mailroom, the courier keeps a copy of the handbook at its depot and charges you a tenth of the weight each time a letter refers to it.
One vendor's published rules make the arithmetic concrete; treat them as an example of the pattern, not a recommendation. In Anthropic's prompt caching documentation (as read in November 2026), writing to the cache costs 1.25 times the base input price with the default 5-minute lifetime or 2 times with a 1-hour lifetime. Reading from the cache costs 0.1 times base input on most models, and 0.05 or 0.025 times on some of the newest ones. Each read refreshes the cache's lifetime at no extra cost.
Do the sum once by hand, using an illustrative base input price of $3 per million tokens (several models in that table list it; substitute your own). A fixed 10,000-token system prompt costs $0.03 every time it is sent uncached. As a cache read at 0.1 times it costs $0.003. That is a 90% cut on that portion of the call, plus a 25% premium every time the cache has to be rewritten, which on the default setting happens whenever more than five minutes pass without a call that uses it. A busy feature rewrites rarely; one called twice an hour rewrites almost every time and may save nothing.
The catch is ordering. Caching works on a stable prefix, the front of the prompt that does not change between calls. Put the customer's name or today's date at the top and everything after it stops matching. Prices and multipliers change, so check your own vendor's current list before you put a number in a business case.
What is model routing and how much does it save?
Model routing means sending each request to the cheapest model that meets the quality bar for that task. In the mailroom: the bicycle messenger takes the letter across the street, and the armoured van is saved for the bank run. Most organisations send everything by armoured van because that is what the pilot used.
How much it saves depends entirely on the task, and the best public evidence shows exactly that. The RouteLLM research from UC Berkeley and Anyscale (summary posted 13 September 2024) reported that, while keeping 95% of GPT-4 performance, routing cut cost by over 85% on one benchmark (MT Bench), 45% on another (MMLU) and 35% on a third (GSM8K).
The lesson is in the spread: same method, same quality target, and the saving more than doubles from one kind of work to another. You find your number by running your own evaluation set, the golden set you built in Lesson 8 on building a golden test set, through a cheaper model and counting what still passes.
What do Crawl, Walk and Run mean for AI spend?
They are the FinOps Foundation's three maturity stages. Crawl is pilots with cost and time limits agreed up front, tracked by hand. Walk is AI inside regular processes, with basic automated tracking, anomaly detection, and separate budgets for running a system and for releasing changes to it. Run is AI in core processes, with unit economics and return on investment in leadership reporting. A Level 3 organisation should land at Walk. You do not need a cost platform to get there. You need tags, one sheet, one owner and a cap.
The AI FinOps Workshop: Build Your AI Unit Economics Sheet This Week
Pick one use case, the one with the largest or fastest-growing bill. One finished sheet teaches the room more than ten half-filled ones.
Who is in the room: the AI product owner, the finance business partner who sent the email, one platform engineer who can change how calls are made, whoever owns the evaluation set from Lesson 8, and for the last step the person who administers the assistant licences (often the software asset manager).
Time: one 2-hour session, then a follow-up hour after a week of tagged data.
Bring: last month's model invoice, call logs from your gateway or vendor console, the licence admin report with each seat's last active date, the golden set, and the vendor's current price list.
The six steps
- Name the owner and tag the spend. Write one person on the sheet with the mandate to set budgets and caps. Not a committee. If the room cannot agree on a name in ten minutes, you have found the real reason the invoice is a mystery. Then agree the tags every call will carry: team, feature, model, and cached or not, with each API key mapped to one team, application and cost centre. If your calls already pass through an AI gateway, the shared platform that Lesson 16 on the internal AI platform and AI gateways builds out, add the tags there once; if not, add them in the one code path every call uses.
- Weigh one typical call. Put a real request on the screen and split its input tokens four ways: system prompt, conversation history, retrieved content, user text. Write the four numbers down. This is usually when finance stops asking why usage is up and starts asking why the handbook is stapled to every letter.
- Compute cost per successful outcome. Total cost of all runs, divided by the runs that met the quality bar. Use the quality definition you already have from the golden set; do not invent a new one in the meeting. Then split it by task type and record the lowest and highest, not only the average.
- Cap anything that can loop, and budget the rest. Give the feature a monthly budget with an alert before it is reached. On anything agentic, add a hard spend cap, a maximum number of iterations per task, and a circuit breaker that stops the agent after a set number of consecutive failures, which is the control Mandiant names for the looping agent. Ask the room one question: what stops the 15,001st call? If the answer is "someone would notice", you do not have a cap.
- Cut waste, with evidence. Three moves, in this order. Cache the stable prefix: move the content that never changes (system prompt, policies, tool definitions) to the front, push anything variable to the end, and switch on caching. Route on the golden set: run it through a cheaper model, grouped by task type, and route only the task types where the pass rate holds; the share that passed is your model fit figure. Batch what can wait: Anthropic's batch processing documentation, for example, charges batch requests at 50% of standard prices, with most batches finishing within an hour and a 24-hour limit; check your own vendor's terms. Recompute step 3 after each.
- Count the seats and the surrounding costs, then show back. From the licence report, count seats with no activity in the last 30 days (a working threshold; pick yours and write it down), and agree who reclaims them before the next renewal. Then add the rows the invoice never shows, which the FinOps Foundation says to count (data stores, orchestration, observability and platform labour), plus the compliance controls and failed outputs its guidance also flags. I will not give you benchmark figures for these, because I have not seen trustworthy ones. Use your own. Finally, send each business unit its showback figure and put the sheet on the agenda of a monthly meeting that already exists, chaired by the named owner.
Why the monthly meeting? Advice that sits in a spreadsheet stays there. Findings need an owner and a route.
Worked example: a support reply assistant and its seats
Everything in this example is invented for illustration. Picture a 500-person business software firm with an assistant that drafts replies to support tickets. A human agent reviews each draft and sends it. The firm also pays for 400 seats of a general-purpose assistant.
Last month the support feature cost $4,200 in API charges for 60,000 runs. The quality bar, taken from the golden set, is simple: the reply was sent with no rewrite. 45,000 runs met it.
In step 2 the team weighs a typical call and finds this on the scale:
| Part of the input | Tokens | Share |
|---|---|---|
| System prompt and support policy (identical on every call) | 8,000 | 49% |
| Conversation history | 2,000 | 12% |
| Retrieved knowledge base articles | 6,000 | 37% |
| The customer's actual ticket | 300 | 2% |
The customer's words are 2% of what the firm pays to read. Half the envelope is the handbook.
The example prices input at an illustrative $3 per million tokens and cache reads at 0.1 times that. Substitute your own price list.
| Measure | Before | After caching the prefix | How it was calculated |
|---|---|---|---|
| Cost per inference | $0.070 | about $0.048 | API charges divided by 60,000 runs |
| Fixed prefix cost per call | $0.024 | $0.0024 | 8,000 tokens at $3 per million, then at 0.1 times |
| Fixed prefix cost per month | $1,440 | $144 | Per-call cost times 60,000 |
| Monthly API charges for the feature | $4,200 | about $2,904 | Minus $1,296, before cache rewrites |
| Cost per successful outcome | $0.093 | about $0.065 | API charges divided by 45,000 successful runs |
| Cache hit rate | 0% | about 49% | Cache reads divided by reads plus writes plus uncached input |
| Model fit (overserved requests) | 40% | to be re-measured | 24,000 of 60,000 runs passed on a cheaper model in the evaluation |
| Assistant seats with no activity in 30 days | 130 of 400 | reclaim at renewal | Licence admin report, last active date |
Four things to notice.
First, the gap between $0.070 and $0.093. The first is what a run costs; the second is what a usable reply costs, because a quarter of drafts were rewritten. Show finance the second, and note that raising the pass rate may lower it more cheaply than cutting tokens.
Second, one layout change removed about $1,296 a month with no change in quality, because the model reads the same text. Rewrites trim that a little. Suppose the cache lapses about 20 times a day, overnight and in quiet spells: 600 rewrites a month at 1.25 times cost about $18 instead of $1.44 as reads, so the true saving is nearer $1,280. Yet the cache hit rate is only about 49%, because history, retrieved articles and the ticket are still uncached input. That is the Foundation's formula pointing at the next target: trim the 6,000 retrieved tokens.
Third, model fit. Two ticket types, password resets and order status, passed just as often on a smaller model, and they were 24,000 of the 60,000 runs. The team routes them, leaves billing disputes on the larger model where the pass rate dropped, and sets a hard daily cap.
Fourth, the seats. At an illustrative $30 per seat per month (use your contract price), the 400 seats cost $12,000 a month, nearly three times the support feature's API charges, and the 130 idle seats alone cost $3,900. The team does not cancel seats in the meeting. It sends each department a showback report of active and idle seats, asks managers to release or justify the idle ones, and books the reclaim for the renewal date.
Template: the AI Unit Economics Sheet
This is the artifact you leave the workshop holding. One sheet per use case, one page.
| Field | What to write | Filled from |
|---|---|---|
| Use case and product owner | Name of the feature and the person accountable for it | Step 1 |
| Owner of AI economics | One named person with authority over budgets and caps | Step 1 |
| Tagging and allocation | Team, feature, model, cached or not; API key to cost centre map; share of spend allocated | Step 1 |
| Token breakdown of a typical call | System, history, retrieved, user, output | Step 2 |
| Monthly cost, runs, successful runs | From the invoice, the call logs and the quality bar | Step 3 |
| Cost per inference and cost per successful outcome | Both figures, with lowest and highest by task type | Step 3 |
| Budget, alert threshold, hard caps | Monthly budget; alert level; caps on spend, rate and iterations; consecutive-failure breaker | Step 4 |
| Anomaly cost percentage | Cost of anomalous spikes divided by total AI spend | Step 4 alerts |
| Cache hit rate | Cache reads divided by cache reads plus cache writes plus uncached input | Step 5 |
| Model fit and routing rules | Share overserved; each route with the evaluation result that justifies it | Step 5 |
| Batch candidates | Work that can wait, and the vendor's batch terms | Step 5 |
| Seats and utilisation | Seats paid, seats active in 30 days, renewal date, who reclaims | Step 6 |
| Surrounding costs | Data stores, orchestration, observability, platform labour, compliance controls, failed outputs | Step 6 |
| Showback and review | Which business units get a report, when chargeback starts, next review date and chair | Step 6 |
After a week of tagged data, hold the follow-up hour: replace estimates with measured values, test that one cap actually fires in a non-production environment, and send the sheet to finance as the reply to that one-line email.
Failure Modes, How to Spot Them, and the Metrics to Track
Each has a one-question test to ask in your next review.
Seven ways AI cost control fails
- One invoice line. No attribution. Test: "What did this single feature cost last month?" If the answer takes more than a minute, calls are not tagged.
- No caps on agents. Test: "What stops the 15,001st call?" The answer must be a configured limit, not a person.
- Premium model for everything. Test: "What is our model fit?" If it has never been measured, assume a large share of requests is overserved.
- Uncached repeated context. Test: "What is the cache hit rate?" The same long prefix in every call with a rate of zero, or unknown, is the handbook going out at full postage.
- Cost per call as the headline. Failures look cheap. Test: look for the quality bar in the cost report. If it is missing, the report is measuring activity.
- Routing without evaluation. The mirror image of failure 3: a cheaper model swapped in, and quality fell quietly. Test: "Show me the golden-set result that justified this route." No result, no route.
- Seats nobody reviews. The token bill gets all the attention while idle licences renew. Test: "How many seats were active in the last 30 days, and when do they renew?" If nobody knows, the quiet line is probably the bigger leak.
Which AI FinOps metrics should you track?
Track six, monthly, per use case. The first four are the FinOps Foundation's own measures, with its formulas; the last two are working measures for this course.
| Metric | How to measure it | What a bad reading tells you |
|---|---|---|
| Cost per successful outcome | Total cost of all runs divided by runs that met the quality bar; report the range | Either tokens are wasted or the pass rate is low. Check both. |
| Model fit | Share of requests sent to a more capable, more expensive model than needed, measured on the evaluation set | Routing has not been tested, or the routes are stale after a model change |
| Cache hit rate | Cache reads divided by cache reads plus cache writes plus uncached input | The prefix is not stable, caching is off, or most input is variable |
| Anomaly cost percentage | Cost of anomalous spikes divided by total AI spend | Caps and alerts are missing or set too loosely |
| Share of spend allocated | Spend carrying a team and feature tag, or a cost centre, divided by all AI spend | Every other number on the sheet is partly a guess |
| Seat utilisation | Seats active in the last 30 days divided by seats paid | Licences are renewing on headcount, not on use |
Two things to watch on the horizon. First, the FinOps Foundation says FOCUS v1.5, the next version of its billing data standard, is scheduled for ratification in December 2026 and is expected to add model identity, input and output labels, cache status and requester attribution. That is a schedule, not a done deal, but tags you add now should line up with it. Second, the caps on one agent are the seed of something bigger: when you run fleets of agents, per-agent spend and call ceilings become kill switches, which Lesson 22 on governed autonomy, agent identity and kill switches builds at Level 5.
Frequently Asked Questions
What is AI FinOps?
AI FinOps is the practice of making AI spend visible, attributable to teams and features, and tied to business value. The FinOps Foundation's guidance, updated on 28 October 2026, centres it on cost per successful outcome, the cost of a result rather than of activity, and it covers seats and commitments as well as tokens.
How do I reduce LLM API costs without hurting quality?
Cut tokens that do not change the answer, and change models only where your test set proves quality holds. In practice: cache the stable front of the prompt, trim retrieved content and history, batch work that can wait, and route simple task types to a cheaper model after running your golden set. Measure the result as cost per successful outcome, so a cheaper but worse answer cannot look like a saving.
How do you calculate cache hit rate?
Divide cache reads by the sum of cache reads, cache writes and uncached input, all in tokens. That is the FinOps Foundation's formula. Because uncached input is in the denominator, a feature that caches only its system prompt but re-sends long, changing context will show a modest rate, which is a signal to trim that context.
Should we charge AI costs back to business units?
Yes, but show back before you charge back. The FinOps Foundation advises reporting each team's AI costs by project and workload first, because visibility alone often changes behaviour. Allocate API keys and seat licences to cost centres, run showback for a quarter or two, then move to chargeback in the budget cycle once the numbers are trusted.
How do we stop an AI agent running up a huge bill?
Put configured limits in its way: a hard spend cap, a maximum number of iterations, rate limits, and a circuit breaker that halts it after repeated failures. Mandiant's September 2026 AI risk and resilience report describes an unnamed firm's accounting agent, without such limits, making over 15,000 calls in under an hour and running up about $50,000 in charges.
Key Takeaways
- Watch the bill, not the price. Price per token has been falling fast (a16z, November 2024), yet bills rise because context, usage and agent loops grow faster.
- The AI bill is more than tokens. Seat licences, commitments and provisioned capacity each waste money in their own way; idle seats are the quiet leak.
- Manage one unit: cost per successful outcome. Total cost of all runs divided by runs that met the quality bar, reported as a range.
- Cap anything that can loop. Spend caps, iteration limits and a consecutive-failure breaker are a few lines of configuration against a reported $50,000 hour (Mandiant, September 2026).
- Cache, route and batch with evidence. Cache reads can cut the repeated portion by 90% or more, routing savings vary widely by task, and the golden set decides every route.
- Name one owner and show back first. Organisations with defined ownership of AI economics were 3.7 times more likely to show value to the CFO (State of Tokenomics, September 2026, via the FinOps Foundation).
- Leave with the AI Unit Economics Sheet. It is the fourth record for each system, and Lesson 15 puts it into the model inventory alongside the release record, data definition and threat map.
Skill.re