←
AI-Ready Foundation
Strategic · M5 · lesson 5 of 5 · queued
Preview — browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll →
The Internal AI Platform: AI Gateways and Reusable Components That Stop Teams Rebuilding
📖
now learning

The Internal AI Platform: AI Gateways and Reusable Components That Stop Teams Rebuilding

15 min

An internal AI platform is the set of shared services every AI use case needs, built once and offered to every team: an AI gateway in front of your model providers, shared retrieval, a registry of tools and agents, an evaluation harness and an intake review. Without one, each team that ships AI builds its own model access, its own key storage, its own data scrubbing and its own logs, and you pay for the same plumbing again and again. This lesson shows what belongs in the platform, how to run it as a product (reliability targets, funding, model retirements, intake at scale), and how to run a two-session workshop that ends with a Paved Road Charter.

The Problem: Every Team Builds Its Own AI Stack, and You Pay for Each One

The AI invoice arrives, and your finance partner forwards it to you with one line: "Which of these are yours?"

Here is how a bill like that comes to exist (the details are illustrative; the pattern is not). You recognize two of its lines. The rest belong, you eventually discover, to three teams who each connected a model provider to their own application. Each holds its own API keys, wrote its own code to strip personal data out of prompts, and invented its own logging format. Each waited in its own queue for a security review that started from a blank page.

Nobody did anything wrong. Each pilot was funded as a project and built what it needed; nobody was funded to build the shared part.

The bill arrives in three forms. First, the invoice: finance cannot say which team or use case spent the money. Second, a provider retires a model and every team patches separately. Third, an auditor asks "where does customer data leave the building?" and the honest answer is "in several places, and we would need to ask around".

If you worked through Lesson 14 on AI FinOps and token economics, you can already price one use case and put a ceiling on anything that can loop. This lesson deals with the lines nobody claims, and the shared plumbing that would have made them claimable.

What does duplicated AI plumbing cost?

No published, audited figure exists for the cost of duplication or the savings from fixing it; be suspicious of anyone who quotes one. The surveys do show the conditions for duplication getting worse.

  • More teams are building. In McKinsey's State of AI 2026 survey (published 25 August 2026, 1,719 respondents, self-reported), 32% said they had skipped buying a software product because they could build it with agentic coding tools.
  • Running costs already bite. In the same McKinsey survey, about 20% of respondents said AI operating costs, including tokens, had constrained their use of AI, and 28% said they spend more than 10% of their ICT budget on AI.
  • Most of the money is outside IT's view. BCG's Applied AI Index 2026 (press release, 30 September 2026, self-reported survey) found AI spending rose from about 1.7% of revenue in late 2025 to 3.3% in 2026, and that more than 80% of AI spending sits outside the enterprise IT budget.

Put those together: spending roughly doubled as a share of revenue, mostly in budgets no single team sees end to end, while a third of organizations build what they once bought. That is a recipe for many small, private AI stacks.

How Uber put one front door in front of its models

The clearest public account of the alternative comes from Uber. On 11 July 2024 the Uber Engineering blog described its GenAI Gateway: one access point that sits in front of OpenAI, Vertex AI and models Uber hosts itself.

The first design choice is the one worth copying. Uber's gateway mirrors the OpenAI HTTP and JSON interface. A team calls the gateway exactly as it would call the provider, reusing client code it already knows, and Uber avoids a home-grown interface that goes stale whenever providers add a feature. The platform team did not ask anyone to learn something new. They asked them to change an address.

Behind that one endpoint sit the components every team would otherwise rebuild: authentication and authorization; PII redaction, which removes personal data before a request leaves for a third-party vendor and restores the values in the response; audit logs; metrics and alerting; and central usage and budget management.

Notice what the audit logs do. Uber uses the same log for three jobs: cost attribution, security audit and quality evaluation. One record answers the finance partner, the CISO and the AI lead, and answers "Which of these are yours?" in seconds.

Access is not automatic either: Uber's Engineering Security team reviews each use case against the company's data-handling standard before granting gateway access. At the time of publication, Uber reported that the gateway served more than 60 LLM use cases for close to 30 customer teams, handling 16 million queries a month with a peak of 25 queries per second. These are Uber's own figures.

Two honest notes. Uber states plainly that its PII redactor hurts both latency and answer quality, without saying how much, and it publishes no number for money saved: the post describes architecture, not return on investment. You will close both gaps yourself: measure your guardrails, and build your own baseline.

A platform is a short list of boring, shared services behind a familiar interface, plus one review step. The boring part is the point.

How an Internal AI Platform Works: the Socket, the Meter and the Fuse Box

Think about an office building's wiring. Nobody on the fourth floor installs a generator. Every desk has the same socket, every floor a meter, and somewhere a fuse box trips before anything burns.

Tenants choose what to plug in. The building decides how power is delivered, how it is measured and how it is cut off. Nobody finds that oppressive; it is why a new tenant can work on day one.

Now and then someone runs a cable out of the window to the street because the socket was in the wrong place. It works, and it is unmetered, unfused and invisible until something goes wrong. In AI terms that cable is a team calling a model provider directly with its own key. Call it a shadow integration.

What is an AI gateway?

An AI gateway is the single front door through which all model calls pass, so that identity, data protection, logging and budgets are applied in one place. You will also hear it called an LLM gateway. In the building, it is the socket, the meter and the fuse box together.

  • The socket is the standard interface. Every application plugs in the same way, whichever model is behind the wall.
  • The meter is the log. Every call is recorded against the team and the use case that made it.
  • The fuse box is the set of controls: authentication, redaction of personal data, and a budget that stops a runaway job before it becomes a five-figure surprise.

If you already run an API gateway, you will ask how this differs. An API gateway routes and secures requests by caller and endpoint. An AI gateway also understands what is inside a model call: it counts tokens and prices them per team, redacts personal data in the prompt and restores it in the response, routes the same request to a different provider or model, and keeps prompts and answers for evaluation. Many organizations run it beside or on top of the API gateway they already operate.

What should an internal AI platform include?

An internal AI platform usually has five parts. People often call it the "paved road for AI": the easy, well-lit route that teams take because it is faster than cutting their own path through the woods. Across published cases, enterprise AI platform architecture converges on the same five reusable AI components.

  1. A gateway. A provider-compatible front door with identity, PII redaction, logging and budgets, as in the Uber example.
  2. Shared retrieval and data connections. Retrieval is the step where an AI system looks up your documents before answering. Shared connections carry lineage, a record of where each piece of data came from, so nobody wires a source of unknown origin into an assistant.
  3. A registry of tools and agents. A catalog of the actions AI systems may take and the agents that exist, each with an owner. LinkedIn's engineering team described on 10 September 2025 how agents on its platform are registered in a central skill registry, ask for clarification, feedback or approval at key decision points, and are moving toward MCP and A2A, open protocols for connecting agents to tools and to other agents. LinkedIn publishes no usage or performance numbers, so read it as design, not proof of results.
  4. An evaluation harness. A shared test rig: sample questions with known good answers, run against each use case on a schedule. An OpenAI customer story on Morgan Stanley (undated, a vendor's account) describes every use case being tested with evals graded by advisors and prompt engineers before deployment, plus a regression suite, a set of tests rerun to catch what has quietly stopped working, run daily. If each team must invent its own inspection, most will not.
  5. An intake review. One route by which a new use case gets checked and admitted.

The common mistake is to treat the gateway as the whole platform. Model access is the easy part. If identity, redaction, logging, evaluation and intake are left to each team, you have centralized the keys and nothing else.

Should an AI gateway be model-agnostic?

Yes. A model-agnostic gateway is one where the application does not know or care which provider answers, so you can swap a model without rewriting the applications that use it. In the building, it is the reason you can change electricity supplier without rewiring every desk.

JPMorgan Chase gave exactly this reason for building its LLM Suite in-house as an abstraction layer. American Banker reported on 22 May 2025 that the tool had 200,000 users as of mid-May 2025, out of a workforce of about 250,000, on a self-service basis. Chief Analytics Officer Derek Waldron put the principle in one line: "we don't want to architect ourselves around one particular model". New models are vetted before they are added, and each data source is confirmed as authoritative before it is connected. The bank's reported gain of one to two hours per person per week is self-reported and "early stages", so keep it out of your business case.

Model-agnostic is not free: you now own an abstraction layer and must keep it current. That price belongs in the platform team's funding.

What does it mean to run an AI platform as a product?

It means the platform has customers (internal teams), a promised service level, a price, a roadmap and a lifecycle, and the AI platform team is judged on adoption, not components built. Building the gateway is a project; running it is where the team's time goes after launch. Four disciplines matter most.

Reliability targets. The fuse box has a catch: once every model call passes through one gateway, the gateway is a single point of failure. Give it its own service level objective, a written target for availability and speed, as you would for payments or identity. It needs an availability target, a latency budget (the milliseconds the gateway itself may add to each call), and a failover route per use case (a pre-approved second model or provider used when the first is unavailable). Decide one more rule in advance: if the redaction service fails, calls carrying personal data fail closed, meaning they are refused rather than sent unredacted. An outage is embarrassing; a leak is worse.

A funding model. There are three options. Central funding treats the platform as a shared cost: simple and good for adoption, but nobody sees a price. Showback reports each team's cost without billing it, which builds trust in the tags. Chargeback bills each team per call or per token, usually at cost plus a share of overhead. A sensible path is central funding with showback in year one, then chargeback once nearly all spend is attributed and teams trust the numbers. One rule protects adoption: never price the paved road above what a team would pay going direct, or you are paying teams to leave the road.

Model retirements, handled once. Providers retire models on their own schedule. Anthropic's model deprecations page (as read on 13 November 2026), for example, promises customers with active deployments at least 60 days' notice before a publicly released model is retired, and says requests to a model past its retirement date will fail. With a platform, the platform team owns the routine instead of every team: the gateway log shows within the hour which use cases call the model, the shared harness runs each one's regression suite against the replacement, and the switch is a configuration change at the gateway on a published date. Lesson 25 on avoiding model lock-in turns this into a full switch drill; the platform makes that drill cheap.

Intake that scales. One review queue works at four requests a quarter; at forty it becomes the gate teams route around. The answer is tiers: a fast lane for reuse of an already approved pattern (same data class, approved model, no write actions), a standard review for a new data source or model, and a full review for personal data at scale, customer-facing decisions or agents that act. Each tier gets its own clock, and the review team plans its capacity in hours per quarter, as a service desk plans for ticket volume.

Does the gateway log help with the EU AI Act?

Yes, if you design it to. Article 26(6) of the EU AI Act requires deployers of high-risk systems to keep the logs those systems generate automatically, to the extent they control them, for an appropriate period of at least six months unless other law says otherwise. Under the Digital Omnibus on AI, Regulation (EU) 2026/1744 (published in the Official Journal on 24 July 2026, in force 27 July 2026), the high-risk duties for Annex III systems apply from 2 December 2027. A gateway log with a use case ID on every call and a written retention period is where much of that evidence will come from. Confirm the articles and dates against the Official Journal text.

Intake does the same for the register: every use case it admits creates or updates an entry in the model inventory you built in Lesson 15 on AI governance and model inventory, so the inventory stays a living register instead of a spreadsheet updated once a year.

The Workshop: How to Build an Internal AI Platform in a Paved Road Design Session

You are now at Level 4 of the ladder, AI Scaled, and this is its first lesson. Lesson 15 gave you a model inventory and a governance routine, and Lesson 14 gave you cost tracking per use case. The next lesson, AI Agents in Real Workflows, builds on what you produce today, because the gateway and the registry become the place where you control what agents may do.

The method runs as two sessions of about three hours, a week apart, with a day of preparation before the first. Session one decides what the platform is (steps 1 to 3). Session two decides how it is run (steps 4 to 7). The people who know where the keys are rarely know how the money and contracts work.

Who is in the room for session one: the head of AI or the platform lead as chair; one engineer from each team already running AI in production or pilot; a security architect; and the data protection lead. Invite engineers, not their managers.

Who joins for session two: the finance partner who forwarded the invoice; the service owner or a site reliability engineer for your existing API gateway, who knows what an availability target costs to keep; and the vendor manager who holds the model provider contracts and their notice terms.

What you bring: the Lesson 15 inventory, the last two AI invoices, the Lesson 14 cost tracking, an architecture sketch per team (a whiteboard photo is fine), and last quarter's count of AI requests to security and data protection.

The seven steps

  1. Map every system, its plumbing and its invoice line. For each production and pilot AI system, write down six things: how it reaches a model, where the keys are held, how personal data is handled, what is logged, how it is evaluated, and which invoice line it lands on. Put it on one wall, mark every component built more than once, and ask each engineer for the engineer-weeks theirs took. No external benchmark exists, so that count is your only baseline.
  2. Agree version one of the five parts. For each part, write one sentence for what version one does and one sentence for what it will not do yet. The second sentence keeps the platform from becoming a year-long build.
  3. Write the gateway contract and its service targets. Decide the interface (compatible with a provider interface your teams already use), how callers authenticate, what is redacted, and what every log record contains: at minimum caller, use case ID, model and version, tokens, cost, latency and whether redaction was applied. Then set the availability target, the latency budget the gateway may add, a failover route per use case, the fail-closed rule for redaction, and the log retention period.
  4. Choose the funding model and the tagging rule. Every call carries a team tag and a use case tag, and untagged calls are rejected. It sounds harsh for about a week; then finance can read the bill. Then pick central funding, showback or chargeback, and name who approves a price change and how much notice teams get before the next budget cycle.
  5. Design a tiered intake with a clock and a capacity plan. Write each tier's questions, who answers, and its service-level target. Multiply last quarter's request count by the hours each tier takes, and check that the reviewers actually have those hours. A review with no clock becomes a gate people avoid; a clock with no capacity behind it is a broken promise.
  6. Write the model retirement routine. Name who watches provider notices, how the gateway log finds affected use cases, who runs regression suites against the replacement, the migration window and the default switch date. Ask the vendor manager what notice each contract actually gives.
  7. Pick two teams to migrate first, publish, and set a 30-day review. Choose one easy and one demanding team, each with a date. Publish a one-page service catalog and an onboarding path a new team can follow without a meeting. Choose three platform metrics, and at the 30-day review measure what your guardrails cost in latency and answer quality: Uber's redactor has a cost, and so will yours.

Worked example: an insurance broker with four AI systems

This example is illustrative, not a real company. Picture a 2,400-person insurance broker with four AI systems live: claims triage, a policy question assistant, a marketing drafting tool and a developer assistant. Each was built by a different team.

Step 1 fills the first two columns below; steps 2 to 7 fill the rest.

ComponentBuilt how many times todayPaved-road version oneOwnerFirst adopter
Model access and keys4One gateway, provider-compatible interface, keys held centrally, failover model per use casePlatform teamPolicy assistant
PII redaction3 (one team has none)Shared redaction service at the gateway, fails closed, effect on quality measured monthlySecurityClaims triage
Logging and cost tags4 formatsOne log schema, mandatory team and use case tags, showback report to each teamPlatform and financeAll
Evaluation2 ad hoc scriptsShared harness, daily regression run per use caseAI leadPolicy assistant
Intake reviewPer project, no clockThree tiers, each with its own clockSecurity and data protectionNew use cases

Read the second column slowly: three different opinions about what counts as personal data, one team sending prompts with no redaction at all, and two of the four systems not tested on any schedule.

Now look at what the broker decides. The policy assistant is the easy first adopter: read-only, modest traffic, a team that wants help with evaluation. Claims triage is the demanding one: it handles the most personal data, so it is the real test of the shared redaction service and its fail-closed rule. And version one explicitly excludes shared retrieval, which waits a quarter. The room wanted it; the chair wrote it in the "will not do yet" column anyway, because a gateway that two teams use in six weeks beats a full platform nobody uses in a year.

In session two the finance partner picks central funding with showback for year one; any move to chargeback needs the platform owner and finance partner to agree, with one budget cycle of notice. The two provider contracts give different retirement notice, so the routine assumes the shorter. Then the room sizes intake from last quarter's 36 requests:

Intake tierWhat qualifiesAnswer withinExpected requests per quarterReviewer hours eachHours per quarter
Fast laneReuse of an approved pattern: approved model, same data class, no write actions2 working days20120
StandardNew data source or new model, internal users only5 working days12448
Full reviewPersonal data at scale, customer-facing decisions, agents that act, possible high-risk use15 working days41664

That is 132 reviewer hours a quarter, about ten a week across security and data protection. Saying the number out loud gets those hours into two people's objectives. And more than half of the requests take the fast lane only because the gateway, redaction service and harness are already approved: reuse is what makes review fast.

Six months on: approved on Wednesday, live the next week

Two quarters later (still illustrative), the renewals team files a fast-lane request on a Monday, is approved on Wednesday, and makes its first production call the next week without a meeting. A provider announces a model retirement; within an hour the platform team knows from the log that two use cases depend on it, the harness runs both regression suites that week, one passes and is switched at the gateway, and the other gets a dated fix plan with its owner. When the invoice arrives, the unallocated line is zero, and nobody asks whose it is.

The artifact: AI Platform Service Catalog and Paved Road Charter

You leave with one document, the AI Platform Service Catalog and Paved Road Charter. The catalog tells teams what they can use; the charter records what the platform promises and who pays. Copy this template.

FieldWhat to write
Platform ownerA named person accountable for the platform as a product. If this is blank, stop here.
Funding modelCentral funding, showback, or chargeback per call; the budget line; who approves price changes and with how much notice.
The five componentsFor each: version-one scope in one sentence, exclusions in one sentence.
Gateway contractInterface, authentication, redaction rules, log schema, log retention period, budget rules.
Service targetsAvailability target, latency budget per call, failover route per use case, fail-closed rule, date the failover was last tested.
Intake tiersFor each tier: what qualifies, the questions, who answers, the clock, and reviewer hours per quarter.
Model retirement routineWho watches notices, how affected use cases are found, regression run, migration window, default switch date.
ExceptionsWho may approve a direct provider key, for how long, and the end date by which it moves onto the gateway.
Migration list and onboardingTeams in order, with dates, first two named; what a new team does, in order, without a meeting.
Platform metrics and review dateThree metrics and the date of the 30-day review.
Known trade-offsFor example, the cost of redaction to latency and answer quality.

Most teams leave out the last row. Write the trade-offs down: engineers trust a platform that admits its costs.

Failure Modes, How to Spot Them, and the Metrics to Track

Platforms fail in a few predictable ways, each with a cheap test you can run before the 30-day review.

How does an AI platform go wrong?

Failure modeWhat it looks likeHow to spot it
Gatekeeper platformIntake answers take weeks. Teams ask for exceptions or quietly keep their own keys.Count the provider keys that exist outside the gateway, and the exceptions past their end date.
Ivory tower platformMonths of build and no onboarded teams.The roadmap has more components than customers.
Gateway as single point of failureAll AI features fail together, or gateway latency eats the answer budget.Check whether the gateway has its own availability target, a latency budget per call, and a failover route that has actually been tested.
Retirement scrambleA provider notice arrives, teams search their own code, and a model swap needs changes in the applications.List every use case on one model from the gateway log alone, then switch one in a test and watch what breaks.
Unmeasured guardrailsRedaction or filters quietly make answers worse.Run the evaluation set with the guardrail and without it.
Untagged spendFinance cannot allocate part of the bill.A growing "unallocated" line on the AI invoice.
Road priced above the woodsChargeback makes the gateway dearer than calling a provider directly, so teams leave.Compare a team's charged price per call with the provider's list price for the same call.
Intake queue overloadEvery request gets the full review, the clock is missed, and the fast lane is empty.Share of requests in each tier, and missed clocks per quarter.
Registry rotThe registry fills with tools and agents nobody owns or uses.Count registry entries with no owner, or no call in the gateway log for 90 days.

The first two are opposites, and most organizations lurch between them: a committee that says no, so teams route around it, or a grand design built for a year before anyone uses it. The cure for both is the same: a small version one, two real customers and a clock on the review.

Your infrastructure lead will ask about the third first, and rightly. Centralizing every model call concentrates risk as well as control, so treat the gateway as tier-one infrastructure and test the failover route before the outage, not during it. If you are an EU financial entity, this is not only good practice: since 17 January 2025, Article 28(8) of DORA, Regulation (EU) 2022/2554, has required documented, periodically tested exit strategies, with identified alternatives, for ICT services supporting critical or important functions. A tested failover route at the gateway is part of the evidence behind that plan.

One note for the registry: as teams connect AI systems to tools through MCP, the Cloud Security Alliance (May 2026) recommends putting an MCP gateway with policy enforcement and logging in front of those connections. Same idea, applied to tools instead of models; Lesson 17 on agent tools, MCP and permissions picks it up.

Which metrics show the platform is working?

Pick three to start and add the rest when you have the data.

  • Teams onboarded and use cases served. Uber reports both.
  • Gateway availability against target, and the latency it adds at the 95th percentile. The two numbers your service targets promise.
  • Share of AI spend attributed to a team and a use case. This is AI platform cost attribution: aim for nearly all of it before chargeback.
  • Daily evaluation regression pass rate. The early warning that a model or guardrail change hurt quality.
  • Median time from intake request to first production call, per tier. Whether the road is paved or merely signposted.
  • Days from a retirement notice to the last affected use case migrated. Compare it with the notice period in the contract.
  • Count of duplicate components retired. This closes the loop with the baseline you counted in step 1.

That last metric is your business case: with no audited savings figure published, yours is your own before and after in duplicate builds, review hours and time to production.

Frequently Asked Questions

What is the difference between an AI gateway and an API gateway?

An API gateway routes and secures requests by caller and endpoint; an AI gateway also understands the model call itself. It counts and prices tokens per team, redacts personal data in prompts, switches providers behind one interface and keeps prompts and answers for evaluation, and it often runs beside or on top of the API gateway you already have.

How much money does an internal AI platform save?

No audited public figure exists. The published cases, including Uber's GenAI Gateway (Uber Engineering, July 2024), describe architecture, not savings, so measure your own baseline of duplicate builds, review hours and time to production before and after.

Can a team keep calling a model provider directly?

Only as a written, time-boxed exception with a named approver and an end date by which it moves onto the gateway. A direct key is a shadow integration: unmetered, outside shared redaction and invisible to the log, so count the keys outside the gateway as a standing platform metric.

What happens if the AI gateway goes down?

If you have not planned for it, every AI feature fails at once, because the gateway is a single point of failure. Plan for it with an availability target, a latency budget, a tested failover route per use case, and a rule that calls carrying personal data fail closed if redaction is unavailable.

Do gateway guardrails slow down or degrade answers?

They can do both. Uber reports that its PII redactor hurts both latency and answer quality, so treat each guardrail as a component to test with your evaluation set, with the guardrail on and off.

Key Takeaways

  • An internal AI platform is a short list of reusable AI components built once: a gateway, shared retrieval and data connections, a registry of tools and agents, an evaluation harness and an intake review.
  • The AI gateway is the socket, the meter and the fuse box. Uber's one audit log serves cost attribution, security audit and quality evaluation.
  • Mirror an interface teams already know and stay model-agnostic, so a model retirement is one configuration change, not a scramble in every team.
  • Run the platform as a product, with an availability target, a latency budget and a tested failover route: the gateway is a single point of failure.
  • Reject untagged calls, start with central funding and showback, and move to chargeback only when attribution is near complete and the road is priced no higher than going direct.
  • Tier the intake, give each tier a clock and planned reviewer hours, and let every admitted use case update the model inventory.
  • No audited savings figure exists: count your own duplicate builds, measure each guardrail, and record both in the Paved Road Charter.