←
AI-Ready Foundation
Strategic · M2 · lesson 2 of 5 · queued
Preview — browse every lesson free. Enroll to mark lessons complete, open partner links and save your progress. Login & enroll →
AI Observability: Tracing, Continuous Evaluation, and Incident Response for AI
📖
now learning

AI Observability: Tracing, Continuous Evaluation, and Incident Response for AI

15 min

AI observability is the ability to see, for any single request, what went in, what the system did step by step, what came out, and whether the answer was any good. Most organizations running AI in production have the first three at best and the fourth not at all, which is why their wrong answers are found by customers with screenshots. This lesson gives you a one-day workshop that ends with a trace record you can replay, live quality signals, a weekly expert review, and a rehearsed AI incident runbook for one production system.

The Problem: A Support Bot Called Sam, and Who Found Out First

In April 2025, users of Cursor, the AI code editor made by Anysphere, kept getting logged out when they switched between machines. One of them emailed support. A reply signed "Sam" came back: this was expected behaviour under a policy of one device per subscription. It sounded official. There was no such policy. "Sam" was an AI support bot, and it had made the rule up.

The exchange went up on Reddit and Hacker News and spread fast. Some users posted that they had cancelled. Co-founder Michael Truell replied in public, "Hey! We have no such policy," and called it "an incorrect response from a front-line AI support bot", as The Register reported on 18 April 2025. The logouts, he said, came from a race condition on slow connections, since fixed. The affected user was refunded, and the company said AI responses in email support were now clearly labelled as AI. The correction came about three hours later, according to CX Network on 24 April 2025.

Three hours is quick for a company and an eternity on the internet. Now replay it from the inside, because this is where your own week would go. Picture yourself as the head of support at a firm like that (an illustration, not a report of what Cursor did internally). Before you can apologise properly you need three answers. How many other customers did the bot tell the same thing? What was it given that made the invented policy sound plausible? And has it stopped? If your logs hold only the question and the reply, you cannot answer any of them. You can only post an apology and hope.

That is the expensive problem. It belongs to the head of AI engineering or platform, shared with the service owner of each customer-facing AI system, the customer service lead, and the legal or compliance lead.

Who usually detects an AI incident first?

The customer does. That is the pattern across the public cases, and it is the single most useful fact in this lesson. Note who did the detecting at Cursor: a user, in public.

You met the other famous case in Lesson 8 on building a golden test set: in Moffatt v. Air Canada (BC Civil Resolution Tribunal, 14 February 2024) the airline was ordered to pay C$812.02 after its chatbot gave wrong fare information, and its lesson for observability is the tribunal's quieter point that the company could not say why its web page was more reliable than its bot. The production database deleted by a coding agent, which anchors Lesson 17 on agent permissions and human approval, adds one more: the agent's own account of what it had done was wrong. An AI system's explanation of itself is not telemetry. It is more generated text.

What does a lack of AI observability cost?

Honestly: nobody has a reliable figure, and you should distrust anyone who quotes one. The research behind this course found no independent, quantified study of what AI incidents cost or what observability saves. The "cost of a hallucination" numbers you see online are mostly vendor marketing. In the public cases the direct damages were small. What was spent was trust, and what was established was ownership of the output.

What we do have is evidence of how exposed most organizations are:

  • In McKinsey's "The state of AI" survey, published 12 March 2025 (fielded July 2024, 1,491 respondents), 27% said employees review all generative AI content before it is used. A similar share said 20% or less is checked. Fewer than one in five said they track well-defined KPIs for their generative AI solutions. These are self-reported answers, so treat them as a rough picture.
  • The Cloud Security Alliance reported in May 2026 that only 28% of organizations can trace agent actions to a responsible human or system in all environments. Put plainly, most could not run a proper investigation after an incident.

Why do capable teams get this wrong?

Not through carelessness. They do what worked for ordinary software, and it does not transfer.

  • They monitor the service, not the answer. Uptime, latency and error rate all look perfect when a wrong answer is delivered quickly. Cursor's bot replied politely and sounded official.
  • Evaluation is a gate, not a watch on real traffic. The test set checks the questions you thought of. Customers ask the others.
  • The logs are missing the two things you need. Without the retrieved context and the model version, a bad answer cannot be reconstructed.
  • Nobody compares the bot with the official page.
  • The runbook stops at rollback. The system gets fixed, and the customers who were told something false are never told the truth.
  • Guardrails are assumed to be safe changes. They are not. Uber's engineering team wrote on 11 July 2024, in its post on its GenAI gateway, that the personal-data redactor creates challenges for both answer quality and latency. A safety feature can cause a regression like any other change.

How AI Observability Works: The Smoke Alarm, the Fire Drill and the Panel Log

Think about how a well-run office building deals with fire. It has three separate things, and each does a different job.

A smoke alarm tells you early, before anyone smells anything. A fire drill means everyone already knows which stairs to take, so nobody calls a meeting in a burning corridor. And the alarm panel's log tells the investigator afterwards which detector tripped first and when, so the cause can be found and fixed.

Most AI systems have none of the three. They have a thermometer in the lobby that reports the building is a comfortable temperature. That thermometer is your uptime dashboard. It is accurate, and it is answering a question nobody asked.

Keep the three in mind for the rest of the lesson: alerts are the smoke alarm, the runbook is the drill, traces are the panel log. Now the proper terms, one at a time.

What is LLM tracing?

LLM tracing means recording, for one request, every step the system took. The record is called a trace: the prompt as actually sent, the context that was retrieved, each tool call, the model and its version, and the cost and latency. Each step inside it is a span. OpenTelemetry is an open standard for recording those steps, so traces from different systems land in the same shape.

There is one test for whether your trace is good enough, and it is not a checklist of fields. Pick one real conversation from last week and try to replay it: show exactly what the model was given and what it returned. If you can, you have a trace. If you can only show the question and the answer, you have a chat transcript, which tells you what happened but never why.

What do Lessons 8 and 11 already give you, and what is new here?

Lessons 8 and 11 test the system before and around a change; this lesson watches what real customers are actually told and handles the day a wrong answer gets out. Keep the boundary clear so you do not build things twice.

  • Lesson 8 gave you the golden test set: real questions with expert-approved answers. You reuse it here; you do not rebuild it.
  • Lesson 11 on shipping and rolling back AI in production made that set a release gate, put it on a scheduled watch, and gave you a rollback runbook, limits and a kill switch held by a named person and a deputy.
  • This lesson adds four things: a per-request trace that explains one specific answer, quality signals from live traffic, a weekly expert review of real conversations, and an incident runbook for harm that has already reached customers.

Think of a restaurant. Lesson 11 is the head chef tasting each batch before service. This lesson is what happens when a diner has already eaten the dish: you need to know which table got it, what went into it, and who tells the other tables.

What is continuous evaluation for AI?

Continuous evaluation for AI means checking quality all the time, not once at launch: the golden test set on every change and on a schedule, plus checks on a sample of live answers. The live checks are the new part. Groundedness is whether an answer is supported by the source material the system was given; scoring it on a sample of real answers is the practical form of hallucination monitoring, because an ungrounded answer is where invented policies live.

What counts as an AI incident?

An AI incident is any case where the system's output or action caused harm, or could have. The "could have" matters. A wrong tariff quoted to a customer who did not act on it is still an incident, because the next customer will.

What is the minimum you need in place?

Six parts. This list is this course's synthesis of published practice, not an industry standard, so use it as a floor and adapt it.

PartWhat it isFire-safety equivalent
1. A trace per requestPrompt, retrieved context, tool calls, model and version, cost, latencyPanel log
2. Online quality signalsUser feedback, escalation rate, groundedness checks on a sample of live answersSmoke alarm sensors
3. Golden test set regressionThe Lesson 8 set, run on every change (Lesson 11's gate) and dailyWeekly alarm test
4. Expert reviewSubject experts grade a sampled slice of real conversationsThe inspector's walk-round
5. AlertsOn drift, cost spikes, refusal rates and error ratesThe alarm itself
6. Incident runbookConfirm from the trace, contain, correct publicly, find root cause from traces, add the case to the regression setFire drill

What does good practice look like in the wild?

  • Traces that feed evaluation. LinkedIn described on 10 September 2025 how it traces during development, records privacy-safe OpenTelemetry spans in production, and feeds those traces into a shared evaluation platform for offline evaluations, regression tests and prompt tuning. Production failures become tomorrow's tests. The article publishes no performance figures, so take it as a design, not a result.
  • Experts grade, and the suite runs daily. An OpenAI customer story on Morgan Stanley (undated, and a vendor source, so read it with that in mind) describes evaluations graded by domain experts before deployment and a regression suite of sample questions run daily.
  • One log, three uses. Uber's gateway post of 11 July 2024 describes audit logs used for cost attribution, security audit and quality evaluation at once. You do not need three logging systems. You need one good record.

Where does the EU AI Act come in?

A typical customer service chatbot is not on the EU AI Act's Annex III high-risk list, but if your system is, logging and reporting duties are coming. Under the EU AI Act as amended by the Digital Omnibus on AI (Regulation (EU) 2026/1744, in force 27 July 2026), duties for Annex III high-risk systems apply from 2 December 2027. Two deployer duties in Article 26, which the Omnibus left unchanged, map straight onto this lesson. Article 26(6): keep the logs the system generates automatically, as far as they are under your control, for a period appropriate to its purpose and of at least six months unless other law, notably data protection law, says otherwise. Article 26(5): monitor the system's operation and, on identifying a serious incident, immediately inform first the provider and then the importer or distributor and the market surveillance authorities. Article 73 gives providers deadlines of no later than 15 days after becoming aware, 2 days for a widespread infringement or a serious and irreversible disruption of critical infrastructure, and 10 days where a person has died.

Confirm the articles and dates against the Official Journal text with your counsel, and design trace retention to meet them now; a trace schema is much harder to change after a year of traffic. If your system is an agent, also run your trace fields past the OWASP Top 10 for Agentic Applications for 2026 (December 2025), which runs from ASI01 Agent Goal Hijack to ASI10 Rogue Agents. Risks such as ASI02 Tool Misuse and Exploitation and ASI08 Cascading Failures are only visible if every tool call is a span.

Where does this sit on the L1 to L5 ladder?

This is Level 4, AI Scaled, lesson 3 of 5. In Lesson 16 you built the shared platform and its gateway log schema. In the previous lesson, AI Agents in Real Workflows, you gave each agent a permission matrix and were told to trace every step. This lesson turns those traces into a working system. The next lesson, Measuring AI ROI Honestly, takes the traces, cost per task and quality scores you produce here and uses them to answer the CFO's question: which of these systems is paying for itself?

The Workshop: The Observability and Runbook Build

You will do this for one production system, in one day. Pick the customer-facing one that would embarrass you most. Resist the urge to design an enterprise framework first. One system done properly teaches you more than a policy for twenty.

Who needs to be in the room, and what do you bring?

  • Morning (trace and signals): the head of AI platform (chair), the service owner, one platform engineer, two domain experts who know what correct answers look like, and the privacy or data protection lead for the retention decision.
  • Afternoon (runbook and drill): the service owner, the customer service lead who will own customer contact, the communications lead, legal or compliance, and for any agent with write tools, the security lead.
  • Inputs: the gateway log schema from Lesson 16, the Golden Test Set from Lesson 8, the Production Release and Rollback Runbook from Lesson 11, the permission matrix from Lesson 17 if the system is an agent, your complaint-handling window, and the authoritative sources the system is supposed to answer from.

The domain experts are not optional. An engineer can tell you whether the system responded. Only a billing specialist can tell you whether the response was right.

What are the eight steps?

  1. Define the trace record and replay one real request. List the fields: prompt, retrieved passages (their text or a version reference, not just IDs), tool calls, model and version, cost, latency, a conversation ID. Then pick one real request and reconstruct it end to end. If a field is missing, you find out now. Decide retention and personal data handling (see below).
  2. Name one authoritative source per topic, then compare. List the topics the system answers (refunds, tariffs, opening hours) and the single source that is correct by definition for each. Ask the system 20 questions and check each answer against that source. Where two sources disagree, retire one.
  3. Promote your golden test set from Lesson 8 into a scheduled job. If you skipped Lesson 8, build one now: 30 to 50 real questions with expert-approved answers, then grow it from every incident and failed review case. Lesson 11 made this set a release gate. Today you confirm the daily run fires even when nobody shipped, so changes you did not make are caught too, and you make every failing case open as a trace.
  4. Choose two or three live signals and their alert thresholds. For example: escalation to a human, thumbs-down rate, and groundedness on a sample. Two or three, not twelve. Set each alert relative to that signal's normal range, and route it to a named person.
  5. Set up weekly expert review. A sampled slice of real conversations goes to the domain experts each week with a simple grading sheet. The golden test set only checks the questions you thought of. Review finds the ones you did not.
  6. Write the incident runbook and its severity levels. Who is paged. How the trace confirms what was said and to whom. How the system is contained: the Lesson 11 kill switch or a fall-back to a human queue. Who contacts affected customers and who approves the public correction. How root cause is found from traces, never from asking the model. Whether the system is high-risk, and so whether the provider must be told.
  7. Run a tabletop drill on the Cursor scenario. "A customer has posted a thread saying our assistant told them about a policy we do not have." Walk through the runbook with a stopwatch on each move, including the query that finds every other customer who got the same answer.
  8. Close the loop. Every incident and every failed review case is added to the golden test set, through whoever owns it since Lesson 8. This is the step that makes the system get better instead of merely being watched.

The retention decision legal must sign

Keep traces at least as long as your complaint-handling window, because a trace you deleted before the complaint arrived cannot be replayed. For an Annex III high-risk system, that also means at least the six months in Article 26(6). Traces hold customer words, so redact or tokenize personal data at the gateway before the trace is stored, keep the key that reverses the tokens under separate access, and write the retention period into the plan. A trace store nobody may open is as useless as one that holds nothing.

What does a filled-in example look like?

This example is an illustration, not a real company. Picture a regional energy retailer whose customer assistant answers tariff and billing questions. Here is what its workshop decided.

ElementDecision in the workshopOwnerCheck
Trace recordPrompt, passage text and version, model version, tool calls, cost, latency, conversation ID; kept 12 months, account numbers tokenizedPlatform engineerOne request replayed in the session
Authoritative sourceThe tariff database is the single source; the old PDF guide is retiredService ownerBot and web page compared on 20 questions
Golden test setThe Lesson 8 set, grown to 80 questions since launch; already the release gate; daily run confirmedBilling leadEach failing case opens as a trace
Live signalsEscalation to human, thumbs-down, sampled groundednessAI platform leadAlert when a signal leaves its normal weekly range
Expert reviewA weekly sample of real conversationsBilling specialistsGraded on a three-point sheet
RunbookHuman queue within minutes; customer service lead contacts affected customers; correction approved by the service ownerService ownerDrill timed end to end

Three moments from this imagined day are worth describing, because they are the ones that tend to happen for real.

The replay fails the first time. At 9:40 the platform engineer tries to replay a request from the previous week and finds the log holds the IDs of the retrieved passages but not their text. The tariff database has been updated since, so the passages the model actually saw are gone. The fix, storing the passage text or a version reference with each trace, takes an afternoon. Finding it during an incident would have taken the incident.

The comparison finds disagreements. The team asks the assistant 20 questions and checks each answer against the public web page. On three of them the two disagree, all traced to the old PDF guide still sitting in the retrieval index. In this illustration the retailer finds its own two-sources-of-truth problem before any customer does, in about forty minutes.

The drill exposes the real gap. Switching the assistant to the human queue takes four minutes. Finding the other customers who got the same answer takes twelve, using the trace store. Agreeing who may publish the correction takes fifty, because nobody had ever decided. That is what a drill is for. The team writes a name into the runbook and the next drill is faster.

The three-point grading sheet the billing specialists use is deliberately plain:

GradeMeaningWhat happens next
CorrectRight answer, supported by the authoritative sourceNothing
Incomplete or unclearNot wrong, but a customer could be misled or left stuckLogged; added to the golden test set if the pattern repeats
WrongContradicts the authoritative source or invents somethingRaised as an incident; added to the golden test set the same week

What does the runbook's first hour look like?

The runbook has five moves, always in this order. The first is a quick confirmation from the trace, measured in minutes; containment follows at once. Finding the cause comes fourth, not first. Engineers want to diagnose; the customer needs you to stop the bleeding.

MoveQuestion it answersWho (fill in a name, not a team)
1. Confirm from the traceWhat was said, to whom, and how many others got it?On-call AI engineer
2. ContainIs the feature switched off or routed to a human?Service owner
3. Correct publiclyHave affected customers and the public record been corrected?Customer service lead with communications
4. Find root causeWhat do the traces show the system was given and did?AI engineering lead
5. Add to the regression setWill tomorrow's daily run catch this if it recurs?Domain expert

Severity sets the clock. Use three levels and write them into the plan:

SeverityWhat it looks likeResponse
Sev 1A wrong answer or action with a financial, legal or safety effect, or many customers affectedContain within 15 minutes; an executive is informed; check whether a serious incident must be reported
Sev 2A wrong answer to individual customers, no action taken on itContain the same day; contact the affected customers
Sev 3Incomplete or unclear answersHandled in the weekly review

Cursor's case would sit at Sev 1 under this scheme, because customers were cancelling. The 15-minute clock is the right instinct for that: a three-hour correction is the outcome of having no clock at all.

What artifact do you leave with?

You leave with an AI Observability Plan and Incident Runbook, one per production system. Copy this template and fill the right-hand column before the room empties.

FieldYour entry
System, owner, and high-risk status (Annex III: yes or no)
Trace schema, retention period, personal data handling
Authoritative sources, by topic
Golden test set (Lesson 8): location, owner, daily run confirmed
Link to the Lesson 11 release and rollback runbook
Live signals and alert thresholds, with named recipient
Expert review cadence and grading sheet
Incident severity levels and response clocks
Containment steps (kill switch, human fallback)
Customer contact owner and public correction approver
Serious incident reporting route (provider, authority)
Last drill date and timings

The test of the finished document: could someone who was not in the workshop use it at 2 a.m. to switch the system off, find every customer who got the wrong answer, tell them the truth, and work out what happened?

A year on: eight systems and one "who detected" chart

The first system takes a day. The second should take an afternoon, because the trace schema becomes a standard field set in the Lesson 16 gateway and every new system inherits it on day one. Picture the retailer a year on (still an illustration): eight AI systems on the gateway, each with a one-page plan, one shared trace store, and a monthly review where the only chart is "who detected" per system. In the first quarter, customers found most problems. By the fourth, the team's own alerts and reviews found most of them first. That chart is what scaled observability looks like.

Failure Modes, How to Spot Them, and the Metrics to Track

Each of these failures has a quick check you can run without buying anything.

Failure modeHow to spot itWhat to do
Green dashboards, wrong answersOnly uptime, latency and errors are on the boardAdd one quality signal next to them
Launch-day evalsThe golden test set has not grown since go-liveAdd every incident and failed review case (step 8)
Unreplayable incidentsA bad answer cannot be reproduced because context or model version was not loggedRun the replay test monthly
Two sources of truthThe bot and the official page differ on the same questionRepeat the 20-question comparison each quarter
Self-reported healthThe team relies on the model's own explanation of what happenedRoot cause comes from traces only
Fixed but never toldThe incident closed at rollback; no affected customer was contactedMake customer contact a runbook move with an owner
Alert noiseAlerts fire daily and have been muted; count how many were acted onCut to the two or three signals someone will act on

A word on alert noise, because it is the failure that arrives after you have done everything right. A smoke alarm that goes off every time someone makes toast gets its battery removed. If fewer than half of the last 30 days of alerts led to any action, you do not have too little monitoring. You have too much.

Which metrics should you track?

MetricWhat it tells you
Time to detect, and who detectedWhether you or a customer found it. In the public cases above, it was the customer.
Share of answers grounded in the named authoritative sourceWhether a two-sources-of-truth problem is waiting
Human-review coverage of outputsHow much of what the system says a person ever looks at
Eval pass rate per release and per daily runWhether a change, yours or a vendor's, made things worse
Cost per resolved taskWhat a good outcome costs, which Lesson 19 will need
Time from detection to public correctionHow long the wrong answer stood. Cursor's was about three hours.

If you track only one, track who detected. The day your own alert or review catches a wrong answer before any customer reports it is the day the system starts paying its way.

Frequently Asked Questions

How do you monitor an AI chatbot in production?

Monitor the answers, not just the service: trace every request, track two or three live quality signals such as escalation rate and sampled groundedness, run your golden test set daily, and have domain experts grade a sample of real conversations each week. Uptime and latency alone will stay green while the bot invents a policy.

How long should you keep LLM traces?

Keep them at least as long as your complaint-handling window, so any complaint can be replayed. For a high-risk system under the EU AI Act, deployers must keep automatically generated logs for at least six months from 2 December 2027, unless other law says otherwise.

Can you use OpenTelemetry for LLM applications?

Yes. OpenTelemetry records each step of a request as a span, and LinkedIn described in September 2025 recording privacy-safe OpenTelemetry spans for its AI systems in production and feeding them into evaluation. Put the prompt, retrieved context, tool calls and model version in those spans.

What should you do first when an AI system gives a customer a wrong answer?

Confirm what was said and to whom from the trace, then contain it within minutes by switching the feature off or falling back to a human. Public correction, root cause from traces and a new regression case follow, in that order.

Do you have to report AI incidents to a regulator?

Sometimes. For high-risk systems under the EU AI Act, from 2 December 2027 a deployer that identifies a serious incident must immediately inform the provider and then the market surveillance authorities; a typical customer service chatbot is not on the Annex III list, so check your system's classification with counsel.

Key Takeaways

  • AI observability means seeing what went in, what the system did, what came out and whether it was good. Uptime and latency answer none of those.
  • In the public cases, from Cursor's "Sam" to Air Canada, a customer found the wrong answer first. Measure who detected each problem; the goal is that it is you.
  • A trace passes one test: can you replay a real bad answer, including the retrieved context and the model version?
  • Reuse the Lesson 8 golden test set and the Lesson 11 gate; add what they cannot see: live quality signals and weekly expert review of real conversations.
  • Write the runbook and drill it: confirm from the trace, contain, correct publicly, find root cause from traces, add the case to the regression set, with severity levels setting the clock.
  • Design trace retention and incident reporting now for EU AI Act high-risk duties that apply from 2 December 2027, and confirm them against the Official Journal text.