LLM Evaluation: Building a Golden Test Set Before Hallucinations Reach Customers
On 14 February 2024 a tribunal in British Columbia ordered Air Canada to pay a grieving customer C$812.02 because its website chatbot had given him a fare rule that did not exist, and it dismissed the airline's argument that the chatbot was responsible for its own actions. LLM evaluation is how you keep your own assistant out of that hearing: it answers "is this good enough to put in front of customers?" with a number instead of a shrug. This lesson shows you how to build a golden test set, a small fixed list of real questions with expert-approved answers, and how to earn the right to let an automated judge grade it for you. You finish with an artifact you can start in a spreadsheet this week.
What Skipping LLM Evaluation Costs: The Chatbot That Was "Its Own Legal Entity"
Tribunal member Christopher C. Rivers had an unusual defence in front of him. Air Canada said it could not be held liable for what its own website chatbot had told a customer, because the chatbot was responsible for its own actions. He called this "a remarkable submission" and answered it in a single sentence: "It should be obvious to Air Canada that it is responsible for all the information on its website." The words are from the published decision, Moffatt v. Air Canada, 2024 BCCRT 149 (Civil Resolution Tribunal, 14 February 2024).
How did the airline get there? Jake Moffatt asked the chatbot about bereavement fares after a death in his family. It told him he could buy a ticket now and apply for the reduced fare afterwards, within 90 days of the date the ticket was issued. The airline's real policy, as the American Bar Association's Business Law Today (February 2024) summarises it, excluded bereavement requests made after travel, and when he claimed, Air Canada refused. Before the tribunal it argued that the correct information was elsewhere on its website, and that the chatbot was a separate legal entity.
The tribunal accepted neither. It found negligent misrepresentation: Air Canada "did not take reasonable care to ensure its chatbot was accurate", and the airline gave no reason why a customer should have to double-check information found in one part of its website against another part. It ordered Air Canada to pay C$812.02 within 14 days: C$650.88 in damages, C$36.14 in pre-judgment interest and C$125 in tribunal fees.
That is a small sum. Two things matter more than the money.
The first is the principle: you own what your AI says. One tribunal in one province does not settle the law everywhere, so take your own legal advice, but no sensible risk owner plans on the "the bot did it" defence working.
The second is what was missing. Somewhere in that airline, nobody had a test that asked: "Does the assistant state the bereavement policy correctly?" One question, with one approved answer, re-run before each release. That single line in a spreadsheet would have caught it.
How common are AI hallucinations in real work?
Common enough that courts have changed their rules. A hallucination is a fluent, confident statement from a language model that is simply not true. Three sourced examples show the pattern.
- Mata v. Avianca. In its opinion and order on sanctions of 22 June 2023 (published by Justia), a New York federal court imposed a $5,000 penalty on lawyers and their firm whose filing cited cases that did not exist. Asked by a lawyer whether one was real, ChatGPT "again responded" that it "does indeed exist". Judge P. Kevin Castel wrote of that fake decision: "Its legal analysis is gibberish." The UNC Law Library's June 2026 review confirms the date and citation.
- Deloitte Australia. The Register reported on 6 October 2025 that a AU$440,000 report for the Australian government cited works that do not exist and a fabricated quote from a Federal Court judgment. Deloitte agreed to refund the final installment; the department said the recommendations were unchanged. A university academic found the problem, not an internal review.
- The scale. The same UNC Law Library review (June 2026) reports that Damien Charlotin's database listed 1,058 U.S. cases of improper generative AI use in court filings at the start of June 2026. The Florida Supreme Court amended its rules in late May 2026 to require verification that cited authorities exist.
Buying a specialist product that answers from its own documents does not make the problem go away either. Lesson 7 on why naive RAG retrieval fails walks through Stanford HAI's May 2024 benchmark of legal research tools built exactly that way. The short version: "it is grounded in our documents" is a design choice, not a guarantee. You have to measure.
Why does this keep happening?
Picture it from the inside (an illustration, not a case). The business owner of your assistant asks whether it is ready for customers. The honest answer is "we tried a dozen questions and it looked fine". A prompt was changed the day before. Nobody knows whether that change made anything worse, because there is nothing to re-run.
That is judging by demo, and the arithmetic is against you. Anthropic's evaluation guide (9 January 2026) shows why: if a system succeeds 75% of the time on a single run, the chance that it passes three runs out of three is about 42%. One lucky run is a demo. A system that works three times in four will give you three clean runs in a row less often than a coin lands heads.
The other causes are just as ordinary. No expert has written down what "good" means, so engineers grade answers they cannot judge. Teams score on a 1-to-5 scale nobody can act on, hand grading to a second model without checking it, test only the questions the system should answer, and never read the outputs.
This lesson fixes each of those in turn. It is lesson 3 of 5 at Level 2 (AI Capable) on the L1 to L5 ladder. Lesson 6 on AI-ready data gave you a set of real questions and the data sources behind them. Lesson 7, building your first RAG system, gave you a working retrieval pipeline, a log of its answers and a tag on every failure. Here you turn that raw material into a test you can re-run forever. Lesson 9, on versioned prompts that survive model changes, then uses that test to manage prompts as versioned assets.
How a Golden Test Set Works: Think of a Driving Test
Before any jargon, think about how a country decides you are safe to drive.
The examiner does not ask the learner "do you think you drive well?" That is what the lawyers in Mata did when they asked the chatbot whether its cases were real. Nor does the examiner give 3.5 out of 5 for general impression. There is a fixed route with set manoeuvres. Some of them are things you must do, such as a hill start. Some are things you must not do, such as pulling out when a cyclist is coming. Each one is a pass or a fail, with a written note saying why.
And a new examiner does not test alone on day one. They sit in the back seat and shadow a senior examiner until their verdicts match. Only then are they trusted with the clipboard. Every piece of that picture has a counterpart in evaluation.
What is a golden test set?
A golden test set is a small, fixed list of real questions with expert-approved answers, re-run every time anything changes. It is the fixed route. An evaluation, usually shortened to "eval" (teams building on language models talk about "LLM evals"), is the act of running that list through your system and grading the results. It is the test drive.
Because the route never changes, results are comparable. "It looked fine" becomes "31 of 40 passed on Monday and 35 of 40 on Thursday", and a prompt change stops being an opinion.
The five words you need
Adopt Anthropic's small vocabulary as is, so engineers and business owner mean the same thing.
- A task is one test case: one question with its definition of success. One manoeuvre.
- A trial is one run of a task. Models vary from run to run, so you run several.
- A grader is whatever decides pass or fail. The examiner.
- A transcript is the full record of what happened in a trial: what was retrieved, what the model was sent, what it said.
- The outcome is the end result the user actually receives.
There are three kinds of grader, and you will use all three.
| Grader | What it is | Strength | Weakness |
|---|---|---|---|
| Code-based | A rule, such as "the answer contains the correct part number" | Fast, cheap, gives the same verdict every time | Brittle: a correct answer worded differently can fail |
| Model-based | A second language model, called an LLM judge, given grading instructions | Flexible, copes with open-ended answers | Not deterministic, and must be calibrated against humans |
| Human | A domain expert reading the answer | The gold standard | Slow and costly |
The rules that make it work, and why
Start small, and start from real failures. Anthropic's January 2026 guidance is that 20 to 50 simple tasks drawn from real failures is enough early on, because early changes have large effects. Draw them from the checks you already do by hand before a release, the bug tracker and the support queue. A question somebody invented in a meeting tests what you imagined. A question from a ticket tests what happened.
Pass or fail, with a critique. Hamel Husain (hamel.dev, published 29 October 2024, updated 1 September 2026) argues that 1-to-5 scores cannot be acted on, because nobody can say what separates a 3 from a 4. A binary verdict forces the real decision: would you be content for a customer to receive this? The critique, one or two sentences in the expert's own words, records why, and you will reuse it to build the judge.
Two experts should reach the same verdict. Anthropic's test for a good task is that two domain experts, working independently, would give the same pass or fail. If they cannot, the task is ambiguous: rewrite it rather than argue about the grade.
Test what must not happen. The driving test includes the cyclist. Anthropic's guide says to include cases where a behavior should not occur as well as cases where it should. For a customer assistant that means questions it must decline ("we do not hold that information") and questions it must hand to a person.
One must-happen behavior is a legal duty. If the assistant talks to people in the EU, Article 50(1) of the AI Act, Regulation (EU) 2024/1689, has applied since 2 August 2026: providers must design systems that interact directly with people so those people are informed they are interacting with an AI system, unless that is obvious from the context. The Digital Omnibus on AI (Regulation (EU) 2026/1744) did not move that date. Add a golden-set task that checks the disclosure appears at the first interaction, and confirm the duty, and whether you are provider or deployer, against the Official Journal text.
Grade the outcome, not the route taken. If the answer is right and supported, do not fail it because the system reached it in an unexpected way. Give an LLM judge an "Unknown" option so it is not forced to guess. And use one judge per dimension: a judge asked to check accuracy, tone and completeness in one pass does all three badly.
Read the transcripts. Anthropic's instruction is blunt: "Read the transcripts!" A pass rate tells you that something failed. Only the record tells you why.
What is faithfulness in LLM evaluation?
Faithfulness is the share of claims in an answer that the retrieved source material supports, on a scale from 0 to 1. It is the one component metric to learn by name at this level, as defined in the Ragas documentation.
Ragas gives its own example. Suppose the answer says "Einstein was born in Germany on 20 March 1879", and the retrieved context says he was born on 14 March. The answer contains two claims: born in Germany, and born on 20 March. The context supports the first and not the second. One supported claim out of two gives a faithfulness of 0.5.
Notice what this measures. It does not ask whether the answer is true in the world. It asks whether the answer stuck to the documents you gave it. That is exactly the Air Canada failure: an answer that sounded like policy and was not in the policy. Ragas lists other RAG evaluation metrics too; learn faithfulness first and add others when a real failure calls for them.
Two suites: the hill and the tripwire
Anthropic separates two kinds of test suite. A capability eval covers things the system cannot yet do well. It starts with a low pass rate and gives the team a hill to climb. A regression eval covers things the system already does correctly. It should sit near 100%, and its only job is to catch breakage.
One honest limit: the Deakin University authors of the January 2024 "Seven Failure Points" paper you met in Lesson 7 observe that validating a RAG system is only feasible during operation, so monitoring continues after launch. Lesson 11 on LLMOps starts that at Level 3, and Lesson 18 on AI observability and continuous evaluation shows how the golden set you build here becomes the baseline that production checks are scored against.
The Workshop: Build Your Golden Test Set by Critique Shadowing in a Spreadsheet
The method, critique shadowing, is adapted from Husain's process and Anthropic's guide. It is the examiner's apprenticeship: an expert grades and explains, and a judge shadows the expert until their verdicts match.
Who is in the room: one principal domain expert (the person whose judgment the business already trusts on this subject), a second expert for blind spot checks, the engineer who will build and maintain the judge, and the product or business owner, who signs off the re-run rule. No steering committee: only the expert can decide what a correct answer is.
Time: a 2-hour kickoff, about 4 to 5 hours of expert grading spread over a week (the golden set plus extra answers to test the judge on), and a 1-hour review.
Inputs: the question set from Lesson 6, the logged answers and failure tags from Lesson 7, and real tickets or complaints.
Tooling: a spreadsheet. The hard part is the expert's judgment, which no product supplies.
The nine steps
- Name the principal domain expert. One person, by name, with time booked. Not a committee. If you cannot name whose "yes, that answer is right" the business would accept, stop and solve that first.
- Assemble 30 to 50 tasks from real sources. Cover each scenario and user type. Include the failures you already found in Lesson 7. Include negative cases: questions the system must decline or escalate.
- Write the expected answer for each task. The expert writes the facts a correct answer must contain, plus the supporting source. "Must state 36 months and must name the region" is easier to grade than a model paragraph.
- Run the system and have the expert grade. Each output gets pass or fail and a one or two sentence critique. No scales. The expert reads the transcript for anything that fails.
- Have the second expert grade a sample blind. Blind means without seeing the first verdicts. Where the two disagree, fix the task wording, not the grade.
- Fix the obvious errors the critiques expose, then re-run. Take the cheap fixes before you automate anything.
- Draft an LLM judge, tune it on one set and score it on another. Split the expert's labelled answers three ways: a few clear passes and fails to paste into the judge prompt as examples, a development set you tune the judge against, and a held-out set you hide until the prompt is frozen and then score once. Give the judge an "Unknown" option and keep one judge per dimension. Report two numbers, from the held-out set only: how many true failures it caught, and how many true passes it agreed with.
- Do error analysis. Group failures by root cause and route each group to its owner: data (Lesson 6), retrieval (Lesson 7) or prompt (Lesson 9).
- Freeze the set as version 1. Mark which tasks form the regression suite. Write the rule down: no prompt, model or pipeline change ships without a re-run, and the judge is re-scored on fresh held-out answers after any material change, such as a model update.
Why report two numbers for the judge?
Because a single agreement figure can hide a judge that catches nothing. Husain calls this the agreement trap. Suppose only 5% of your cases are real failures. A judge that says "pass" to everything agrees with the expert 95% of the time, and has not caught one problem.
So always ask two questions: of the answers the expert failed, how many did the judge fail, and of those she passed, how many did it pass? The first number protects customers.
A worked example: the warranty assistant
This example is invented for illustration. It continues the 900-person industrial equipment distributor from Lessons 6 and 7, whose service desk wants an assistant that answers warranty and parts questions from field technicians.
The head of warranty is named principal expert, and a senior service engineer agrees to be second reviewer. In the kickoff she and the engineer build 40 tasks from recent tickets and the Lesson 7 failure log. The IDs are fixed for life, and Lesson 9 refers to them:
- T-01 to T-30: answerable questions, each with required facts and a source.
- T-31 to T-36: six questions where the right behavior is "we do not hold that information". These are the content gaps Lesson 6 found.
- T-37 to T-40: four questions where the right behavior is "escalate to a human", such as anything mentioning an injury.
Three graded rows from her sheet:
| ID | Question | Required facts and source | Type | Verdict | Expert critique |
|---|---|---|---|---|---|
| T-17 | Is the hydraulic pump on a Series 4 loader still covered in month 30? | Must state 36 months for this product line and must name the region. Source: current warranty policy, section 3. | Should answer | Fail | "Says 24 months. That is the term for a different region, and it never says which region it is answering for." |
| T-33 | What is the torque setting for the mounting bolts on the older bracket? | No written source exists. Must say so and offer to raise it. | Should decline | Pass | "Correctly says it has no source and points to the service desk. Good." |
| T-38 | The pump failed and an operator was hurt. Can I approve a replacement? | Must hand over to a person. Must not rule on the claim. | Should escalate | Fail | "Answers the warranty question and ignores the injury. This must go to a human every time." |
She grades the whole first run in about two hours: 28 pass, 12 fail. The second reviewer grades 15 tasks blind and disagrees on two. Both turn out to be questions that did not name a product line, so the tasks are reworded. Nobody changes a grade to make the numbers agree. Her critiques point to three cheap fixes (two broken links to source documents and one missing formatting rule), and after them the re-run scores 31 of 40. That 31 is the starting score Lesson 9 picks up.
How do you know the judge is any good?
You score it once, on expert-labelled answers it was never tuned against. A judge prompt revised again and again against the same examples learns those examples, and its score on them flatters it. Husain's guidance is to split the labels three ways: roughly 10 to 20% as examples for the prompt, and 40 to 45% each for a development set you revise against and a test set hidden until the prompt is frozen.
So the expert first labels more answers than the golden set holds: 52 logged answers from the Lesson 7 runs, on questions outside the 40 tasks, in about two more hours. The engineer splits all 92.
| Set | What is in it | How it is used |
|---|---|---|
| Examples (12) | Clear Lesson 7 passes and fails, with critiques | Pasted into the judge prompt |
| Development (40) | The first-run answers to the 40 golden-set tasks: 12 fail, 28 pass | Compare and revise as often as needed; never in the prompt |
| Held-out test (40) | The other 40 Lesson 7 answers: 14 fail, 26 pass | Hidden until the prompt is frozen; scored once |
| Round | Judge catches true failures | Judge agrees on true passes | Overall agreement | Decision |
|---|---|---|---|---|
| v1, development (generic "is this correct?") | 7 of 12 (58%) | 26 of 28 (93%) | 33 of 40 (82.5%) | Revise: misses 5 failures |
| v2, development (expert critiques as examples) | 10 of 12 (83%) | 26 of 28 (93%) | 36 of 40 (90%) | Revise: still passes two wrong-region answers |
| v3, development (separate "right region and product line" judge) | 11 of 12 (92%) | 27 of 28 (96%) | 38 of 40 (95%) | Freeze the prompt and score the held-out set |
| v3 frozen, held-out set, scored once | 12 of 14 (86%) | 24 of 26 (92%) | 36 of 40 (90%) | Provisionally used for regression runs; the expert still reviews every failure and samples passes weekly |
Read row 1 slowly, because it is the whole lesson in one line. An agreement of 82.5% sounds acceptable in a steering committee. Behind it, the judge was missing 5 of 12 real failures. One number would have automated a grader that waved through nearly half the wrong answers.
Row 3 shows "one judge per dimension" earning its keep: the wrong-region error was invisible to a general correctness judge and obvious to one asked only "Does the answer name the right region and product line?"
Now compare rows 3 and 4. On answers it was tuned against, judge v3 caught 11 of 12 failures; on answers never used to tune it, 12 of 14, and pass agreement fell from 27 of 28 to 24 of 26. Only row 4 goes in the record, because only row 4 predicts next week's behavior. Tuning and reporting on the same 40 tasks would have quoted 95% for a judge that measures nearer 90%.
They also record a limit honestly. Husain's guidance is about 100 labelled examples per failure mode to validate a judge, and he warns that below 60 examples the confidence intervals are often too wide to support a useful conclusion. A held-out set of 40, with 14 failures, is well short. So the decision is provisional: the judge runs the regression suite, the expert reviews every answer it fails and samples its passes weekly, and newly graded answers grow the held-out set until it is large enough to trust.
Error analysis sorts the 12 first-run failures into four piles: wrong region or product line, T-17 and T-22 among them (prompt owner, Lesson 9); answering when it should decline, in two of T-31 to T-36 (prompt owner); missing escalation, such as T-38 (prompt and pipeline rule); and contradicting policy versions (data owner, Lesson 6). The team freezes version 1, flags 15 tasks that already pass reliably as the regression suite, and writes the re-run rule into the release checklist.
What should you expect? Husain describes Honeycomb's Query Assistant, whose LLM judge reached over 90% agreement with domain expert Phillip Carter after three iterations in a spreadsheet. Carter said that seeing the model's reasoning "made me realize I wasn't being consistent".
The artifact: Golden Test Set v1 and Judge Calibration Record
You leave the workshop with one document in two parts. Copy this template into a spreadsheet: one row per task on the first tab, one record on the second.
| Part | Field | What to write |
|---|---|---|
| Per task | ID | A stable identifier that never gets reused |
| Per task | Question | The real wording, from a ticket, complaint or log |
| Per task | Expected answer or required facts | What a correct answer must contain |
| Per task | Supporting source | The document and section that proves it |
| Per task | Type | Should answer, should decline, or should escalate |
| Per task | Regression flag | Yes if a failure here must block a release |
| Per task | Expert verdict and critique | Pass or fail, plus one or two sentences in the expert's words |
| Per task | Failure category | The root-cause group from error analysis |
| Record | Principal expert and second-reviewer agreement | Names, and how many blind verdicts matched |
| Record | Judge prompt version | Which judge, for which dimension |
| Record | Judge data split | How many labelled answers sit in the examples, development and held-out sets, and their pass and fail counts |
| Record | Judge versus expert | Failures caught and passes agreed on the held-out set, as two separate numbers |
| Record | Pass rate per run | Tasks passed over tasks run, with the date |
| Record | Faithfulness on a sample | Supported claims over total claims |
| Record | Three-run consistency | Share of tasks passing three runs of three |
| Record | Change log and re-validation trigger | What changed, and what forces the judge to be re-checked |
Failure Modes and the Metrics to Track
Evaluation programs rarely fail loudly. They decay. Here are seven ways it happens and the sign that gives each one away.
Seven failure modes and how to spot them
| Failure mode | How you spot it | What to do |
|---|---|---|
| Vibe checks | "Good enough" rests on a handful of tries that nobody recorded | Ask to see the run. If there is none, there was no evaluation. |
| Invented test questions | No task traces back to a ticket, complaint or log | Replace them with real wording from the support queue |
| Likert drift | Two graders give the same answer a 3 and a 4 and neither can explain why | Switch to pass or fail with a critique |
| Uncalibrated judge | Only one agreement number is reported, or the judge has never been compared with the expert | Report failures caught and passes agreed separately |
| Judge marked on its own homework | The judge's agreement figure comes from the same examples its prompt was revised against | Hold out a test set, hide it until the prompt is frozen, and report only that score |
| Happy-path-only set | The set has no decline or escalate cases | Add tasks where the correct behavior is to refuse or hand over |
| Frozen in the past | No task added since launch, or a 100% pass rate for months on a system users still complain about | Add the newest real failures from the ticket queue as new tasks, every week |
The last one feels like success. A capability suite that reads 100% has stopped teaching you anything: move those tasks into the regression suite and find harder ones in the ticket queue.
Which LLM evaluation metrics should you track?
Track five, and put them in the record with a date every time you run.
- Golden-set pass rate: tasks passed divided by tasks run, per run, graded by the expert or a calibrated judge.
- Faithfulness: supported claims divided by total claims in an answer, measured on a sample, using the Ragas definition.
- Judge calibration: on the held-out set, the share of expert-marked failures the judge catches, and the share of expert-marked passes it agrees with. Always two numbers, never the development-set figures.
- Consistency: the share of tasks that pass three runs out of three. This is the number that separates a system from a demo.
- Regression suite pass rate: should sit near 100%. Any drop blocks the change until someone has read the transcript.
If your steering committee sees only one figure, make it this pair: the pass rate on the golden set, and the share of real failures your judge catches. The first says how good the system is. The second says whether you can believe the first.
Frequently Asked Questions
How many examples does a golden dataset for an LLM need?
Start with 20 to 50 tasks drawn from real failures; Anthropic's January 2026 guidance is that this is enough early on, because early changes have large effects. Validating an LLM judge needs more labelled answers than that, about 100 per failure mode by Husain's guidance, so keep growing the set from real tickets.
Can an LLM as a judge replace our human reviewers?
Not entirely, and not until it has been scored against a named domain expert on answers it was never tuned on. Even then, keep the expert reviewing the answers the judge fails and sampling the ones it passes, and report failures caught and passes agreed as two separate numbers.
Is a company liable for what its chatbot tells customers?
In Moffatt v. Air Canada (14 February 2024) a British Columbia tribunal said yes: it held the airline responsible for all the information on its website, rejected the argument that the chatbot was responsible for its own actions, and ordered it to pay C$812.02. Take legal advice for your own jurisdiction, but plan on owning what your system says.
How often should we re-run the golden test set?
Before every prompt, model, data or pipeline change ships, with no exceptions. Re-score the judge itself on fresh held-out answers after any material change, such as a model update, because a judge that matched the expert on the old model may not match her on the new one.
Do we need to buy an LLM evaluation platform to start?
No. A spreadsheet with one row per task, the expert's verdict and critique, and a record of each run is enough for a first golden set. The scarce input is the expert's judgment, and no product supplies that; consider tooling once the number of runs makes the spreadsheet the bottleneck.
Key Takeaways
- You own what your AI says. In Moffatt v. Air Canada (February 2024) the "responsible for its own actions" argument failed, the airline paid C$812.02, and one golden-set question would have caught the error.
- A golden test set is a fixed route: 30 to 50 real questions with expert-approved answers, re-run before any prompt, model or pipeline change ships.
- Name one principal domain expert. Grade pass or fail with a written critique, never on a 1-to-5 scale.
- Include the cases where the right behavior is to decline, to escalate, or to tell an EU user they are talking to an AI, not only the questions the system should answer.
- An LLM judge earns trust by shadowing the expert. Tune it on a development set, score it once on a held-out set, and report failures caught and passes agreed as two separate numbers.
- Keep two suites: a capability suite that gives you a hill to climb, and a regression suite that sits near 100% and blocks any change that breaks it.
- Your artifact is the Golden Test Set v1 and Judge Calibration Record. Lesson 9 uses it to turn every prompt change into a test run.
Skill.re