Measuring your LLM app: golden datasets and LLM-as-judge

Shipping an LLM-based feature to production and then coasting on "it feels good" means you'll quietly miss a regression that breaks a handful of scenarios weeks later. Without the measurement discipline of an LLM eval golden dataset, you can't eyeball which scenarios a small prompt tweak just broke. In this post you'll see, step by step, how to build a golden dataset, when to pair deterministic metrics with LLM-as-judge, and how to make human evaluation affordable.

💡 Pro Tip: Start your eval set with code-graded (exact match/string match) scenarios first; bring in LLM-as-judge only for the nuanced outputs where code-graded falls short. This ordering is also prioritized in Anthropic's own eval guide.

Table of Contents

• Why "eyeballing it" isn't measurement
• Golden dataset: picking examples and labeling them
• Deterministic metrics: exact match, schema, latency
• LLM-as-judge: writing rubrics and bias traps
• Making human evaluation affordable through sampling
• Wiring eval into CI and setting a threshold
• Growing the eval set from production traffic
• FAQ
• How do you test an LLM application?
• How do you build a golden dataset, and how many examples are enough?
• Is LLM-as-judge reliable, and when does it fail?
• How do I know a prompt change caused a regression?
• When is human grading needed?
• Update (September 2026)
• Conclusion
• Sources

Why "eyeballing it" isn't measurement

Changing a prompt, reading five or ten examples by hand, and saying "looks better" is an impression, not a measurement. Anthropic's eval guide stresses that success criteria should be Specific, Measurable, Achievable, Relevant: "Specific... Measurable... Achievable... Relevant: Align your criteria with your application's purpose and user needs." That means even the "haziest"-looking topics (e.g. safety) can be reduced to a numeric threshold; the guide's own example is: "Less than 0.1% of outputs out of 10,000 trials flagged for toxicity by the content filter."

The real problem with eyeballing isn't that it doesn't scale — it's that it isn't repeatable. Re-read the same ten examples a week later and you may form a different impression; two different developers can look at the same output and reach different conclusions. A golden dataset plus a clear criterion removes that subjectivity and turns it into a comparable number: "the previous version passed 34 of these 40 scenarios, the new version passed 37." That's also what lets you track which step caused a regression in multi-step LLM flows, the same way you would in agentic AI architectures.

This gap becomes especially critical as a team grows. Working solo, "it feels good" can work to a degree since one mind makes the call; but once multiple developers touch the same prompt, each person's definition of "good" drifts slightly. A clear success criterion collapses these different mental models into one shared reference — the debate shifts from "I think it's better" to "the golden set score dropped from 0.91 to 0.87, which category broke." That's a discipline close to code review: you argue the change's impact with a number, not a hunch, before merging.

Golden dataset: picking examples and labeling them

Two of Anthropic's eval design principles apply directly when building a golden dataset. The first is task-specificity: "Be task-specific: Design evals that mirror your real-world task distribution. Don't forget to factor in edge cases!" In other words, your set should reflect the actual input distribution you'll face in production — not just easy, "nice" examples, but edge cases like empty input, very long input, and contradictory instructions.

The second principle is favoring volume over quality: "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." In practice this means hundreds of automatically-gradable examples (exact match, or LLM-judge with a clear rubric) give a more reliable signal than dozens of meticulously hand-labeled ones. During labeling, add three fields to every example: the input, the expected output (or rubric), and which grading method will be used (code/human/LLM). As you'll see in the next section, this three-way split makes it easier to decide which scenario gets tested with which method.

The biggest time-waster during labeling is having one person produce all the examples in a single session — that yields a narrow distribution reflecting only their own mental model. It's healthier to feed the set from different sources: scenarios the product team predicts, examples support compiles from real complaints, and edge cases developers deliberately try to break. A category tag on each example (e.g. "core flow," "edge case," "abuse attempt") lets you later see exactly which category's score dropped — that breakdown surfaces a regression far faster than one aggregate score.

This three-way labeling isn't just record-keeping; it also manages the set's growth. Before adding a new example, ask: will a code-graded check evaluate this (specific format, specific keyword), or does it need a nuanced rubric? If the answer isn't clear, you've probably defined the example too broadly — narrowing the rubric and splitting the example in two makes both labeling and reading which category broke easier. As the set grows, each category gets its own sub-score, making a regression hidden behind one aggregate percentage (e.g. "edge case" drops while "core flow" stays flat) visible — and lets you prioritize which section needs relabeling or expansion, so growing the set becomes work focused on the weakest category rather than something random.

Deterministic metrics: exact match, schema, latency

Anthropic's grading framework defines three methods: code-based, human, LLM-based. For code-based grading, two basic examples are given — exact match ("output == golden_answer") and string match ("key_phrase in output"). This method is described as "Fastest and most reliable, extremely scalable, but also lacks nuance for more complex judgments that require less rule-based rigidity": fastest and most reliable, but weak on complex judgments that need nuance.

Here's what a working example on the code side looks like — a simple script that computes how many of 12 scenarios matched exactly:

1# How many of 12 test scenarios matched golden_answer exactly
2results = [True, True, False, True, True, True, False, True, True, True, False, True]
3exact_match_rate = sum(results) / len(results)
4print(f"{exact_match_rate:.2%}")  # 75.00%

These deterministic checks scale to anything measurable — JSON schema validation, required-field presence, output length, or response latency. As Anthropic emphasizes, this method is the fastest, so putting as much of your eval set as possible into code-graded scenarios frees up your remaining budget for cases that genuinely need nuance.

In practice, split code-graded checks into two layers. The "hard" layer: is the output valid JSON, are required fields filled, does it contain a banned word — fail these and the score is automatically zero, never even reaching the LLM-judge, since the output is unusable. The "soft" layer: does the keyword appear, is the length in range, is latency under threshold — fail these and the score is lowered but not zeroed. This two-layer split distinguishes what kind of error occurred (format vs. content) instead of a single "pass/fail" flag.

LLM-as-judge: writing rubrics and bias traps

Where code-based grading falls short, LLM-based grading takes over: "LLM-based grading: Fast and flexible, scalable and suitable for complex judgment. Test to ensure reliability first then scale." But there are three concrete rules for making this reliable. First, write a clear rubric — Anthropic's own example: "Have detailed, clear rubrics: 'The answer should always mention Acme Inc. in the first sentence. If it does not, the answer is automatically graded as incorrect.'" Second, keep the rubric empirical and specific: "instruct the LLM to output only 'correct' or 'incorrect', or to judge from a scale of 1–5" — a fixed scale instead of free text. Third, have the judge reason first and then discard that reasoning: "Encourage reasoning: Ask the LLM to reason first before producing an evaluation score, and then discard the reasoning."

A simple rubric definition might look like this:

1{
2  "rubric": "The answer must mention the product name in the first sentence. If it does not, it is automatically graded as incorrect.",
3  "scale": "correct | incorrect",
4  "reasoning_before_score": true
5}

LLM-as-judge also has documented limits: Zheng et al.'s MT-Bench/Chatbot Arena study shows that judge models carry "position, verbosity, and self-enhancement biases, as well as limited reasoning ability." The same study also documents that strong judge models (their GPT-4 example) can reach "over 80% agreement" with human preferences — on par with agreement between humans — so LLM-judge isn't entirely unreliable, but you do need to use it knowing its bias types:

Making human evaluation affordable through sampling

In Anthropic's framework, human grading comes with a clear warning: "Human grading: Most flexible and high quality, but slow and expensive. Avoid if possible." That doesn't mean you need to drop human evaluation entirely — most of the time it's enough to run it on a sample rather than the whole set. My usual order: first run code-graded and LLM-judge on the entire set, then pull the rows the judge scored with low confidence, or where two different judge runs disagreed, into a separate bucket, and spend human review only on that bucket. That way the expensive, slow human effort concentrates on the genuinely ambiguous cases, while the large remaining majority passes quickly through automated grading.

This approach also lets you continuously check the judge's calibration: if a human labeler's decision systematically disagrees with the LLM-judge's decision, that's a signal either your rubric is ambiguous or the judge carries one of the biases discussed in the previous section.

When organizing human review, rather than relying on one person, have at least part of the ambiguous bucket independently labeled by two. If they reach different decisions, your rubric isn't clear even to human eyes — and the LLM-judge will likely struggle with the same ambiguity. Keeping these "even humans can't agree" rows separate and using them to sharpen the rubric raises agreement over time on both sides.

Wiring eval into CI and setting a threshold

Wiring eval into a CI step is a natural extension of the guide's automation principle ("Automate when possible: Structure questions to allow for automated grading") and of the iterative prompt-engineering loop (test scenarios → draft prompt → iterative test/refine → final validation → ship). I generally prefer adding the eval script as a CI step with a minimum score threshold; if the script falls below that threshold, the build goes red and a prompt or model change gets caught before it's merged:

1# Simple example: run the eval script as a CI step, fail the build under the threshold
2python run_evals.py --dataset golden_set.jsonl --min-score 0.85

Don't set the threshold number (0.85 in this example) once and forget it — revisiting the threshold every time the model or prompt changes meaningfully keeps your false-positive/false-negative rate low.

Set the threshold too high and every small fluctuation turns the build red, so the team eventually starts ignoring the eval; set it too low and real regressions slip through silently. You can start the threshold based on the current prompt's score (e.g. if the current score is 0.91, set the threshold slightly below at 0.85), then fine-tune it over a few weeks by watching how often false alarms occur.

When a threshold breach breaks the build, reporting which category (core flow or edge case) dropped, separately, gives a far more useful signal than a single aggregate score; otherwise the team has to manually investigate the same question ("what broke?") on every red build. It also helps to combine this with the two-layer check (hard/soft) mentioned earlier: failing a hard check breaks the build directly, while small dips in soft checks are only logged as warnings — letting the team distinguish genuinely critical regressions from small fluctuations. Printing the threshold together with the category breakdown into the CI output also speeds up PR review — the reviewer can read the report directly instead of guessing which prompt line affected which category.

Growing the eval set from production traffic

Production traffic surfaces the edge cases you can't see when you build a golden dataset from developer guesses alone — real user queries reveal the corner cases you didn't originally anticipate. The practical approach is to periodically review real production interactions where the judge scored low or the user reacted negatively (retrying, complaining, abandoning the session), and add representative examples to the golden set. That lets the set converge toward production's real distribution over time — much like a RAG vs. fine-tuning decision, you update your call with real usage data instead of assumptions.

Turning this into a regular habit only needs a simple filter script — pull the rows from production logs where the judge score fell below the threshold into a separate queue:

1# Extract rows with low judge score from prod logs into a golden-candidate queue
2threshold = 0.6
3candidates = [row for row in prod_logs if row["judge_score"] < threshold]
4print(f"{len(candidates)} candidate rows sent for review")

This queue shouldn't be added to the set automatically — you need to have a human review every candidate row and decide whether it's genuinely representative and a recurring pattern, or just one-off noise. Otherwise your set can drift away from the "real task distribution" the principles call for, toward just "the weird examples the judge struggled with."

FAQ

How do you test an LLM application?

By mixing the three grading methods based on task type: code-graded (exact match/string match) for clear right/wrong outputs, LLM-as-judge with a clear rubric for cases that need nuance but also scale, and human evaluation only on rows where the judge remains ambiguous. Anthropic's framework calls these three Code-based grading, Human grading, and LLM-based grading, and recommends pushing automation as far as possible.

How do you build a golden dataset, and how many examples are enough?

There's no fixed ideal number; what matters isn't the count but the priority: Anthropic's stated principle is "more questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals." So aim to build a broader, automatically-gradable set that covers the real task distribution and edge cases, rather than a small number of perfectly labeled examples.

Is LLM-as-judge reliable, and when does it fail?

Strong judge models can show high agreement with human preferences — according to Zheng et al.'s study, "over 80% agreement," on par with agreement between humans. But the same study also shows judges carry position, verbosity, and self-enhancement bias and have limited reasoning ability; so the judge can fail when the rubric isn't clear or when answer order/source isn't controlled for.

How do I know a prompt change caused a regression?

By running the golden dataset through the same script on every prompt/model change and comparing against the previous score. If you wire this into a CI step with a minimum score threshold, a regression gets caught at the build stage before it slips past human eyes.

When is human grading needed?

Anthropic's advice is clear: avoid it if possible, because it's slow and expensive. But when nuance is critical, and in scenarios where the judge model is ambiguous or its reliability hasn't been tested yet, applying human review to a small sample rather than the whole set keeps the cost reasonable.

Update (September 2026)

This article was written based on the version and tools current as of 2025-11-18. Since then, the following change happened in OpenAI's eval tooling ecosystem: OpenAI announced on June 3, 2026 that the Evals platform (the dashboard product) is being deprecated — "On June 3, 2026, we notified developers using the Evals platform that the product is being deprecated." Per the announcement, existing evals become read-only on October 31, 2026 ("Oct 31, 2026 — Existing evals become read-only"), and the Evals dashboard and API are scheduled to shut down entirely on November 30, 2026 ("Nov 30, 2026 — The Evals dashboard and API are scheduled to shut down"). This post did not recommend any code step based on the OpenAI Evals API; the information above is for awareness only — if you use an eval workflow on OpenAI's platform, plan your migration around these dates.

Related posts published later:

• Agentic AI: Tool Use, Planner Loops, and Production Agent Architecture
• Prompt Engineering Patterns: Practical Techniques from a 10-Year Archive
• RAG or Fine-Tuning? The Definitive Guide to Production LLM Decisions
• LLM Benchmarks 2026: MMLU, HumanEval, SWE-bench and Real-World Performance
• Production agent architecture with LangGraph
• Unit test generation with Claude Code and Cursor

Conclusion

The right mix of a golden dataset and grading methods takes your LLM application out of "eyeballing it" and makes it genuinely measurable: get a fast, cheap signal from code-graded checks first, move to a clearly-rubricked LLM-as-judge where nuance is needed, and reserve human effort only for the rows that remain ambiguous.

Sources

• Anthropic — Develop test cases — the primary source for SMART success criteria, eval design principles, and the code/human/LLM-graded grading framework.
• Zheng et al., "Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena" (arXiv:2306.05685) — LLM-judge biases (position/verbosity/self-enhancement) and human-agreement rate data.
• OpenAI — Deprecations: Evals platform — deprecation/read-only/shutdown dates for the Evals platform.
• OpenAI — Evals guide — the general conceptual framework of the Evals ecosystem.
• GitHub — openai/evals — the open-source eval framework's repo status and currency.