Evaluate a prompt

“It looks better” is not evidence. An evaluation — an eval — runs your prompt over a fixed set of test cases and scores every output automatically, turning prompt engineering from guesswork into measurement. With evals you can compare prompt versions, switch models confidently, and catch regressions before users do. This lesson builds a complete eval harness from scratch.

Last updated: 01 Nov, 2026

Definitions

Definition · Evaluation (eval)

A repeatable test that runs a prompt (or a whole AI system) over a dataset of inputs and scores the outputs against expected results or quality criteria, producing metrics you can compare across versions.

Definition · Test set (golden dataset)

A curated collection of inputs, each paired with an expected output or grading criteria. It should represent real traffic, including edge cases and known past failures.

Definition · Grader

The function that scores one output. Code-based graders check things deterministically (exact match, valid JSON, contains a keyword). Model-based graders (“LLM-as-judge”) use another model with a rubric for qualities code cannot check, such as helpfulness or faithfulness.

Step 1 — Build a test set

Start small: 20–50 cases is enough to begin; grow it over time.
Sample real inputs from logs (anonymised), not just ones you invent.
Cover categories evenly, plus edge cases: empty input, very long input, other languages, off-topic, adversarial.
Add every production failure you fix — this is how the set becomes valuable.
Store it as JSONL in version control next to the prompt.

eval_set.jsonl

{"id": "t1", "input": "I was charged twice this month", "expected": "billing"}
{"id": "t2", "input": "Backup job fails with error 504", "expected": "technical"}
{"id": "t3", "input": "How do I add a teammate?", "expected": "account"}
{"id": "t4", "input": "Can you recommend a good pizza place?", "expected": "other"}

Step 2 — Choose metrics

Task type	Good metrics
Classification	Accuracy, per-class precision/recall, confusion matrix
Extraction to JSON	Schema validity rate, field-level exact match
Question answering with sources	Correctness, faithfulness to sources, citation validity
Summarisation / writing	Rubric scores (coverage, accuracy, concision) via LLM-as-judge, length limits
All tasks	Cost per case, latency, refusal rate, format-error rate

Prefer code-based graders whenever possible: they are fast, free, and perfectly consistent. Use LLM-as-judge for the qualities only a reader can assess.

Step 3 — Build the harness

The harness below compares two prompt versions on the same test set. To keep it runnable offline, fake_model simulates how each prompt behaves: v1 is a vague prompt that misses some categories, v2 is the improved prompt. In real use, replace fake_model with an API call.

Python

from collections import Counter

TEST_SET = [
    {"id": "t1", "input": "I was charged twice this month", "expected": "billing"},
    {"id": "t2", "input": "Backup job fails with error 504", "expected": "technical"},
    {"id": "t3", "input": "How do I add a teammate?", "expected": "account"},
    {"id": "t4", "input": "Recommend a pizza place?", "expected": "other"},
    {"id": "t5", "input": "Refund for my annual plan", "expected": "billing"},
    {"id": "t6", "input": "Restore stuck at 90%", "expected": "technical"},
    {"id": "t7", "input": "Change my login email", "expected": "account"},
    {"id": "t8", "input": "What's the weather today?", "expected": "other"},
]

# Simulated behaviour of two prompt versions (stand-in for real API calls).
SIMULATED = {
    "v1": {"t1": "Billing", "t2": "technical", "t3": "technical", "t4": "account",
           "t5": "billing", "t6": "technical", "t7": "Account issue", "t8": "account"},
    "v2": {"t1": "billing", "t2": "technical", "t3": "account", "t4": "other",
           "t5": "billing", "t6": "technical", "t7": "account", "t8": "account"},
}
LABELS = {"billing", "technical", "account", "other"}


def fake_model(prompt_version: str, case: dict) -> str:
    return SIMULATED[prompt_version][case["id"]]


def grade(output: str, expected: str) -> dict:
    return {
        "valid_label": output in LABELS,          # format check
        "correct": output == expected,            # exact match
    }


def run_eval(prompt_version: str) -> dict:
    results = []
    for case in TEST_SET:
        output = fake_model(prompt_version, case)
        results.append({**case, "output": output, **grade(output, case["expected"])})
    n = len(results)
    return {
        "version": prompt_version,
        "accuracy": sum(r["correct"] for r in results) / n,
        "format_ok": sum(r["valid_label"] for r in results) / n,
        "failures": [(r["id"], r["expected"], r["output"]) for r in results if not r["correct"]],
    }


for version in ["v1", "v2"]:
    report = run_eval(version)
    print(f"{report['version']}: accuracy={report['accuracy']:.0%}  format_ok={report['format_ok']:.0%}")
    for failure in report["failures"]:
        print("   fail:", failure)

v1: accuracy=38% format_ok=75% fail: ('t1', 'billing', 'Billing') fail: ('t3', 'account', 'technical') fail: ('t4', 'other', 'account') fail: ('t7', 'account', 'Account issue') fail: ('t8', 'other', 'account') v2: accuracy=88% format_ok=100% fail: ('t8', 'other', 'account')

The report tells a clear story: v2 fixed the formatting problems (capitalised and invented labels) and most misclassifications, raising accuracy from 38% to 88%. One failure remains — t8, an off-topic question classified as account — which tells you exactly what to work on next.

LLM-as-judge

For open-ended outputs, a second model grades each answer against a rubric. Good judge prompts are specific, use a small scale, and ask for reasoning before the score.

Judge prompt

You are grading an AI support answer.

<question>{question}</question>
<reference_docs>{docs}</reference_docs>
<answer>{answer}</answer>

Score each criterion from 1 (poor) to 3 (good):
- faithfulness: every claim is supported by the reference docs
- completeness: the answer fully addresses the question
- concision: no unnecessary content; under 5 sentences

Return JSON: {"reasoning": "...", "faithfulness": n, "completeness": n, "concision": n}

Judge best practice	Why
Use a strong model as the judge	Weak judges are noisy and easily fooled.
Grade one criterion per score, on a small scale (1–3 or pass/fail)	Fine-grained 1–10 scales are inconsistent.
Ask for reasoning before the score	Improves consistency and lets you audit judgements.
Provide a reference answer or source docs	Grounds the judge in the truth.
Check the judge against human labels on ~30 cases	Confirms the judge agrees with people before you trust it.
Watch for biases	Judges can favour longer answers, their own model family, or the first option shown.

Step 4 — Compare versions properly

Change one thing at a time so you know what caused the difference.
Run the full set every time — improving one category often breaks another.
Mind the sample size: on 20 cases, one case is 5 percentage points. Do not celebrate a 3-point gain on a tiny set; grow the set instead.
Repeat runs at temperature > 0 and average, since outputs vary.
Track cost and latency alongside quality — a 1% gain that doubles cost may not be worth it.

Step 5 — Prevent regressions

Definition · Regression

A drop in quality on cases that previously worked, caused by a prompt edit, a model upgrade, or a change elsewhere in the pipeline.

Python

BASELINE = {"accuracy": 0.875, "format_ok": 1.0}
TOLERANCE = 0.02


def check_regression(current: dict, baseline: dict = BASELINE) -> list[str]:
    problems = []
    for metric, base_value in baseline.items():
        if current[metric] < base_value - TOLERANCE:
            problems.append(f"{metric} dropped {base_value:.0%} -> {current[metric]:.0%}")
    return problems


print(check_regression({"accuracy": 0.88, "format_ok": 1.0}) or "no regressions")
print(check_regression({"accuracy": 0.75, "format_ok": 0.9}))

no regressions ['accuracy dropped 88% -> 75%', 'format_ok dropped 100% -> 90%']

Run this in CI whenever a prompt file changes, and before switching model versions. A failing check blocks the merge, just like a failing unit test.

Tools

The harness above is all you need to start. As your evals grow, dedicated tools add dashboards, dataset management, judge templates, and tracing — for example open-source promptfoo and OpenAI Evals, or hosted platforms such as LangSmith and Braintrust. The concepts — test set, graders, metrics, comparison, regression — are the same everywhere.

Try it

Write 20 test cases for a prompt you use, stored as JSONL.
Replace fake_model with a real API call and run the harness for your current prompt.
Make one improvement, re-run, and keep it only if accuracy improves without hurting format or cost.
Save the scores as your baseline and add check_regression to a script you run before every prompt change.

Key takeaways

An eval runs a prompt over a fixed test set and scores outputs automatically.
Build the test set from real inputs, cover edge cases, and add every production failure.
Prefer code-based graders; use LLM-as-judge with rubrics, reasoning, and human calibration for subjective qualities.
Change one thing at a time, run the full set, and respect sample size.
Keep a baseline and block regressions in CI — prompts are code and deserve tests.

Where to go next

You have completed Prompt Engineering. Continue with Building RAG Systems to ground your prompts in your own documents, or Generative AI & Agents to build tool-using assistants.