Evaluate and optimize a single LLM call

This is the smallest, most common eval: one prompt, one call, one output per input. It is also where most “prompt engineering” goes wrong, because a change that looks better on three examples often quietly breaks five others. An eval replaces the vibe check with a number you can move.

Here is that number moving. Same task, same dataset, same grader, three versions of one prompt:

Nothing here is specific to any one tool. The loop is: score the prompt, read the failures, change one thing, score again.

1. Build a dataset

Pick a task a single call can do, where the output is easy to grade: classification, extraction, a short answer. Each case is an input paired with the correct output. Say you are routing support tickets into four categories:

Twenty to fifty cases is plenty to start. Draw them from real traffic if you can, and make sure the hard and ambiguous ones are represented, since those are what a prompt change actually has to fix.

2. Pick an evaluator

Single-call outputs are usually cheap to grade. For a fixed label, exact match is enough: deterministic, fast, and unambiguous. For structured output, check the JSON parses and the fields are valid. Only reach for an LLM judge when the output is open-ended (a summary, a rewrite) and exact match cannot work.

Whatever you pick, keep the dataset and the grader fixed across every prompt version, or the scores stop being comparable.

3. Run the baseline

Run the current prompt across the dataset and score it. That is your baseline: v1 lands at 72%. In Respan this is an experiment, a prompt run over a dataset with the grader attached, so you get the score and every wrong row in one view.

A useful habit: prototype quickly and loosely first (paste a few inputs, eyeball the outputs, get the prompt roughly right), then run the full scored experiment. The quick pass is for exploring; the experiment is the number of record.

4. Improve the prompt

Now optimize, driven by the failures rather than by guessing. Open the rows the grader marked wrong and look for a pattern:

• v1 was one line, so the model invented its own category names and mislabeled edge cases. Adding the exact category list with a one-line definition each took it to 88%.
• The remaining misses were genuinely ambiguous tickets. Adding three labeled examples to the prompt resolved most of them: 94%.

Change one thing at a time and re-run, so you know which edit earned the points. This is prompt optimization done by hand; automating the search is a later page in this guide. The manual version is where you should start, because it teaches you what your failures actually are.

5. Do not fool yourself

The same prompt does not return the same output twice, so a small score move can be noise, not progress. Two guards:

• Pin down variance. Set temperature to 0 for a classification task, and run each prompt version a few times. If v3 swings between 92% and 95% run to run, a one-point “win” over v2 is inside the noise.
• Trust only moves that clear the spread. The jumps that matter here (72 to 88 to 94) are far larger than the run-to-run wobble, so they are real. A 90 to 91 would not be.

What this gives you

• Prompt changes become measured, not argued. You ship the version with the higher number, and you can prove it.
• A regression guard. Re-run the dataset before shipping any prompt edit, and you catch the “improvement” that quietly breaks something else.
• The base unit. A model, a RAG pipeline, and a multi-turn assistant are all built out of calls like this one. Everything later in this guide reuses the same loop: dataset, evaluator, experiment, compare.

Next steps

• Automate the prompt search once the manual loop is not enough, covered in the optimization guide.
• Run this dataset in CI on every prompt change, so a regression blocks the merge instead of reaching production.