Benchmark model choice for an agent

Which model should your agent run on? Most teams pick the biggest and hope, or the cheapest and hope. This tutorial replaces the hope with a plot.

Here is where three models landed when we ran the same agent (GPT Researcher) across all of them, on the same 20 questions:

gpt-5.4 sits in the corner you want: 95% accuracy at a fraction of the frontier model’s cost, and the smallest bubble (fastest). claude-opus-4-8 buys the last five points of accuracy at roughly 3x the cost and about twice the time. gpt-5-mini is cheap, but pays for it in accuracy.

Producing a plot like this for your own agent is four steps. None of them are specific to any one tool.

1. Choose the models to compare

Pick a spread across cost tiers so the plot has a story to tell, for example a frontier model, a mid-tier, and a cheap one. We compared gpt-5.4, claude-opus-4-8, and gpt-5-mini.

You will run the same agent on each one, so all you need is a way to point it at a single model at a time. Most agents expose that as a config value or an environment variable; you set it, run, then change it and run again in the next step.

2. Run each model to build the dataset

You need real outputs to score, and the only way to get them is to run the agent. Run it once per model over the inputs you care about, and record four things per run: the input, the agent’s final output, the total cost, and the total latency. A run is a multi-step trace (planning, searches, scrapes, many LLM calls), so cost and latency are the totals across all of its steps. Those captured runs are your dataset, taken from what actually happened rather than hand-written or re-run later.

Choose inputs that reflect what your agent is actually for, and where you can pin down a correct answer. The most useful ones are what your agent has to look up, recent data, niche details, or facts specific to your domain, rather than general trivia the model already knows. Say it researches compliance and policy questions; one model’s captured runs look like this:

Aim for 15 to 25 inputs to start, then grow the set from your agent’s real failures. If you are scoring correctness, keep the known answer for each input as a separate key for the grader to compare against.

Capturing all of that by hand, wrapping every step and summing cost and latency across the trace, is the tedious part. Respan tracing does it for you: instrument the agent once, and each run lands as one whole trace, not scattered per-call logs, with its total cost, total latency, and final output already captured.

$
pip install respan-ai
$
export RESPAN_API_KEY="your key"

1
from respan import Respan
2
3
Respan()  # every LLM call, tool call, and agent step is now captured as one trace

(Routing calls through the Respan gateway reaches every model with one key too; that suits a single LLM call more than a multi-step agent, so it is the next tutorial.)

3. Prepare your evaluators

An evaluator scores each run on one dimension. What you measure depends on your use case; common ones:

• Correctness. Does the output match the known-good answer. For open-ended text, an LLM judge grades it against the reference.
• Relevance. Does it actually answer the input, without drifting off-topic.
• Tone. Right voice and register for your product, a stand-in for user satisfaction.
• Format. Valid JSON, required fields present, matches your schema. Deterministic and cheap.
• Cost. The real cost per run from actual token usage, not a framework estimate (ours was off up to 9x).
• Latency. Wall-clock time per run.

This benchmark plots three of them, correctness, cost, and latency. You can compute them with your own grading script or on an eval platform. Whatever you use, keep the grader and any judge model fixed across every model so the scores stay comparable.

4. Run the experiment and read the plot

The experiment does not run the agent again, its outputs already exist from the previous step. It takes each captured input and output, applies your evaluator, and compares the scores across models. You score one input and one final output per case; the agent’s internal steps (GPT Researcher plans, searches, scrapes, and writes, so one question is many calls) never enter the eval, you grade the report it produced, not the path it took.

On the platform this is a pass over the traces you already captured. Cost and latency are on every trace, so two of your three numbers are done; for the third, run a correctness evaluator over the traced outputs, or let an online eval score them as they land. Then compare accuracy, cost, and latency across models. Prefer to stay in code? Read the same numbers from the traces with a script and grade correctness yourself.

Either way you end up with three numbers per model:

Plot cost on x, accuracy on y, and latency as bubble area: the model you want sits toward the top-left with a small bubble (accurate, cheap, and fast). That is the plot at the top of this guide.

What this tells you

• There is a real accuracy-versus-cost curve. More capable models cost more and answer more correctly. Now you can price it.
• The mid-tier was the sweet spot. gpt-5.4 hit 95% accuracy at a third of the frontier cost and the lowest latency. The extra five points from opus-4-8 cost roughly 3x more and ran about twice as long.
• Compare on cost per correct answer, not cost per token. gpt-5-mini came out cheapest on both here ($0.013 per correct answer), but at only 85% accuracy. Cost per correct answer is the right lens, then you decide whether the missing accuracy matters for your use case.
• The cheap model was not the fast one. gpt-5-mini ran slower end to end than gpt-5.4 (median 65s vs 35s). A weaker model drives a longer agent loop, more search and reasoning steps to reach an answer, so cheaper did not mean faster here.
• Do not trust a framework’s cost number. Ours was off by up to 9x, while the real spread was roughly 20x. Take the cost from the trace, computed on real token usage, not the framework’s table.

Next steps

• Turn this into a tracked experiment so runs are versioned and comparable over time.
• Once you have picked a model, keep scoring its quality on live traffic with online evals.