EvaluationsDatasets

Datasets

Build the set of test cases your experiments and evaluators run on.

A dataset is a collection of test cases you run evaluations against. Each row is one test case. Before you build one, it helps to know what a row holds and which fields you actually need, because that depends on what you are trying to evaluate.

What is a dataset?

A dataset is a table of rows, where each row represents a single test case. You use datasets to run experiments: Respan takes each row, produces an output (or reuses one you already have), and scores it with your evaluators.

You can build a dataset from your production logs or from a CSV file. Either way, the rows share the same shape.

What a row holds

Every row has three fields that matter for evaluation:

FieldRequiredWhat it holds
inputYesThe test case fed into your prompt or model. Stored as JSON, so it can hold a single value or several named variables.
expected_outputNoThe reference or golden answer. Graders can compare a result against it using {{expected_output}}.
outputNoA response already produced for this input. Lets you score existing outputs without generating new ones.

When you build a dataset by sampling production logs, each row also carries the original request’s telemetry (model, cost, latency, tokens) and any scores it already received. Rows you bring in from a CSV carry only the fields you map.

Which setup do you need?

When you create an experiment, you pick a Task type: Prompt, Model, or Dataset outputs. That choice decides where each row’s output comes from, and therefore which columns your dataset needs.

ColumnPromptModelDataset outputs
inputRequired, as prompt variablesRequired, as the model requestOptional, for context
outputGenerated during the runGenerated during the runRequired
expected_outputOptionalOptionalOptional

Across all three, expected_output is optional: add it for graders that compare against a known answer, and skip it for reference-free checks such as tone or format. Each setup has its own walkthrough:

Datasets for prompt optimization

Score a saved prompt template. The dataset supplies input variables; Respan generates the output.

Datasets for model comparison

Score a raw model with no prompt in the loop. The input is the request itself.

Datasets for production data

Score answers you already have. Each row supplies its own output — nothing is generated.

The example used below

Every setup walkthrough runs on the same tiny dataset — “world capitals” — so you can see how only the setup changes. Six rows, each a question and its correct answer:

inputexpected_output
{ "question": "What is the capital of France?" }Paris
{ "question": "What is the capital of Japan?" }Tokyo
{ "question": "What is the capital of Australia?" }Canberra
……

The same Correctness evaluator scores every run: it gives 1 when the output matches expected_output and 0 when it does not. The Datasets for production data walkthrough adds a stored output to each row (with two deliberate mistakes) so you can watch the evaluator catch them.

Add rows to a dataset

Click New Dataset, give it a name, and you land on an empty dataset with two ways to add rows (also available later from the Insert rows menu):

  • Insert by sampling pulls rows straight from your production request logs using filters, a time range, and a sampling percentage. Sampled rows arrive with their input, output, and telemetry already filled in, so it is the fastest way to test against real traffic.
  • Insert from CSV uploads curated or golden cases. You map each column to a dataset field (input, expected_output, output) and can import up to 500 rows at a time. Mapping several columns to input wraps them into one JSON object keyed by column name.

Each setup page walks through both methods for its own field requirements: prompt optimization, model comparison, and production data.

You can also skip the empty step: create a dataset already populated by sampling your logs as you create it, or by duplicating an existing dataset.

CSV template (Google Drive)

Next steps