Api V 2 Experiments List

GET

https://api.respan.ai/api/v2/experiments/

GET

/api/v2/experiments/

$ curl https://api.respan.ai/api/v2/experiments/ \
>      -H "Authorization: Bearer <token>"

200Retrieved

1 {
2   "count": 25,
3   "results": [
4     {
5       "dataset_name": "string",
6       "model": "string",
7       "created_at": "2024-01-15T09:30:00Z",
8       "tags": [
9         {
10           "id": "string",
11           "name": "string",
12           "color": "string",
13           "created_at": "2024-01-15T09:30:00Z",
14           "updated_at": "2024-01-15T09:30:00Z"
15         }
16       ],
17       "prompt_name": "string",
18       "evaluator_names": [
19         "string"
20       ],
21       "evaluator_scores": [
22         {}
23       ],
24       "id": "string",
25       "name": "string",
26       "dataset": "string",
27       "workflow_count": 1,
28       "status": "draft",
29       "progress": 1.1,
30       "started_at": "2024-01-15T09:30:00Z",
31       "completed_at": "2024-01-15T09:30:00Z",
32       "description": "string",
33       "is_starred": true
34     }
35   ],
36   "next": "string",
37   "previous": "string",
38   "total_count": 100,
39   "current_filters": {},
40   "filters_data": {}
41 }

GET: List experiments POST: Create and run an experiment workflow execution.

Design Philosophy: Dataset Log → [Arbitrary Workflow Tasks] → Run Eval Steps → Scores for Comparison

Workflow tasks can be arbitrary (any combination, end with whatever)
evaluator_ids field specifies which evaluators produce scores (preferred)
evaluator_slugs is a deprecated alias for evaluator_ids (backward compat)
Workflow step type eval is canonical; legacy evaluator is still accepted on input
Multiple evaluators can run for comprehensive comparison (e.g., quality, safety, cost)

This endpoint processes a dataset through a sequence of workflow tasks and generates traces for experiment tracking.

Request Body:

1 {
2     "dataset_id": "dataset-123",
3     "workflow": [
4         {
5             "type": "prompt",
6             "config": {
7                 "prompt_id": "my-prompt-slug"
8             }
9         },
10         {
11             "type": "completion",
12             "config": {
13                 "model": "gpt-4",
14                 "temperature": 0.7,
15                 "max_tokens": 1000,
16                 "top_p": 1.0,
17                 "frequency_penalty": 0,
18                 "presence_penalty": 0
19             }
20         }
21     ],
22     "evaluator_ids": ["abc123-evaluator-uuid", "def456-evaluator-uuid"],
23     "experiment_id": "exp-run-456",
24     "name": "My Experiment",
25     "description": "Testing GPT-4 with my prompt",
26     "span_workflow_name": "My Experiment",
27     "enable_tracing": true
28 }

Response:

1 {
2     "task_id": "celery-task-id",
3     "task_tracker_id": "tracker-id",
4     "status": "pending",
5     "message": "Workflow execution task started",
6     "experiment_id": "exp-run-456"
7 }

Workflow Types & Configurations:

Key Fields:

workflow: Array of workflow tasks (can be any combination)
evaluator_ids: Array of evaluator IDs (optional) - runs after workflow tasks complete; omit or pass [] to produce outputs with no scores
evaluator_slugs: Deprecated alias for evaluator_ids (still accepted)

Example Configurations:

workflow: [prompt, completion] + evaluator_ids: ["abc123"] ✓
workflow: [completion] + evaluator_ids: ["abc123", "def456"] ✓
workflow: [] + evaluator_ids: ["abc123"] ✗ (workflow is required)
workflow: [prompt, completion] + evaluator_ids: [] ✓ (runs the workflow, produces outputs with no scores)

Prompt Workflow (type: "prompt"):
- Variables are dynamically filled from dataset entries (don’t pass in config)
- Config fields:
  - prompt_id: Prompt identifier (required)
- Input: Dataset entry fields (mapped to prompt variables)
- Output: Rendered messages array
Completion Workflow (type: "completion"):
- Uses LLM configuration fields (temperature, max_tokens, etc.)
- Messages come from previous workflow step or dataset input field
- Config fields (all optional):
  - model: Model identifier (e.g., “gpt-4”, “claude-3-opus”)
  - temperature: Sampling temperature (0-2)
  - max_tokens: Maximum completion tokens
  - top_p: Nucleus sampling parameter
  - frequency_penalty: Frequency penalty (-2 to 2)
  - presence_penalty: Presence penalty (-2 to 2)
  - stop: Stop sequences (string or array)
  - n: Number of completions to generate
  - stream: Enable streaming (not recommended for experiments)
  - response_format: Response format (e.g., {“type”: “json_object”})
  - tools: Function calling tools array
  - tool_choice: Tool choice strategy
  - reasoning_effort: Reasoning effort for o1 models
- Input: Messages array (from “input” field in unified log format)
- Output: Response message object (stored in “output” field)
Eval Workflow (type: "eval"):
- Runs an evaluator on the unified log format (can be in workflow or via evaluator_slugs)
- Produces scores for experiment comparison
- Config fields:
  - evaluator_slug: Evaluator identifier (required)
- Input: Unified log format with input/output/metrics/metadata
- Output: Evaluation result with score

Note: Eval steps can be in workflow OR specified via evaluator_slugs field (recommended). The evaluator_slugs field at experiment level ensures all specified evaluators run after workflow tasks complete.

Condition Workflow (type: "condition"):
- Evaluates condition policies (future implementation)
- Config fields: TBD
Duplicate Workflow (type: "duplicate"):
- Passes through dataset entry’s input/output as-is (no LLM inference)
- Useful for batch evaluation on existing production data
- Config fields (all optional):
  - name: Display name for the workflow span
- Input: Dataset entry’s input field
- Output: Dataset entry’s output field (unchanged)
- Evaluators then run on this duplicated data

Authentication:

Supports both JWT (internal) and API Key (public) authentication

GET: List experiments POST: Create and run an experiment workflow execution. **Design Philosophy:** Dataset Log → [Arbitrary Workflow Tasks] → Run Eval Steps → Scores for Comparison - Workflow tasks can be arbitrary (any combination, end with whatever) - `evaluator_ids` field specifies which evaluators produce scores (preferred) - `evaluator_slugs` is a deprecated alias for `evaluator_ids` (backward compat) - Workflow step type `eval` is canonical; legacy `evaluator` is still accepted on input - Multiple evaluators can run for comprehensive comparison (e.g., quality, safety, cost) This endpoint processes a dataset through a sequence of workflow tasks and generates traces for experiment tracking. **Request Body:** ```json { "dataset_id": "dataset-123", "workflow": [ { "type": "prompt", "config": { "prompt_id": "my-prompt-slug" } }, { "type": "completion", "config": { "model": "gpt-4", "temperature": 0.7, "max_tokens": 1000, "top_p": 1.0, "frequency_penalty": 0, "presence_penalty": 0 } } ], "evaluator_ids": ["abc123-evaluator-uuid", "def456-evaluator-uuid"], "experiment_id": "exp-run-456", "name": "My Experiment", "description": "Testing GPT-4 with my prompt", "span_workflow_name": "My Experiment", "enable_tracing": true } ``` **Response:** ```json { "task_id": "celery-task-id", "task_tracker_id": "tracker-id", "status": "pending", "message": "Workflow execution task started", "experiment_id": "exp-run-456" } ``` **Workflow Types & Configurations:** **Key Fields:** - `workflow`: Array of workflow tasks (can be any combination) - `evaluator_ids`: Array of evaluator IDs (optional) - runs after workflow tasks complete; omit or pass `[]` to produce outputs with no scores - `evaluator_slugs`: Deprecated alias for `evaluator_ids` (still accepted) **Example Configurations:** - workflow: `[prompt, completion]` + evaluator_ids: `["abc123"]` ✓ - workflow: `[completion]` + evaluator_ids: `["abc123", "def456"]` ✓ - workflow: `[]` + evaluator_ids: `["abc123"]` ✗ (workflow is required) - workflow: `[prompt, completion]` + evaluator_ids: `[]` ✓ (runs the workflow, produces outputs with no scores) 1. **Prompt Workflow** (`type: "prompt"`): - Variables are dynamically filled from dataset entries (don't pass in config) - Config fields: - `prompt_id`: Prompt identifier (required) - Input: Dataset entry fields (mapped to prompt variables) - Output: Rendered messages array 2. **Completion Workflow** (`type: "completion"`): - Uses LLM configuration fields (temperature, max_tokens, etc.) - Messages come from previous workflow step or dataset input field - Config fields (all optional): - `model`: Model identifier (e.g., "gpt-4", "claude-3-opus") - `temperature`: Sampling temperature (0-2) - `max_tokens`: Maximum completion tokens - `top_p`: Nucleus sampling parameter - `frequency_penalty`: Frequency penalty (-2 to 2) - `presence_penalty`: Presence penalty (-2 to 2) - `stop`: Stop sequences (string or array) - `n`: Number of completions to generate - `stream`: Enable streaming (not recommended for experiments) - `response_format`: Response format (e.g., {"type": "json_object"}) - `tools`: Function calling tools array - `tool_choice`: Tool choice strategy - `reasoning_effort`: Reasoning effort for o1 models - Input: Messages array (from "input" field in unified log format) - Output: Response message object (stored in "output" field) 3. **Eval Workflow** (`type: "eval"`): - Runs an evaluator on the unified log format (can be in workflow or via evaluator_slugs) - Produces scores for experiment comparison - Config fields: - `evaluator_slug`: Evaluator identifier (required) - Input: Unified log format with input/output/metrics/metadata - Output: Evaluation result with score **Note:** Eval steps can be in workflow OR specified via `evaluator_slugs` field (recommended). The `evaluator_slugs` field at experiment level ensures all specified evaluators run after workflow tasks complete. 4. **Condition Workflow** (`type: "condition"`): - Evaluates condition policies (future implementation) - Config fields: TBD 5. **Duplicate Workflow** (`type: "duplicate"`): - Passes through dataset entry's input/output as-is (no LLM inference) - Useful for batch evaluation on existing production data - Config fields (all optional): - `name`: Display name for the workflow span - Input: Dataset entry's input field - Output: Dataset entry's output field (unchanged) - Evaluators then run on this duplicated data **Authentication:** - Supports both JWT (internal) and API Key (public) authentication

Authentication

AuthorizationBearer

JWT access token or Respan API key

Query parameters

pageintegerOptional

A page number within the paginated result set.

page_sizeintegerOptional

Number of results to return per page.

Response

countinteger

resultslist of objects

nextstring or nullformat: "uri"

previousstring or nullformat: "uri"

total_countinteger

current_filtersobject

Pydantic model for FilterParamDict. A dictionary that maps metric names to their filter parameters.

Each key is a metric name (str), and each value can be:

A single MetricFilterParamPydantic (one condition)
A List[MetricFilterParamPydantic] (multiple conditions for same metric)
A FilterBundlePydantic (nested filter bundle with connector)

Note: Uses extra=“allow” for dynamic metric name fields. The pydantic_extra annotation tells Pydantic what types to expect for extra fields, and generates typed additionalProperties in JSON Schema.

filters_dataobject

1	{
2	"dataset_id": "dataset-123",
3	"workflow": [
4	{
5	"type": "prompt",
6	"config": {
7	"prompt_id": "my-prompt-slug"
8	}
9	},
10	{
11	"type": "completion",
12	"config": {
13	"model": "gpt-4",
14	"temperature": 0.7,
15	"max_tokens": 1000,
16	"top_p": 1.0,
17	"frequency_penalty": 0,
18	"presence_penalty": 0
19	}
20	}
21	],
22	"evaluator_ids": ["abc123-evaluator-uuid", "def456-evaluator-uuid"],
23	"experiment_id": "exp-run-456",
24	"name": "My Experiment",
25	"description": "Testing GPT-4 with my prompt",
26	"span_workflow_name": "My Experiment",
27	"enable_tracing": true
28	}

1	{
2	"task_id": "celery-task-id",
3	"task_tracker_id": "tracker-id",
4	"status": "pending",
5	"message": "Workflow execution task started",
6	"experiment_id": "exp-run-456"
7	}