Retrieve evaluator | Respan Docs

Retrieves detailed information about a specific evaluator, including full configuration.

Authentication

All endpoints require API key authentication:

$ Authorization: Bearer YOUR_API_KEY

Path Parameters

Parameter	Type	Description
`evaluator_id`	string	The unique ID of the evaluator

Examples

Python

1 evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947"
2 url = f"https://api.respan.ai/api/evaluators/{evaluator_id}/"
3 headers = {
4     "Authorization": "Bearer YOUR_API_KEY"
5 }
6 
7 response = requests.get(url, headers=headers)
8 print(response.json())

cURL

$ curl -X GET "https://api.respan.ai/api/evaluators/0f4325f9-55ef-4c20-8abe-376694419947/" \
>   -H "Authorization: Bearer YOUR_API_KEY"

Response

Status: 200 OK

LLM Evaluator Response (New Format)

1 {
2   "id": "0f4325f9-55ef-4c20-8abe-376694419947",
3   "name": "Response Quality Evaluator",
4   "evaluator_slug": "response_quality_v2",
5   "type": "llm",
6   "score_value_type": "numerical",
7   "eval_class": "",
8   "description": "Evaluates response quality on a 1-5 scale",
9   "score_config": {
10     "min_score": 1,
11     "max_score": 5,
12     "choices": [
13       {"name": "Poor", "value": 1},
14       {"name": "Fair", "value": 2},
15       {"name": "Good", "value": 3},
16       {"name": "Great", "value": 4},
17       {"name": "Excellent", "value": 5}
18     ]
19   },
20   "passing_conditions": {
21     "primary_score": {
22       "operator": "gte",
23       "value": 3
24     }
25   },
26   "llm_config": {
27     "model": "gpt-4o-mini",
28     "evaluator_definition": "Rate the quality:\n<input>{{input}}</input>\n<output>{{output}}</output>",
29     "scoring_rubric": "1=Poor, 5=Excellent",
30     "temperature": 0.1,
31     "max_tokens": 200
32   },
33   "code_config": null,
34   "configurations": {},
35   "created_by": {
36     "first_name": "Respan",
37     "last_name": "Team",
38     "email": "admin@respan.ai"
39   },
40   "updated_by": {
41     "first_name": "Respan",
42     "last_name": "Team",
43     "email": "admin@respan.ai"
44   },
45   "created_at": "2025-09-11T09:43:55.858321Z",
46   "updated_at": "2025-09-11T09:43:55.858331Z",
47   "custom_required_fields": [],
48   "categorical_choices": null,
49   "starred": false,
50   "organization": 2,
51   "tags": []
52 }

LLM Evaluator Response (Legacy Format)

1 {
2   "id": "0f4325f9-legacy",
3   "name": "Response Quality Evaluator (Legacy)",
4   "evaluator_slug": "response_quality_v1",
5   "type": "llm",
6   "score_value_type": "numerical",
7   "eval_class": "",
8   "description": "Evaluates response quality on a 1-5 scale",
9   "configurations": {
10     "evaluator_definition": "Rate the response quality based on accuracy, relevance, and completeness.\n<llm_input>{{llm_input}}</llm_input>\n<llm_output>{{llm_output}}</llm_output>",
11     "scoring_rubric": "1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent",
12     "llm_engine": "gpt-4o-mini",
13     "model_options": {
14       "temperature": 0.1,
15       "max_tokens": 200
16     },
17     "min_score": 1.0,
18     "max_score": 5.0,
19     "passing_score": 3.0
20   },
21   "score_config": null,
22   "passing_conditions": null,
23   "llm_config": null,
24   "code_config": null,
25   "created_by": {
26     "first_name": "Respan",
27     "last_name": "Team",
28     "email": "admin@respan.ai"
29   },
30   "updated_by": {
31     "first_name": "Respan",
32     "last_name": "Team",
33     "email": "admin@respan.ai"
34   },
35   "created_at": "2025-09-11T09:43:55.858321Z",
36   "updated_at": "2025-09-11T09:43:55.858331Z",
37   "custom_required_fields": [],
38   "categorical_choices": null,
39   "starred": false,
40   "organization": 2,
41   "tags": []
42 }

Human Evaluator with LLM Assistance (New Format)

This example shows how a human evaluator can have LLM automation configured, demonstrating the decoupling of annotation method from evaluator type.

1 {
2   "id": "human-llm-123",
3   "name": "Human Review with AI Assistance",
4   "evaluator_slug": "human_ai_assist_v1",
5   "type": "human",
6   "score_value_type": "numerical",
7   "eval_class": "",
8   "description": "Human review with LLM-suggested scores",
9   "score_config": {
10     "min_score": 1,
11     "max_score": 5
12   },
13   "passing_conditions": {
14     "primary_score": {
15       "operator": "gte",
16       "value": 3
17     }
18   },
19   "llm_config": {
20     "model": "gpt-4o-mini",
21     "evaluator_definition": "Suggest a quality score for this response",
22     "temperature": 0.1
23   },
24   "code_config": null,
25   "configurations": {},
26   "categorical_choices": null,
27   "created_by": {
28     "first_name": "Respan",
29     "last_name": "Team",
30     "email": "admin@respan.ai"
31   },
32   "updated_by": {
33     "first_name": "Respan",
34     "last_name": "Team",
35     "email": "admin@respan.ai"
36   },
37   "created_at": "2025-09-11T09:44:00.000000Z",
38   "updated_at": "2025-09-11T09:44:00.000000Z",
39   "custom_required_fields": [],
40   "starred": false,
41   "organization": 2,
42   "tags": []
43 }

Human Single-Select Evaluator Response (Legacy Format)

1 {
2   "id": "cat-eval-123",
3   "name": "Content Quality Assessment",
4   "evaluator_slug": "content_quality_categorical",
5   "type": "human",
6   "score_value_type": "single_select",
7   "eval_class": "",
8   "description": "Human assessment of content quality with predefined categories",
9   "configurations": {},
10   "score_config": null,
11   "passing_conditions": null,
12   "llm_config": null,
13   "code_config": null,
14   "categorical_choices": [
15     { "name": "Excellent", "value": 5 },
16     { "name": "Good", "value": 4 },
17     { "name": "Average", "value": 3 },
18     { "name": "Poor", "value": 2 },
19     { "name": "Very Poor", "value": 1 }
20   ],
21   "created_by": {
22     "first_name": "Respan",
23     "last_name": "Team",
24     "email": "admin@respan.ai"
25   },
26   "updated_by": {
27     "first_name": "Respan",
28     "last_name": "Team",
29     "email": "admin@respan.ai"
30   },
31   "created_at": "2025-09-11T09:44:00.000000Z",
32   "updated_at": "2025-09-11T09:44:00.000000Z",
33   "custom_required_fields": [],
34   "starred": false,
35   "organization": 2,
36   "tags": []
37 }

Code Evaluator Response (New Format)

1 {
2   "id": "bool-eval-456",
3   "name": "Length Check",
4   "evaluator_slug": "length_check_v1",
5   "type": "code",
6   "score_value_type": "boolean",
7   "eval_class": "",
8   "description": "Checks if response is longer than 10 characters",
9   "score_config": {},
10   "passing_conditions": null,
11   "llm_config": null,
12   "code_config": {
13     "eval_code_snippet": "def main(eval_inputs):\n    output = eval_inputs.get('output', '')\n    return len(str(output)) > 10"
14   },
15   "configurations": {},
16   "categorical_choices": [],
17   "created_by": {
18     "first_name": "Respan",
19     "last_name": "Team",
20     "email": "admin@respan.ai"
21   },
22   "updated_by": {
23     "first_name": "Respan",
24     "last_name": "Team",
25     "email": "admin@respan.ai"
26   },
27   "created_at": "2025-09-11T09:45:00.000000Z",
28   "updated_at": "2025-09-11T09:45:00.000000Z",
29   "custom_required_fields": [],
30   "starred": false,
31   "organization": 2,
32   "tags": ["automation", "validation"]
33 }

Response Fields

New Format: Evaluators now include score_config, passing_conditions, llm_config, and code_config fields. These allow any evaluator type to have both LLM and code automation configured, decoupling annotation method from evaluator type.

Field	Type	Description
`id`	string	Unique evaluator identifier
`name`	string	Display name of the evaluator
`evaluator_slug`	string	URL-friendly identifier
`type`	string	Evaluator type: `llm`, `human`, or `code`
`score_value_type`	string	Score format: `numerical`, `boolean`, `percentage`, `single_select`, `multi_select`, `json`, `text`
`eval_class`	string	Pre-built template class (if used)
`description`	string	Description of the evaluator
`score_config`	object	New: Score type configuration (min/max, choices, etc.)
`passing_conditions`	object	New: Passing conditions using universal filter format
`llm_config`	object	New: LLM automation config (if configured)
`code_config`	object	New: Code automation config (if configured)
`configurations`	object	Legacy type-specific configuration settings
`categorical_choices`	array	Legacy choices (use `score_config.choices` in new format)
`created_by`	object	User who created the evaluator
`updated_by`	object	User who last updated the evaluator
`created_at`	string	ISO timestamp of creation
`updated_at`	string	ISO timestamp of last update
`custom_required_fields`	array	Additional required fields
`starred`	boolean	Whether the evaluator is starred
`organization`	integer	Organization ID
`tags`	array	Tags associated with the evaluator

Configuration Fields by Type (Legacy)

LLM Evaluators (`type: "llm"`)

Field	Type	Description
`evaluator_definition`	string	The evaluation prompt/instruction with template variables
`scoring_rubric`	string	Description of the scoring criteria
`llm_engine`	string	LLM model to use (e.g., “gpt-4o-mini”, “gpt-4o”)
`model_options`	object	LLM parameters like temperature, max_tokens
`min_score`	number	Minimum possible score
`max_score`	number	Maximum possible score
`passing_score`	number	Score threshold for passing

Code Evaluators (`type: "code"`)

Field	Type	Description
`eval_code_snippet`	string	Python code with evaluate() function

Human Evaluators (`type: "human"`)

No specific configuration fields
Use categorical_choices field when score_value_type is "categorical"

Error Responses

404 Not Found

1 {
2   "detail": "Not found."
3 }

401 Unauthorized

1 {
2   "detail": "Your API key is invalid or expired, please check your API key at https://platform.respan.ai/platform/api/api-keys"
3 }

403 Forbidden

1 {
2   "detail": "You do not have permission to access this evaluator."
3 }

Retrieves detailed information about a specific evaluator, including full configuration. ## Authentication All endpoints require API key authentication: ```bash Authorization: Bearer YOUR_API_KEY ``` ## Path Parameters | Parameter | Type | Description | |-----------|------|-------------| | `evaluator_id` | string | The unique ID of the evaluator | ## Examples ```python Python evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947" url = f"https://api.respan.ai/api/evaluators/{evaluator_id}/" headers = { "Authorization": "Bearer YOUR_API_KEY" } response = requests.get(url, headers=headers) print(response.json()) ``` ```bash cURL curl -X GET "https://api.respan.ai/api/evaluators/0f4325f9-55ef-4c20-8abe-376694419947/" \ -H "Authorization: Bearer YOUR_API_KEY" ``` ## Response **Status: 200 OK** ### LLM Evaluator Response (New Format) ```json { "id": "0f4325f9-55ef-4c20-8abe-376694419947", "name": "Response Quality Evaluator", "evaluator_slug": "response_quality_v2", "type": "llm", "score_value_type": "numerical", "eval_class": "", "description": "Evaluates response quality on a 1-5 scale", "score_config": { "min_score": 1, "max_score": 5, "choices": [ {"name": "Poor", "value": 1}, {"name": "Fair", "value": 2}, {"name": "Good", "value": 3}, {"name": "Great", "value": 4}, {"name": "Excellent", "value": 5} ] }, "passing_conditions": { "primary_score": { "operator": "gte", "value": 3 } }, "llm_config": { "model": "gpt-4o-mini", "evaluator_definition": "Rate the quality:\n<input>{{input}}</input>\n<output>{{output}}</output>", "scoring_rubric": "1=Poor, 5=Excellent", "temperature": 0.1, "max_tokens": 200 }, "code_config": null, "configurations": {}, "created_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "updated_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "created_at": "2025-09-11T09:43:55.858321Z", "updated_at": "2025-09-11T09:43:55.858331Z", "custom_required_fields": [], "categorical_choices": null, "starred": false, "organization": 2, "tags": [] } ``` ### LLM Evaluator Response (Legacy Format) ```json { "id": "0f4325f9-legacy", "name": "Response Quality Evaluator (Legacy)", "evaluator_slug": "response_quality_v1", "type": "llm", "score_value_type": "numerical", "eval_class": "", "description": "Evaluates response quality on a 1-5 scale", "configurations": { "evaluator_definition": "Rate the response quality based on accuracy, relevance, and completeness.\n<llm_input>{{llm_input}}</llm_input>\n<llm_output>{{llm_output}}</llm_output>", "scoring_rubric": "1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent", "llm_engine": "gpt-4o-mini", "model_options": { "temperature": 0.1, "max_tokens": 200 }, "min_score": 1.0, "max_score": 5.0, "passing_score": 3.0 }, "score_config": null, "passing_conditions": null, "llm_config": null, "code_config": null, "created_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "updated_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "created_at": "2025-09-11T09:43:55.858321Z", "updated_at": "2025-09-11T09:43:55.858331Z", "custom_required_fields": [], "categorical_choices": null, "starred": false, "organization": 2, "tags": [] } ``` ### Human Evaluator with LLM Assistance (New Format) <Note> This example shows how a **human** evaluator can have LLM automation configured, demonstrating the decoupling of annotation method from evaluator type. </Note> ```json { "id": "human-llm-123", "name": "Human Review with AI Assistance", "evaluator_slug": "human_ai_assist_v1", "type": "human", "score_value_type": "numerical", "eval_class": "", "description": "Human review with LLM-suggested scores", "score_config": { "min_score": 1, "max_score": 5 }, "passing_conditions": { "primary_score": { "operator": "gte", "value": 3 } }, "llm_config": { "model": "gpt-4o-mini", "evaluator_definition": "Suggest a quality score for this response", "temperature": 0.1 }, "code_config": null, "configurations": {}, "categorical_choices": null, "created_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "updated_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "created_at": "2025-09-11T09:44:00.000000Z", "updated_at": "2025-09-11T09:44:00.000000Z", "custom_required_fields": [], "starred": false, "organization": 2, "tags": [] } ``` ### Human Single-Select Evaluator Response (Legacy Format) ```json { "id": "cat-eval-123", "name": "Content Quality Assessment", "evaluator_slug": "content_quality_categorical", "type": "human", "score_value_type": "single_select", "eval_class": "", "description": "Human assessment of content quality with predefined categories", "configurations": {}, "score_config": null, "passing_conditions": null, "llm_config": null, "code_config": null, "categorical_choices": [ { "name": "Excellent", "value": 5 }, { "name": "Good", "value": 4 }, { "name": "Average", "value": 3 }, { "name": "Poor", "value": 2 }, { "name": "Very Poor", "value": 1 } ], "created_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "updated_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "created_at": "2025-09-11T09:44:00.000000Z", "updated_at": "2025-09-11T09:44:00.000000Z", "custom_required_fields": [], "starred": false, "organization": 2, "tags": [] } ``` ### Code Evaluator Response (New Format) ```json { "id": "bool-eval-456", "name": "Length Check", "evaluator_slug": "length_check_v1", "type": "code", "score_value_type": "boolean", "eval_class": "", "description": "Checks if response is longer than 10 characters", "score_config": {}, "passing_conditions": null, "llm_config": null, "code_config": { "eval_code_snippet": "def main(eval_inputs):\n output = eval_inputs.get('output', '')\n return len(str(output)) > 10" }, "configurations": {}, "categorical_choices": [], "created_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "updated_by": { "first_name": "Respan", "last_name": "Team", "email": "admin@respan.ai" }, "created_at": "2025-09-11T09:45:00.000000Z", "updated_at": "2025-09-11T09:45:00.000000Z", "custom_required_fields": [], "starred": false, "organization": 2, "tags": ["automation", "validation"] } ``` ## Response Fields <Note> **New Format**: Evaluators now include `score_config`, `passing_conditions`, `llm_config`, and `code_config` fields. These allow any evaluator type to have both LLM and code automation configured, decoupling annotation method from evaluator type. </Note> | Field | Type | Description | |-------|------|-------------| | `id` | string | Unique evaluator identifier | | `name` | string | Display name of the evaluator | | `evaluator_slug` | string | URL-friendly identifier | | `type` | string | Evaluator type: `llm`, `human`, or `code` | | `score_value_type` | string | Score format: `numerical`, `boolean`, `percentage`, `single_select`, `multi_select`, `json`, `text` | | `eval_class` | string | Pre-built template class (if used) | | `description` | string | Description of the evaluator | | `score_config` | object | **New**: Score type configuration (min/max, choices, etc.) | | `passing_conditions` | object | **New**: Passing conditions using universal filter format | | `llm_config` | object | **New**: LLM automation config (if configured) | | `code_config` | object | **New**: Code automation config (if configured) | | `configurations` | object | Legacy type-specific configuration settings | | `categorical_choices` | array | Legacy choices (use `score_config.choices` in new format) | | `created_by` | object | User who created the evaluator | | `updated_by` | object | User who last updated the evaluator | | `created_at` | string | ISO timestamp of creation | | `updated_at` | string | ISO timestamp of last update | | `custom_required_fields` | array | Additional required fields | | `starred` | boolean | Whether the evaluator is starred | | `organization` | integer | Organization ID | | `tags` | array | Tags associated with the evaluator | ## Configuration Fields by Type (Legacy) ### LLM Evaluators (`type: "llm"`) | Field | Type | Description | |-------|------|-------------| | `evaluator_definition` | string | The evaluation prompt/instruction with template variables | | `scoring_rubric` | string | Description of the scoring criteria | | `llm_engine` | string | LLM model to use (e.g., "gpt-4o-mini", "gpt-4o") | | `model_options` | object | LLM parameters like temperature, max_tokens | | `min_score` | number | Minimum possible score | | `max_score` | number | Maximum possible score | | `passing_score` | number | Score threshold for passing | ### Code Evaluators (`type: "code"`) | Field | Type | Description | |-------|------|-------------| | `eval_code_snippet` | string | Python code with evaluate() function | ### Human Evaluators (`type: "human"`) - No specific configuration fields - Use `categorical_choices` field when `score_value_type` is `"categorical"` ## Error Responses ### 404 Not Found ```json { "detail": "Not found." } ``` ### 401 Unauthorized ```json { "detail": "Your API key is invalid or expired, please check your API key at https://platform.respan.ai/platform/api/api-keys" } ``` ### 403 Forbidden ```json { "detail": "You do not have permission to access this evaluator." } ```

Authentication

AuthorizationBearer

API key authentication. Get your API key from https://platform.respan.ai/platform/api-keys

Path parameters

evaluator_idstringRequired

Evaluator Id

Response

Successful response for Retrieve evaluator

idstring

namestring

evaluator_slugstring

typestring

score_value_typestring

eval_classstring

descriptionstring

score_configobject

passing_conditionsobject

llm_configobject

code_configstring

configurationsobject

created_byobject

updated_byobject

created_atstring

updated_atstring

custom_required_fieldslist of strings

categorical_choiceslist of objects or null

starredboolean

organizationinteger

tagslist of strings

Errors

401

Unauthorized Error

Retrieves detailed information about a specific evaluator, including full configuration.

Authentication

All endpoints require API key authentication:

$ Authorization: Bearer YOUR_API_KEY

Path Parameters

Parameter	Type	Description
`evaluator_id`	string	The unique ID of the evaluator

Examples

Python

1 evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947"
2 url = f"https://api.respan.ai/api/evaluators/{evaluator_id}/"
3 headers = {
4     "Authorization": "Bearer YOUR_API_KEY"
5 }
6 
7 response = requests.get(url, headers=headers)
8 print(response.json())

cURL

$ curl -X GET "https://api.respan.ai/api/evaluators/0f4325f9-55ef-4c20-8abe-376694419947/" \
>   -H "Authorization: Bearer YOUR_API_KEY"

Response

Status: 200 OK

LLM Evaluator Response (New Format)

1 {
2   "id": "0f4325f9-55ef-4c20-8abe-376694419947",
3   "name": "Response Quality Evaluator",
4   "evaluator_slug": "response_quality_v2",
5   "type": "llm",
6   "score_value_type": "numerical",
7   "eval_class": "",
8   "description": "Evaluates response quality on a 1-5 scale",
9   "score_config": {
10     "min_score": 1,
11     "max_score": 5,
12     "choices": [
13       {"name": "Poor", "value": 1},
14       {"name": "Fair", "value": 2},
15       {"name": "Good", "value": 3},
16       {"name": "Great", "value": 4},
17       {"name": "Excellent", "value": 5}
18     ]
19   },
20   "passing_conditions": {
21     "primary_score": {
22       "operator": "gte",
23       "value": 3
24     }
25   },
26   "llm_config": {
27     "model": "gpt-4o-mini",
28     "evaluator_definition": "Rate the quality:\n<input>{{input}}</input>\n<output>{{output}}</output>",
29     "scoring_rubric": "1=Poor, 5=Excellent",
30     "temperature": 0.1,
31     "max_tokens": 200
32   },
33   "code_config": null,
34   "configurations": {},
35   "created_by": {
36     "first_name": "Respan",
37     "last_name": "Team",
38     "email": "admin@respan.ai"
39   },
40   "updated_by": {
41     "first_name": "Respan",
42     "last_name": "Team",
43     "email": "admin@respan.ai"
44   },
45   "created_at": "2025-09-11T09:43:55.858321Z",
46   "updated_at": "2025-09-11T09:43:55.858331Z",
47   "custom_required_fields": [],
48   "categorical_choices": null,
49   "starred": false,
50   "organization": 2,
51   "tags": []
52 }

LLM Evaluator Response (Legacy Format)

1 {
2   "id": "0f4325f9-legacy",
3   "name": "Response Quality Evaluator (Legacy)",
4   "evaluator_slug": "response_quality_v1",
5   "type": "llm",
6   "score_value_type": "numerical",
7   "eval_class": "",
8   "description": "Evaluates response quality on a 1-5 scale",
9   "configurations": {
10     "evaluator_definition": "Rate the response quality based on accuracy, relevance, and completeness.\n<llm_input>{{llm_input}}</llm_input>\n<llm_output>{{llm_output}}</llm_output>",
11     "scoring_rubric": "1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent",
12     "llm_engine": "gpt-4o-mini",
13     "model_options": {
14       "temperature": 0.1,
15       "max_tokens": 200
16     },
17     "min_score": 1.0,
18     "max_score": 5.0,
19     "passing_score": 3.0
20   },
21   "score_config": null,
22   "passing_conditions": null,
23   "llm_config": null,
24   "code_config": null,
25   "created_by": {
26     "first_name": "Respan",
27     "last_name": "Team",
28     "email": "admin@respan.ai"
29   },
30   "updated_by": {
31     "first_name": "Respan",
32     "last_name": "Team",
33     "email": "admin@respan.ai"
34   },
35   "created_at": "2025-09-11T09:43:55.858321Z",
36   "updated_at": "2025-09-11T09:43:55.858331Z",
37   "custom_required_fields": [],
38   "categorical_choices": null,
39   "starred": false,
40   "organization": 2,
41   "tags": []
42 }

Human Evaluator with LLM Assistance (New Format)

This example shows how a human evaluator can have LLM automation configured, demonstrating the decoupling of annotation method from evaluator type.

1 {
2   "id": "human-llm-123",
3   "name": "Human Review with AI Assistance",
4   "evaluator_slug": "human_ai_assist_v1",
5   "type": "human",
6   "score_value_type": "numerical",
7   "eval_class": "",
8   "description": "Human review with LLM-suggested scores",
9   "score_config": {
10     "min_score": 1,
11     "max_score": 5
12   },
13   "passing_conditions": {
14     "primary_score": {
15       "operator": "gte",
16       "value": 3
17     }
18   },
19   "llm_config": {
20     "model": "gpt-4o-mini",
21     "evaluator_definition": "Suggest a quality score for this response",
22     "temperature": 0.1
23   },
24   "code_config": null,
25   "configurations": {},
26   "categorical_choices": null,
27   "created_by": {
28     "first_name": "Respan",
29     "last_name": "Team",
30     "email": "admin@respan.ai"
31   },
32   "updated_by": {
33     "first_name": "Respan",
34     "last_name": "Team",
35     "email": "admin@respan.ai"
36   },
37   "created_at": "2025-09-11T09:44:00.000000Z",
38   "updated_at": "2025-09-11T09:44:00.000000Z",
39   "custom_required_fields": [],
40   "starred": false,
41   "organization": 2,
42   "tags": []
43 }

Human Single-Select Evaluator Response (Legacy Format)

1 {
2   "id": "cat-eval-123",
3   "name": "Content Quality Assessment",
4   "evaluator_slug": "content_quality_categorical",
5   "type": "human",
6   "score_value_type": "single_select",
7   "eval_class": "",
8   "description": "Human assessment of content quality with predefined categories",
9   "configurations": {},
10   "score_config": null,
11   "passing_conditions": null,
12   "llm_config": null,
13   "code_config": null,
14   "categorical_choices": [
15     { "name": "Excellent", "value": 5 },
16     { "name": "Good", "value": 4 },
17     { "name": "Average", "value": 3 },
18     { "name": "Poor", "value": 2 },
19     { "name": "Very Poor", "value": 1 }
20   ],
21   "created_by": {
22     "first_name": "Respan",
23     "last_name": "Team",
24     "email": "admin@respan.ai"
25   },
26   "updated_by": {
27     "first_name": "Respan",
28     "last_name": "Team",
29     "email": "admin@respan.ai"
30   },
31   "created_at": "2025-09-11T09:44:00.000000Z",
32   "updated_at": "2025-09-11T09:44:00.000000Z",
33   "custom_required_fields": [],
34   "starred": false,
35   "organization": 2,
36   "tags": []
37 }

Code Evaluator Response (New Format)

1 {
2   "id": "bool-eval-456",
3   "name": "Length Check",
4   "evaluator_slug": "length_check_v1",
5   "type": "code",
6   "score_value_type": "boolean",
7   "eval_class": "",
8   "description": "Checks if response is longer than 10 characters",
9   "score_config": {},
10   "passing_conditions": null,
11   "llm_config": null,
12   "code_config": {
13     "eval_code_snippet": "def main(eval_inputs):\n    output = eval_inputs.get('output', '')\n    return len(str(output)) > 10"
14   },
15   "configurations": {},
16   "categorical_choices": [],
17   "created_by": {
18     "first_name": "Respan",
19     "last_name": "Team",
20     "email": "admin@respan.ai"
21   },
22   "updated_by": {
23     "first_name": "Respan",
24     "last_name": "Team",
25     "email": "admin@respan.ai"
26   },
27   "created_at": "2025-09-11T09:45:00.000000Z",
28   "updated_at": "2025-09-11T09:45:00.000000Z",
29   "custom_required_fields": [],
30   "starred": false,
31   "organization": 2,
32   "tags": ["automation", "validation"]
33 }

Response Fields

Field	Type	Description
`id`	string	Unique evaluator identifier
`name`	string	Display name of the evaluator
`evaluator_slug`	string	URL-friendly identifier
`type`	string	Evaluator type: `llm`, `human`, or `code`
`score_value_type`	string	Score format: `numerical`, `boolean`, `percentage`, `single_select`, `multi_select`, `json`, `text`
`eval_class`	string	Pre-built template class (if used)
`description`	string	Description of the evaluator
`score_config`	object	New: Score type configuration (min/max, choices, etc.)
`passing_conditions`	object	New: Passing conditions using universal filter format
`llm_config`	object	New: LLM automation config (if configured)
`code_config`	object	New: Code automation config (if configured)
`configurations`	object	Legacy type-specific configuration settings
`categorical_choices`	array	Legacy choices (use `score_config.choices` in new format)
`created_by`	object	User who created the evaluator
`updated_by`	object	User who last updated the evaluator
`created_at`	string	ISO timestamp of creation
`updated_at`	string	ISO timestamp of last update
`custom_required_fields`	array	Additional required fields
`starred`	boolean	Whether the evaluator is starred
`organization`	integer	Organization ID
`tags`	array	Tags associated with the evaluator

Configuration Fields by Type (Legacy)

LLM Evaluators (`type: "llm"`)

Field	Type	Description
`evaluator_definition`	string	The evaluation prompt/instruction with template variables
`scoring_rubric`	string	Description of the scoring criteria
`llm_engine`	string	LLM model to use (e.g., “gpt-4o-mini”, “gpt-4o”)
`model_options`	object	LLM parameters like temperature, max_tokens
`min_score`	number	Minimum possible score
`max_score`	number	Maximum possible score
`passing_score`	number	Score threshold for passing

Code Evaluators (`type: "code"`)

Field	Type	Description
`eval_code_snippet`	string	Python code with evaluate() function

Human Evaluators (`type: "human"`)

No specific configuration fields
Use categorical_choices field when score_value_type is "categorical"

Error Responses

404 Not Found

1 {
2   "detail": "Not found."
3 }

401 Unauthorized

1 {
2   "detail": "Your API key is invalid or expired, please check your API key at https://platform.respan.ai/platform/api/api-keys"
3 }

403 Forbidden

1 {
2   "detail": "You do not have permission to access this evaluator."
3 }

1	evaluator_id = "0f4325f9-55ef-4c20-8abe-376694419947"
2	url = f"https://api.respan.ai/api/evaluators/{evaluator_id}/"
3	headers = {
4	"Authorization": "Bearer YOUR_API_KEY"
5	}
6
7	response = requests.get(url, headers=headers)
8	print(response.json())

$	curl -X GET "https://api.respan.ai/api/evaluators/0f4325f9-55ef-4c20-8abe-376694419947/" \
>	-H "Authorization: Bearer YOUR_API_KEY"

1	{
2	"id": "0f4325f9-55ef-4c20-8abe-376694419947",
3	"name": "Response Quality Evaluator",
4	"evaluator_slug": "response_quality_v2",
5	"type": "llm",
6	"score_value_type": "numerical",
7	"eval_class": "",
8	"description": "Evaluates response quality on a 1-5 scale",
9	"score_config": {
10	"min_score": 1,
11	"max_score": 5,
12	"choices": [
13	{"name": "Poor", "value": 1},
14	{"name": "Fair", "value": 2},
15	{"name": "Good", "value": 3},
16	{"name": "Great", "value": 4},
17	{"name": "Excellent", "value": 5}
18	]
19	},
20	"passing_conditions": {
21	"primary_score": {
22	"operator": "gte",
23	"value": 3
24	}
25	},
26	"llm_config": {
27	"model": "gpt-4o-mini",
28	"evaluator_definition": "Rate the quality:\n<input>{{input}}</input>\n<output>{{output}}</output>",
29	"scoring_rubric": "1=Poor, 5=Excellent",
30	"temperature": 0.1,
31	"max_tokens": 200
32	},
33	"code_config": null,
34	"configurations": {},
35	"created_by": {
36	"first_name": "Respan",
37	"last_name": "Team",
38	"email": "admin@respan.ai"
39	},
40	"updated_by": {
41	"first_name": "Respan",
42	"last_name": "Team",
43	"email": "admin@respan.ai"
44	},
45	"created_at": "2025-09-11T09:43:55.858321Z",
46	"updated_at": "2025-09-11T09:43:55.858331Z",
47	"custom_required_fields": [],
48	"categorical_choices": null,
49	"starred": false,
50	"organization": 2,
51	"tags": []
52	}

1	{
2	"id": "0f4325f9-legacy",
3	"name": "Response Quality Evaluator (Legacy)",
4	"evaluator_slug": "response_quality_v1",
5	"type": "llm",
6	"score_value_type": "numerical",
7	"eval_class": "",
8	"description": "Evaluates response quality on a 1-5 scale",
9	"configurations": {
10	"evaluator_definition": "Rate the response quality based on accuracy, relevance, and completeness.\n<llm_input>{{llm_input}}</llm_input>\n<llm_output>{{llm_output}}</llm_output>",
11	"scoring_rubric": "1=Poor, 2=Fair, 3=Good, 4=Very Good, 5=Excellent",
12	"llm_engine": "gpt-4o-mini",
13	"model_options": {
14	"temperature": 0.1,
15	"max_tokens": 200
16	},
17	"min_score": 1.0,
18	"max_score": 5.0,
19	"passing_score": 3.0
20	},
21	"score_config": null,
22	"passing_conditions": null,
23	"llm_config": null,
24	"code_config": null,
25	"created_by": {
26	"first_name": "Respan",
27	"last_name": "Team",
28	"email": "admin@respan.ai"
29	},
30	"updated_by": {
31	"first_name": "Respan",
32	"last_name": "Team",
33	"email": "admin@respan.ai"
34	},
35	"created_at": "2025-09-11T09:43:55.858321Z",
36	"updated_at": "2025-09-11T09:43:55.858331Z",
37	"custom_required_fields": [],
38	"categorical_choices": null,
39	"starred": false,
40	"organization": 2,
41	"tags": []
42	}

1	{
2	"id": "human-llm-123",
3	"name": "Human Review with AI Assistance",
4	"evaluator_slug": "human_ai_assist_v1",
5	"type": "human",
6	"score_value_type": "numerical",
7	"eval_class": "",
8	"description": "Human review with LLM-suggested scores",
9	"score_config": {
10	"min_score": 1,
11	"max_score": 5
12	},
13	"passing_conditions": {
14	"primary_score": {
15	"operator": "gte",
16	"value": 3
17	}
18	},
19	"llm_config": {
20	"model": "gpt-4o-mini",
21	"evaluator_definition": "Suggest a quality score for this response",
22	"temperature": 0.1
23	},
24	"code_config": null,
25	"configurations": {},
26	"categorical_choices": null,
27	"created_by": {
28	"first_name": "Respan",
29	"last_name": "Team",
30	"email": "admin@respan.ai"
31	},
32	"updated_by": {
33	"first_name": "Respan",
34	"last_name": "Team",
35	"email": "admin@respan.ai"
36	},
37	"created_at": "2025-09-11T09:44:00.000000Z",
38	"updated_at": "2025-09-11T09:44:00.000000Z",
39	"custom_required_fields": [],
40	"starred": false,
41	"organization": 2,
42	"tags": []
43	}

1	{
2	"id": "cat-eval-123",
3	"name": "Content Quality Assessment",
4	"evaluator_slug": "content_quality_categorical",
5	"type": "human",
6	"score_value_type": "single_select",
7	"eval_class": "",
8	"description": "Human assessment of content quality with predefined categories",
9	"configurations": {},
10	"score_config": null,
11	"passing_conditions": null,
12	"llm_config": null,
13	"code_config": null,
14	"categorical_choices": [
15	{ "name": "Excellent", "value": 5 },
16	{ "name": "Good", "value": 4 },
17	{ "name": "Average", "value": 3 },
18	{ "name": "Poor", "value": 2 },
19	{ "name": "Very Poor", "value": 1 }
20	],
21	"created_by": {
22	"first_name": "Respan",
23	"last_name": "Team",
24	"email": "admin@respan.ai"
25	},
26	"updated_by": {
27	"first_name": "Respan",
28	"last_name": "Team",
29	"email": "admin@respan.ai"
30	},
31	"created_at": "2025-09-11T09:44:00.000000Z",
32	"updated_at": "2025-09-11T09:44:00.000000Z",
33	"custom_required_fields": [],
34	"starred": false,
35	"organization": 2,
36	"tags": []
37	}

1	{
2	"id": "bool-eval-456",
3	"name": "Length Check",
4	"evaluator_slug": "length_check_v1",
5	"type": "code",
6	"score_value_type": "boolean",
7	"eval_class": "",
8	"description": "Checks if response is longer than 10 characters",
9	"score_config": {},
10	"passing_conditions": null,
11	"llm_config": null,
12	"code_config": {
13	"eval_code_snippet": "def main(eval_inputs):\n output = eval_inputs.get('output', '')\n return len(str(output)) > 10"
14	},
15	"configurations": {},
16	"categorical_choices": [],
17	"created_by": {
18	"first_name": "Respan",
19	"last_name": "Team",
20	"email": "admin@respan.ai"
21	},
22	"updated_by": {
23	"first_name": "Respan",
24	"last_name": "Team",
25	"email": "admin@respan.ai"
26	},
27	"created_at": "2025-09-11T09:45:00.000000Z",
28	"updated_at": "2025-09-11T09:45:00.000000Z",
29	"custom_required_fields": [],
30	"starred": false,
31	"organization": 2,
32	"tags": ["automation", "validation"]
33	}

1	{
2	"detail": "Your API key is invalid or expired, please check your API key at https://platform.respan.ai/platform/api/api-keys"
3	}

1	{
2	"detail": "You do not have permission to access this evaluator."
3	}

Authentication

Path Parameters

Examples

Response

LLM Evaluator Response (New Format)

LLM Evaluator Response (Legacy Format)

Human Evaluator with LLM Assistance (New Format)

Human Single-Select Evaluator Response (Legacy Format)

Code Evaluator Response (New Format)

Response Fields

Configuration Fields by Type (Legacy)

LLM Evaluators (type: "llm")

Code Evaluators (type: "code")

Human Evaluators (type: "human")

Error Responses

404 Not Found

401 Unauthorized

403 Forbidden

Authentication

Path parameters

Response

Errors

Authentication

Path Parameters

Examples

Response

LLM Evaluator Response (New Format)

LLM Evaluator Response (Legacy Format)

Human Evaluator with LLM Assistance (New Format)

Human Single-Select Evaluator Response (Legacy Format)

Code Evaluator Response (New Format)

Response Fields

Configuration Fields by Type (Legacy)

LLM Evaluators (type: "llm")

Code Evaluators (type: "code")

Human Evaluators (type: "human")

Error Responses

404 Not Found

401 Unauthorized

403 Forbidden

LLM Evaluators (`type: "llm"`)

Code Evaluators (`type: "code"`)

Human Evaluators (`type: "human"`)

LLM Evaluators (`type: "llm"`)

Code Evaluators (`type: "code"`)

Human Evaluators (`type: "human"`)