Create an evaluator | Respan Docs

curl -X POST https://api.respan.ai/api/evaluators/ \
     -H "Authorization: Bearer <respanApiKey>" \
     -H "Content-Type: application/json" \
     -d '{
  "name": "Response Quality",
  "evaluator_slug": "response_quality",
  "type": "llm",
  "score_value_type": "numerical",
  "eval_class": "keywordsai_custom_llm",
  "score_config": {
    "min_score": 1,
    "max_score": 5
  },
  "passing_conditions": {
    "primary_score": {
      "operator": "gte",
      "value": 3
    }
  },
  "llm_config": {
    "model": "gpt-4o-mini",
    "evaluator_definition": "Rate the quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
    "scoring_rubric": "1=Poor, 5=Excellent"
  }
}'

{
  "id": "evl_550e8400",
  "name": "Response Quality",
  "type": "llm",
  "score_value_type": "numerical",
  "version_id": "evlv_550e8400",
  "version": 1,
  "is_read_only": false,
  "version_description": "Tightened rubric wording",
  "evaluator_slug": "response_quality",
  "eval_class": "keywordsai_custom_llm",
  "description": "Grades whether the response is accurate and complete.",
  "score_config": {
    "min_score": 1,
    "max_score": 5,
    "choices": [
      {
        "name": "Excellent",
        "value": "excellent"
      }
    ]
  },
  "passing_conditions": {
    "primary_score": {
      "operator": "gte",
      "value": 3
    }
  },
  "llm_config": {
    "model": "gpt-4o-mini",
    "evaluator_definition": "Rate the response quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
    "scoring_rubric": "1=Poor, 5=Excellent",
    "temperature": 0.1,
    "max_tokens": 200,
    "top_p": 1.1,
    "frequency_penalty": 1.1,
    "presence_penalty": 1.1,
    "stop": [
      "string"
    ],
    "response_format": {},
    "tools": [
      {}
    ],
    "verbosity": "string"
  },
  "code_config": {
    "eval_code_snippet": "def main(eval_inputs):\n    return 1 if eval_inputs.get('output') else 0"
  },
  "configurations": {},
  "categorical_choices": [
    {
      "name": "Excellent",
      "value": "excellent"
    }
  ],
  "starred": false,
  "created_at": "2024-01-15T09:30:00Z",
  "updated_at": "2024-01-15T09:30:00Z",
  "created_by": {
    "id": 1,
    "first_name": "string",
    "last_name": "string",
    "email": "string"
  },
  "updated_by": {
    "id": 1,
    "first_name": "string",
    "last_name": "string",
    "email": "string"
  },
  "editor": {
    "id": 1,
    "first_name": "string",
    "last_name": "string",
    "email": "string"
  },
  "tags": [
    {
      "id": 1,
      "name": "string",
      "color": "string"
    }
  ],
  "is_public": true
}

Create a grader/evaluator. Current public evaluator types are llm, human, and code; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (score_config, passing_conditions, llm_config, code_config) instead of relying on legacy configurations.

Create a grader/evaluator. Current public evaluator types are `llm`, `human`, and `code`; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (`score_config`, `passing_conditions`, `llm_config`, `code_config`) instead of relying on legacy `configurations`.

Authentication

AuthorizationBearer

Use your Respan API key for Respan API authentication. Enter only the Respan API key value; clients send Authorization: Bearer <RESPAN_API_KEY>. For /api/responses, provider credentials such as Perplexity, OpenAI, or Azure OpenAI go in Settings -> Providers or respan_params.credential_override in the request body, not in this authentication field.

AuthorizationBearer

Use a dashboard JWT only for dashboard-authenticated endpoints. Respan API-key endpoints use the respanApiKey auth field instead.

Request

This endpoint expects an object.

namestringRequired

evaluator_slugstringOptional

Organization-scoped evaluator slug.

typeenumOptional

Allowed values:

score_value_typeenumOptional

eval_classenumOptional

Optional pre-built evaluator template.

Allowed values:

descriptionstring or nullOptional

score_configobjectOptional

Score configuration. For numerical/percentage scores, use min_score and max_score. For select scores, use choices.

passing_conditionsobject or nullOptional

Passing conditions in the standard Respan filter format.

llm_configobject or nullOptional

LLM grader configuration. The backend validates this against the selected evaluator form.

code_configobject or nullOptional

Code grader configuration.

configurationsmap from strings to anyOptional

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

categorical_choiceslist of objectsOptional

starredbooleanOptional

Response

Created evaluator.

idstring

Stable evaluator ID shared by all versions.

namestring

typeenum

score_value_typeenum

version_idstringOptional

Unique ID for this evaluator version.

versionintegerOptional

Version number. The first draft is version 0.

is_read_onlybooleanOptional

Whether this version is committed and immutable.

version_descriptionstringOptional

Commit message for this version.

evaluator_slugstringOptional

Organization-scoped evaluator slug.

eval_classenumOptional

Optional pre-built evaluator template.

descriptionstring or nullOptional

score_configobjectOptional

Score configuration. For numerical/percentage scores, use min_score and max_score. For select scores, use choices.

passing_conditionsobject or nullOptional

Passing conditions in the standard Respan filter format.

llm_configobject or nullOptional

LLM grader configuration. The backend validates this against the selected evaluator form.

code_configobject or nullOptional

Code grader configuration.

configurationsmap from strings to anyOptional

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

categorical_choiceslist of objects or nullOptional

starredbooleanOptional

created_atdatetimeOptional

updated_atdatetimeOptional

created_byobjectOptional

updated_byobjectOptional

editorobjectOptional

tagslist of objectsOptional

is_publicbooleanOptionalRead-only

Whether this is a Respan-managed public evaluator.

Errors

400

Bad Request Error

401

Unauthorized Error