Create an evaluator | Respan Docs

curl -X POST https://api.respan.ai/api/evaluators/ \
     -H "Authorization: Bearer sk_live_xxxxx" \
     -H "Content-Type: application/json" \
     -d '{
  "name": "Response Quality",
  "type": "llm",
  "score_value_type": "numerical",
  "evaluator_slug": "response_quality",
  "eval_class": "keywordsai_custom_llm",
  "score_config": {
    "min_score": 1,
    "max_score": 5
  },
  "passing_conditions": {
    "primary_score": {
      "operator": "gte",
      "value": 3
    }
  },
  "llm_config": {
    "model": "gpt-4o-mini",
    "evaluator_definition": "Rate the quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
    "scoring_rubric": "1=Poor, 5=Excellent"
  }
}'

{
  "id": "evl_550e8400",
  "name": "Response Quality",
  "type": "llm",
  "score_value_type": "numerical",
  "version_id": "evlv_550e8400",
  "version": 1,
  "is_read_only": false,
  "version_description": "Tightened rubric wording",
  "evaluator_slug": "response_quality",
  "eval_class": "keywordsai_custom_llm",
  "description": "Grades whether the response is accurate and complete.",
  "score_config": {
    "min_score": 1,
    "max_score": 5,
    "choices": [
      {
        "name": "Excellent",
        "value": "excellent"
      }
    ]
  },
  "passing_conditions": {
    "primary_score": {
      "operator": "gte",
      "value": 3
    }
  },
  "llm_config": {
    "model": "gpt-4o-mini",
    "evaluator_definition": "Rate the response quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
    "scoring_rubric": "1=Poor, 5=Excellent",
    "temperature": 0.1,
    "max_tokens": 200,
    "top_p": 0.9,
    "frequency_penalty": 0.5,
    "presence_penalty": 0.3,
    "stop": [
      "###"
    ],
    "response_format": {},
    "tools": [
      {}
    ],
    "verbosity": "detailed"
  },
  "code_config": {
    "eval_code_snippet": "def main(eval_inputs):\n    return 1 if eval_inputs.get('output') else 0"
  },
  "configurations": {},
  "categorical_choices": [
    {
      "name": "Excellent",
      "value": "excellent"
    }
  ],
  "starred": false,
  "created_at": "2024-01-15T09:30:00Z",
  "updated_at": "2024-01-15T09:30:00Z",
  "created_by": {
    "id": 101,
    "first_name": "Alice",
    "last_name": "Johnson",
    "email": "alice.johnson@respan.ai"
  },
  "updated_by": {
    "id": 102,
    "first_name": "Bob",
    "last_name": "Smith",
    "email": "bob.smith@respan.ai"
  },
  "editor": {
    "id": 103,
    "first_name": "Carol",
    "last_name": "Davis",
    "email": "carol.davis@respan.ai"
  },
  "tags": [
    {
      "id": 10,
      "name": "Quality",
      "color": "#4CAF50"
    }
  ]
}

Create a grader/evaluator. Current public evaluator types are llm, human, and code; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (score_config, passing_conditions, llm_config, code_config) instead of relying on legacy configurations.

Create a grader/evaluator. Current public evaluator types are `llm`, `human`, and `code`; legacy human-specific types remain readable for older evaluators. New clients should use the flat config fields (`score_config`, `passing_conditions`, `llm_config`, `code_config`) instead of relying on legacy `configurations`.

Request

This endpoint expects an object.

namestringRequired

typeenumRequired

Allowed values:

score_value_typeenumRequired

evaluator_slugstringOptional

Organization-scoped evaluator slug.

eval_classenumOptional

Optional pre-built evaluator template.

Allowed values:

descriptionstring or nullOptional

score_configobjectOptional

Score configuration. For numerical/percentage scores, use min_score and max_score. For select scores, use choices.

passing_conditionsobject or nullOptional

Passing conditions in the standard Respan filter format.

llm_configobject or nullOptional

LLM grader configuration. The backend validates this against the selected evaluator form.

code_configobject or nullOptional

Code grader configuration.

configurationsmap from strings to anyOptional

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

categorical_choiceslist of objectsOptional

starredbooleanOptional

Response

Created evaluator.

idstring

Stable evaluator ID shared by all versions.

namestring

typeenum

Allowed values:

score_value_typeenum

version_idstring

Unique ID for this evaluator version.

versioninteger

Version number. The first draft is version 0.

is_read_onlyboolean

Whether this version is committed and immutable.

version_descriptionstring

Commit message for this version.

evaluator_slugstring

Organization-scoped evaluator slug.

eval_classenum

Optional pre-built evaluator template.

Allowed values:

descriptionstring or null

score_configobject

Score configuration. For numerical/percentage scores, use min_score and max_score. For select scores, use choices.

passing_conditionsobject or null

Passing conditions in the standard Respan filter format.

llm_configobject or null

LLM grader configuration. The backend validates this against the selected evaluator form.

code_configobject or null

Code grader configuration.

configurationsmap from strings to any

Legacy user-facing configuration object. New clients should prefer llm_config, code_config, score_config, and passing_conditions.

categorical_choiceslist of objects or null

starredboolean

created_atdatetime

updated_atdatetime

created_byobject

updated_byobject

editorobject

tagslist of objects

Errors

400

Bad Request Error

401

Unauthorized Error

Headers

Request

Response

Errors