Run an evaluator | Respan Docs

curl -X POST https://api.respan.ai/api/evaluators/evaluator_id/run/ \
     -H "Authorization: Bearer sk_live_xxxxx" \
     -H "Content-Type: application/json" \
     -d '{
  "inputs": {
    "input": "What is the capital of France?",
    "output": "The capital of France is Paris.",
    "expected_output": "Paris",
    "metrics": {
      "latency": 0.45,
      "cost": 0.0023
    },
    "metadata": {
      "model": "gpt-4o-mini"
    }
  }
}'

{
  "id": "res_123e4567-e89b-12d3-a456-426614174000",
  "created_at": "2024-01-15T09:30:00Z",
  "type": "numerical",
  "environment": "production",
  "numerical_value": 4.5,
  "string_value": "Good response",
  "boolean_value": true,
  "categorical_value": [
    "accurate"
  ],
  "json_value": "{\"score\":4.5,\"comments\":\"Well done\"}",
  "is_passed": true,
  "cost": 0.0023,
  "evaluator_id": "evl_550e8400",
  "evaluator_slug": "response_quality",
  "scorer": "gpt-4o-mini",
  "log_id": "log_987654321",
  "prompt_id": "prm_123456789",
  "prompt_version_number": 2,
  "dataset_id": "ds_abc123",
  "automation_id": "auto_456def",
  "status": "completed",
  "error_message": "",
  "inputs": {
    "input": "What is the capital of France?",
    "output": "The capital of France is Paris.",
    "expected_output": "Paris",
    "metrics": {},
    "metadata": {},
    "llm_input": "What is the capital of France?",
    "llm_output": "The capital of France is Paris."
  },
  "evaluator": {
    "id": "evl_550e8400",
    "name": "Response Quality",
    "type": "llm",
    "score_value_type": "numerical",
    "version_id": "evlv_550e8400",
    "version": 1,
    "is_read_only": false,
    "version_description": "Tightened rubric wording",
    "evaluator_slug": "response_quality",
    "eval_class": "keywordsai_custom_llm",
    "description": "Grades whether the response is accurate and complete.",
    "score_config": {
      "min_score": 1,
      "max_score": 5,
      "choices": [
        {
          "name": "Excellent",
          "value": "excellent"
        }
      ]
    },
    "passing_conditions": {
      "primary_score": {
        "operator": "gte",
        "value": 3
      }
    },
    "llm_config": {
      "model": "gpt-4o-mini",
      "evaluator_definition": "Rate the response quality.\n<input>{{input}}</input>\n<output>{{output}}</output>",
      "scoring_rubric": "1=Poor, 5=Excellent",
      "temperature": 0.1,
      "max_tokens": 200,
      "top_p": 1.1,
      "frequency_penalty": 1.1,
      "presence_penalty": 1.1,
      "stop": [
        "\n"
      ],
      "response_format": {},
      "tools": [
        {}
      ],
      "verbosity": "detailed"
    },
    "code_config": {
      "eval_code_snippet": "def main(eval_inputs):\n    return 1 if eval_inputs.get('output') else 0"
    },
    "configurations": {},
    "categorical_choices": [
      {
        "name": "Excellent",
        "value": "excellent"
      }
    ],
    "starred": false,
    "created_at": "2024-01-15T09:30:00Z",
    "updated_at": "2024-01-15T09:30:00Z",
    "created_by": {
      "id": 101,
      "first_name": "Alice",
      "last_name": "Johnson",
      "email": "alice.johnson@respan.ai"
    },
    "updated_by": {
      "id": 102,
      "first_name": "Bob",
      "last_name": "Smith",
      "email": "bob.smith@respan.ai"
    },
    "editor": {
      "id": 103,
      "first_name": "Carol",
      "last_name": "Davis",
      "email": "carol.davis@respan.ai"
    },
    "tags": [
      {
        "id": 1,
        "name": "quality",
        "color": "#4CAF50"
      }
    ]
  }
}

Run an evaluator against raw unified inputs. The evaluator ID may include a version suffix such as evl_abc123:2 to run a specific version.

Path parameters

evaluator_idstringRequired

Evaluator ID. To run a specific version, pass an ID with a version suffix where supported, for example evl_abc123:2.

Request

This endpoint expects an object.

inputsobjectRequired

Unified evaluator inputs.

generation_methodenumOptionalDefaults to auto

Optional method override for evaluators that support multiple execution modes.

Allowed values:

evaluation_idstringOptional

Legacy evaluator ID field. Prefer the path parameter or evaluator_id.

evaluator_idstringOptional

Optional evaluator ID override. Supports version suffixes such as evl_abc123:2.

Response

Evaluation result.

idstring

Score/result ID.

created_atdatetime

typestring

environmentstring

numerical_valuedouble or null

string_valuestring or null

boolean_valueboolean or null

categorical_valuelist of strings or null

json_valuestring or null

is_passedboolean or null

costdouble or null

evaluator_idstring

evaluator_slugstring

scorerstring or null

log_idstring or null

prompt_idstring or null

prompt_version_numberinteger or null

dataset_idstring or null

automation_idstring or null

statusstring or null

error_messagestring or null

inputsobject

Unified evaluator inputs.

evaluatorobject

Errors

400

Bad Request Error

401

Unauthorized Error

404

Not Found Error

Path parameters

Headers

Request

Response

Errors