Api Evaluators List

## Creating an Evaluator ### LLM Evaluators For LLM evaluators, the frontend should first fetch available evaluation forms from the `/eval-forms/` endpoint to get the template configuration, then fill in the form and submit. **Required fields:** - `name` (str): Display name for the evaluator - `evaluator_slug` (str): Unique identifier for the evaluator within the organization - `type` (str): llm, human_boolean, human_categorical, human_numerical, human_text - `configurations` (dict): Complete evaluation form configuration - `description` (str, optional): Description of what this evaluator does - `enabled` (bool, optional): Whether the evaluator is active (default: False) **Example request body for LLM evaluator:** ```json { "name": "Output Length Checker", "type": "llm", "description": "Checks if the output meets character count requirements", "enabled": true, "configurations": { "eval_class": "output_char_count", "type": "function", "note": "", "display_name": "Output Character Count", "description": "Evaluates the length of the output text", "special_fields": [], "required_fields": [ { "name": "llm_output", "display_name": "LLM Output", "type": "textarea", "description": "The output text to evaluate", "required": true, "default_value": null, "placeholder": "", "choices": [], "value": null } ], "inference_filters": [], "allow_conditions": true, "score_mapping": { "primary_score": "output_char_count", "secondary_score": null, "tertiary_score": null, "quaternary_score": null }, "category": "custom" } } ``` ### Human Annotation Evaluators For human annotation evaluators, specify the type and provide choices for categorical evaluators. **Human Boolean Evaluator:** ```json { "name": "Quality Check", "evaluator_slug": "quality_check", "type": "human_boolean", "description": "Manual quality assessment" } ``` **Human Categorical Evaluator:** ```json { "name": "Sentiment Rating", "evaluator_slug": "sentiment_rating", "type": "human_categorical", "description": "Manual sentiment classification", "categorical_choices": [ {"name": "Positive", "value": 1}, {"name": "Neutral", "value": 0}, {"name": "Negative", "value": -1} ] } ``` **Human Numerical Evaluator:** ```json { "name": "Quality Score", "evaluator_slug": "quality_score", "type": "human_numerical", "description": "Rate quality from 1-10" } ``` **Human Text Evaluator:** ```json { "name": "Feedback Comments", "evaluator_slug": "feedback_comments", "type": "human_text", "description": "Detailed feedback comments" } ``` ## Response Returns the created evaluator with all fields populated, including auto-generated fields like `id`, `created_at`, `updated_at`, and `evaluator_slug`. ## Validation - For LLM evaluators: The `configurations` field is validated against the corresponding evaluation form schema from `EVAL_FORMS_MAP` - For human categorical evaluators: `categorical_choices` must be a list of objects with `name` and `value` fields - The `eval_class` in configurations must exist in the available evaluation forms ## Notes - The `organization`, `created_by`, and `updated_by` fields are automatically set from the authenticated user - Each evaluator gets a unique `evaluator_slug` within the organization - LLM evaluators require a valid `eval_class` that maps to an available evaluation function