EvaluationsEvaluators

Route failures to human review

Use a condition to send only the outputs that fail an automated check to a human reviewer.

LLM graders scale, but some outputs deserve a human’s judgment. The most effective pattern is a hybrid: grade everything automatically, and route only the outputs that look weak to a person. This keeps human effort focused on the cases that need it.

This walkthrough builds a conditional evaluator: an automated Helpfulness check runs on every support reply, and any reply scoring below 3 is escalated to a human reviewer. It continues the support example from Weighted-average scoring.

The flow

A support reply is scored by the Helpfulness grader; if the score is below 3 it routes to human review, otherwise it auto-passes, and both paths lead to the final score.

A condition block gates the flow: it runs the automated grader first, checks the score, and sends only the low ones to the human grader. Everything else passes automatically.

Walkthrough

1

Create the human grader

In the Graders section, add a grader named Human review and set its type to Human. Choose a numeric range of 1 to 5. Write a clear description; it is shown to the reviewer while they score. No LLM or code config is needed.

Human review grader
The Human review grader: a human-type grader scoring the reply 1 to 5, with review instructions.
2

Add the automated check

Drag the Helpfulness grader (from Weighted-average scoring) onto the canvas so it scores the Original input first. This is the signal the condition will branch on.

3

Add the condition

Add an If … Then … Else condition. Set it to check the Helpfulness score with < 3.

  • Then (score is low): route to the Human review grader.
  • Else (score is fine): send a passing constant to Final result.
Condition routing low scores to human review
The condition block: Helpfulness below 3 routes to Human review, otherwise the reply auto-passes.
4

Assign the reviewer

On the Human review grader, assign who reviews escalated outputs. Add Amy Kodama or Frank Chen (any workspace member can be a reviewer). Escalated outputs land in their review queue in the platform.

Assigning a human reviewer
Assigning a workspace member as the reviewer for the Human review grader.
5

Test and deploy

Test run the flow. A strong reply scores high on Helpfulness and auto-passes; a weak reply falls below 3 and is routed to the reviewer. Deploy when the routing behaves the way you want.

You can gate on a metric instead of a grader score. For example, add a condition on Cost or Latency so an expensive LLM grader only runs on the spans that matter. See the block reference.

What’s next