ProductsEvaluationsEvaluators

Route failures to human review

Use a condition to send only the outputs that fail an automated check to a human reviewer.

LLM graders scale, but some outputs deserve a human’s judgment. The most effective pattern is a hybrid: grade everything automatically, and route only the outputs that look weak to a person. This keeps human effort focused on the cases that need it.

This walkthrough builds a conditional evaluator: an automated Helpfulness check runs on every support reply, and any reply scoring below 3 is escalated to a human reviewer. It continues the support example from Weighted-average scoring.

The flow

A support reply is scored by the Helpfulness grader; if the score is below 3 it routes to human review, otherwise it auto-passes, and both paths lead to the final score.

A condition block gates the flow: it runs the automated grader first, checks the score, and sends only the low ones to the human grader. Everything else passes automatically.

Walkthrough

1

Create the human grader

In the Graders section, add a grader named Human review and set its type to Human. Choose a numeric range of 1 to 5. Write a clear description; it is shown to the reviewer while they score. No LLM or code config is needed.

Human review grader
The Human review grader: a human-type grader scoring the reply 1 to 5, with review instructions.
2

Add the automated check

Drag the Helpfulness grader (from Weighted-average scoring) onto the canvas so it scores the Original input first. This is the signal the condition will branch on.

3

Add the condition

Add an If … Then … Else condition. Set it to check the Helpfulness score with < 3.

  • Then (score is low): route to the Human review grader.
  • Else (score is fine): send a passing constant to Final result.
Condition routing low scores to human review
The condition block: Helpfulness below 3 routes to Human review, otherwise the reply auto-passes.
4

Assign the reviewer

On the Human review grader, assign who reviews escalated outputs. Add Amy Kodama or Frank Chen (any workspace member can be a reviewer). Escalated outputs land in their review queue in the platform.

Assigning a human reviewer
Assigning a workspace member as the reviewer for the Human review grader.
5

Test and deploy

Test run the flow. A strong reply scores high on Helpfulness and auto-passes; a weak reply falls below 3 and is routed to the reviewer. Deploy when the routing behaves the way you want.

Complete assigned reviews

After a deployed evaluator routes an output to a human grader, the assigned reviewer works from Evals → Reviews. The queue is reviewer-specific: it shows items assigned to the signed-in workspace member, not every human-review item in the project.

Open the Reviews queue

Go to Evals → Reviews in the Respan platform.

1

Check the queue status

Read the completed-item counter in the header, then refresh the queue when a new assignment is expected. Use the queue filter when you need to narrow the items currently assigned to you.

2

Review the routed output

Open an assigned item and read the human grader’s instructions before scoring. Inspect the available input, output, and evaluator context, then choose the score or label defined by that grader. The control can differ between numeric, categorical, and pass/fail human graders.

3

Submit and continue

Complete the review to attach the human grader result to that evaluator run. When the human grader feeds another block or the final result, the evaluator continues with the submitted value. Confirm that the completed counter changes before moving to the next assignment.

Nothing appears in Reviews

The empty state says Nothing to review when no items are assigned to the signed-in reviewer. Check these conditions in order:

  1. The human grader assigns the intended workspace member.
  2. The evaluator version containing that assignment is deployed.
  3. A production or test input reached the branch that routes to the human grader.
  4. The reviewer opened the same workspace and project where the evaluator ran.

Changing a reviewer does not create an assignment by itself; an evaluator run must still reach the human-review branch.

You can gate on a metric instead of a grader score. For example, add a condition on Cost or Latency so an expensive LLM grader only runs on the spans that matter. See the block reference.

What’s next