Skip to main content
Evaluations let you measure how an agent behaves against a fixed set of scenarios before you rely on it in production. You write an evaluation spec that describes the scenarios to run, the checks to score, and how many times to repeat each scenario. The evaluation engine runs the trials, scores each check, and reports the results on separate axes so you can see structural correctness, behavioral quality, and cost independently.
Evaluations are gated per account. If the Evals section does not appear in the agent editor, the feature is not enabled for your account yet.

​
Evaluation spec

An evaluation spec is a document with three parts: The engine validates the spec before running. A spec that exceeds the limits or uses an unsupported check type is rejected.

​
Check types

The engine supports three check types across two categories. Structural checks assert that a specific tool call happened. Behavioral checks score the quality of a response.

​
hasToolCall

Use hasToolCall to assert that the agent called a named tool. The tool field is the name of the tool to look for.

​
toolCallContains

Use toolCallContains to assert that the agent called a named tool with specific arguments. The tool field is the tool name, and the arguments object holds the argument values that must be present on the call.

​
llmJudge

Use llmJudge to score a response against a rubric with a language model. The name field identifies the judge in the results, and the rubric field describes the scoring criteria.

​
llmPreferences

An llmJudge check accepts an optional llmPreferences field to control which provider runs the judge. Provider names are normalized to canonical uppercase values, so anthropic becomes ANTHROPIC and openai becomes OPENAI.

​
Limits

The engine enforces hard limits on every spec. A spec that exceeds any limit is rejected during validation. The total number of trials is the number of samples multiplied by the number of epochs, capped at 300. Each trial has a fixed 600-second timeout.

​
Results and aggregation

The engine reports results on three separate axes and never blends them into a single collapsed score. This keeps structural correctness, behavioral quality, and cost independently visible.

​
Structural

The structural axis reports whether all structural checks passed. The summary.structural.all_passed field is true when every structural check passed, false when at least one failed, and null when the spec has no structural checks.

​
Behavioral

The behavioral axis reports the mean and standard deviation per judge across epochs, along with the count of completed and abstained trials. Reporting per judge across epochs surfaces variance rather than hiding it behind an average.

​
Efficiency and tokens

The efficiency axis tracks gross billed tokens alongside the evaluator’s own overhead. The tokens.eval_overhead field meters the evaluator’s spend on emulator responses and LLM judges separately from the candidate agent’s spend, so you can attribute cost precisely. The engine also reports latency averages.