> ## Documentation Index
> Fetch the complete documentation index at: https://docs.guild.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluations

> Measure agent behavior against a versioned specification of samples, checks, and epochs.

Evaluations let you measure how an agent behaves against a fixed set of scenarios before you rely on it in production. You write an evaluation spec that describes the scenarios to run, the checks to score, and how many times to repeat each scenario. The evaluation engine runs the trials, scores each check, and reports the results on separate axes so you can see structural correctness, behavioral quality, and cost independently.

<Note>
  Evaluations are gated per account. If the Evals section does not appear in the agent editor, the feature is not enabled for your account yet.
</Note>

## Evaluation spec

An evaluation spec is a document with three parts:

| Field     | Description                                                                                       |
| --------- | ------------------------------------------------------------------------------------------------- |
| `samples` | The scenarios to run against the candidate agent. Each sample defines one input scenario.         |
| `checks`  | The scoring rules applied to each trial. See [Check types](#check-types).                         |
| `epochs`  | How many times to repeat each sample. Repetition surfaces variance in non-deterministic behavior. |

The engine validates the spec before running. A spec that exceeds the [limits](#limits) or uses an unsupported check type is rejected.

## Check types

The engine supports three check types across two categories. Structural checks assert that a specific tool call happened. Behavioral checks score the quality of a response.

| Type               | Category   | Required fields     |
| ------------------ | ---------- | ------------------- |
| `hasToolCall`      | Structural | `tool`              |
| `toolCallContains` | Structural | `tool`, `arguments` |
| `llmJudge`         | Behavioral | `name`, `rubric`    |

### hasToolCall

Use `hasToolCall` to assert that the agent called a named tool. The `tool` field is the name of the tool to look for.

```json theme={null}
{
  "type": "hasToolCall",
  "tool": "create_issue"
}
```

### toolCallContains

Use `toolCallContains` to assert that the agent called a named tool with specific arguments. The `tool` field is the tool name, and the `arguments` object holds the argument values that must be present on the call.

```json theme={null}
{
  "type": "toolCallContains",
  "tool": "create_issue",
  "arguments": {
    "priority": "high"
  }
}
```

### llmJudge

Use `llmJudge` to score a response against a rubric with a language model. The `name` field identifies the judge in the results, and the `rubric` field describes the scoring criteria.

```json theme={null}
{
  "type": "llmJudge",
  "name": "tone",
  "rubric": "The response is polite and addresses every question the user asked."
}
```

#### llmPreferences

An `llmJudge` check accepts an optional `llmPreferences` field to control which provider runs the judge. Provider names are normalized to canonical uppercase values, so `anthropic` becomes `ANTHROPIC` and `openai` becomes `OPENAI`.

```json theme={null}
{
  "type": "llmJudge",
  "name": "tone",
  "rubric": "The response is polite and addresses every question the user asked.",
  "llmPreferences": ["ANTHROPIC", "OPENAI"]
}
```

## Limits

The engine enforces hard limits on every spec. A spec that exceeds any limit is rejected during validation.

| Limit                | Value       |
| -------------------- | ----------- |
| Maximum samples      | 100         |
| Maximum epochs       | 3           |
| Maximum total trials | 300         |
| Trial timeout        | 600 seconds |

The total number of trials is the number of samples multiplied by the number of epochs, capped at 300. Each trial has a fixed 600-second timeout.

## Results and aggregation

The engine reports results on three separate axes and never blends them into a single collapsed score. This keeps structural correctness, behavioral quality, and cost independently visible.

### Structural

The structural axis reports whether all structural checks passed. The `summary.structural.all_passed` field is `true` when every structural check passed, `false` when at least one failed, and `null` when the spec has no structural checks.

### Behavioral

The behavioral axis reports the mean and standard deviation per judge across epochs, along with the count of completed and abstained trials. Reporting per judge across epochs surfaces variance rather than hiding it behind an average.

### Efficiency and tokens

The efficiency axis tracks gross billed tokens alongside the evaluator's own overhead. The `tokens.eval_overhead` field meters the evaluator's spend on emulator responses and LLM judges separately from the candidate agent's spend, so you can attribute cost precisely. The engine also reports latency averages.