Skip to main content
Evaluation checks let you assert conditions about the output of an agent run so you can measure quality automatically. Each check inspects a sample from a run and returns a pass, fail, or abstention. Guild supports four check types. Three are structural — hasToolCall, toolCallContains, and outputMatches — and compare values directly. The fourth, llmJudge, asks a model to judge the output. Use a structural check when the correct answer is knowable without reasoning, and reserve llmJudge for open-ended judgments. This page documents outputMatches.

​
outputMatches

Use outputMatches to assert whether a specific substring is present or absent in the final output of an agent run. It is a deterministic lexical check: it compares strings directly rather than asking a model to judge the output. Unlike llmJudge, outputMatches does not require an LLM judge, which reduces latency and cost.

​
Parameters

Exactly one of value or referencePath must be provided. value must be a non-empty string. referencePath must resolve to a non-empty string.

​
Match against a literal value

Assert that the output contains a literal substring:
Assert that the output does not contain a substring:

​
Match against reference data

Use referencePath to compare the output against a value stored in the sample’s reference data. This lets you assert against per-sample expected values instead of hardcoding them:
If referencePath does not resolve to a non-empty string, the check does not fail. It abstains with the code SETUP_INVALID. See Aggregation.

​
Case-insensitive matching

Set caseSensitive to false to match regardless of letter case. Matching is Unicode-aware and compares the casefold() representations of both values:

​
Aggregation

Results of outputMatches checks are aggregated by spec index and check type. If a check cannot resolve its referencePath, it does not count as a pass or a fail. It is reported as an abstention under abstained and abstain_reasons with the code SETUP_INVALID.