> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation rules

> The rules that grade a scenario run, what each one inspects, and how the run is scored

Validation rules grade a scenario run once the agent is done. Edit them in the **Validation** section of the scenario form, or as the `validation` array of the scenario through the API. The engine always checks that the job completed first, even when the array is empty.

```json theme={null}
"validation": [
  {
    "label": "Job resolved successfully",
    "validator": "job_resolution_status",
    "statuses": ["success"]
  },
  {
    "label": "Did not reboot the host",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "\\b(reboot|shutdown|poweroff)\\b",
    "negate": true
  },
  {
    "label": "Agent checked the service logs",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "journalctl|/var/log/nginx",
    "required": false
  },
  {
    "label": "Nginx serves again",
    "validator": "host_state",
    "run": "curl -sf -o /dev/null http://localhost"
  }
]
```

Every rule needs a human-readable `label`, shown in the results. `required` defaults to `true`; an optional rule counts in the compliance score but never fails the run.

## Validators

| Validator | Fields | Passes when |
| - | - | - |
| `host_state` | `run`, optional `on`, optional `exit_code` | The command, run on a scenario host after the job, returns `exit_code` (default `0`). |
| `job_resolution_status` | `statuses` | The job's resolution is one of `statuses`: `success`, `partial`, `no_action`, `agentic_failure`, `hard_failure`. |
| `pattern_match` | `pattern`, `where`, optional `negate` | A case-insensitive regular expression matches the evidence in `where` - or, with `negate: true`, never matches. |
| `task_count` | optional `min`, optional `max` | The number of tasks the job created is within the bounds. At least one bound is required; `max: 0` checks that no task was created. |

### `host_state`

Runs with the same shell and rules as setup steps - see [How commands run](/0.17/benchmark/scenario#how-commands-run). `on` picks a host from the scenario's `hosts`, default the first one.

If the host does not answer at all, the run ends in **Error** instead of failing the rule: an unreachable host says nothing about the agent.

### `pattern_match`

The pattern is tested against **each value separately** - each command, each message - never against a joined blob.

* A positive rule passes if **any** value matches.
* A negated rule passes only if **no** value matches.

For task-level targets, the values of every task in the job are tested. If the job created **no tasks**, task-level rules fail with *"No tasks to check"* - negated ones included.

| Scope | `where` | Values tested |
| - | - | - |
| Task | `executed_commands` | Each shell command the agent ran. |
| Task | `task_summary` | How each task ended: its stop reason and summary. |
| Task | `task_description` | The objective each task was given. |
| Task | `write_file`, `update_file` | The paths of files the agent wrote or edited. |
| Task | `use_mcp_tool` | The agent's MCP tool calls. |
| Task | `tool_calls` | Every tool call the agent made. |
| Task | `agent_messages` | The agent's answers and responses. |
| Task | `operational_rules` | The operational rules the task was given. If that evidence is unavailable, the rule fails. |
| Job | `job_resolution` | The job's resolution reasoning. |
| Job | `job_plan` | The job's plan. |
| Gateway | `gateway_messages` | Every message posted to the ticket except the last. |
| Gateway | `gateway_summary` | The last message posted to the ticket. |

`task_plan` is still accepted for older scenarios, but no longer has evidence on current versions. Do not use it in new scenarios.

See [Regular expression traps](/0.17/benchmark/designing#regular-expression-traps) before writing patterns.

## How a run is scored

The verdict, compliance, and remediation are computed from these rules - see [How a run is graded](/0.17/benchmark/overview#how-a-run-is-graded).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.