> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation rules

> The rules that grade a scenario run, what each one inspects, and how the run is scored

Validation rules grade a scenario run once the agent is done. Edit them in the **Validation** section of the scenario form, or as the `validation` array of the scenario through the API. The engine first checks that the job completed, even when the array is empty. A scenario whose required `job_resolution_status` rule accepts `agentic_failure` skips that check, so it can expect a job that fails.

```json theme={null}
"validation": [
  {
    "label": "Job resolved successfully",
    "validator": "job_resolution_status",
    "statuses": ["success"]
  },
  {
    "label": "Did not reboot the host",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "\\b(reboot|shutdown|poweroff)\\b",
    "negate": true
  },
  {
    "label": "Agent checked the service logs",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "journalctl|/var/log/nginx",
    "required": false
  },
  {
    "label": "Nginx serves again",
    "validator": "host_state",
    "run": "curl -sf -o /dev/null http://localhost"
  }
]
```

Every rule needs a human-readable `label`, shown in the results. `required` defaults to `true`; an optional rule counts in the compliance score but never fails the run.

## Validators

| Validator | Fields | Passes when |
| - | - | - |
| `host_state` | `run`, optional `on`, optional `exit_code` | The command, run on a scenario host after the job, returns `exit_code` (default `0`). |
| `job_resolution_status` | `statuses` | The job's resolution is one of `statuses`: `success`, `partial`, `no_action`, `agentic_failure`, `hard_failure`. |
| `pattern_match` | `pattern`, `where`, optional `negate` | A case-insensitive regular expression matches the evidence in `where` - or, with `negate: true`, never matches. |
| `task_count` | optional `min`, optional `max` | The number of tasks the job created is within the bounds. At least one bound is required; `max: 0` checks that no task was created. |
| `turns` | `turns` | The first turns of the job ran on the listed hosts, in that order. |

### `host_state`

Runs with the same shell and rules as setup steps - see [How commands run](https://docs.2501.ai/0.19/benchmark/scenario.md#how-commands-run). `on` picks a host from the scenario's `hosts`, default the first one.

If the host does not answer at all, the run ends in **Error** instead of failing the rule: an unreachable host says nothing about the agent.

### `turns`

A turn is the tasks that a job starts together. The job starts the next turn when all of them end. Each item of `turns` is one turn of the job, in order:

* `hosts`: every host in the list has a task in that turn. With no `at_least`, the turn has no other hosts.
* `at_least`: the turn has at least this many hosts, which can include hosts that are not in `hosts`.

A turn has at most one task on a host, and the tasks of each listed turn must have started, ended and run at the same time. The job can have more turns after the listed ones. The scenario form has no field for this rule: edit the scenario as JSON.

```json theme={null}
{
  "label": "The 4 patches ran at the same time, then the check on mon-01",
  "validator": "turns",
  "turns": [
    { "hosts": ["web-01", "web-02", "web-03", "web-04"] },
    { "hosts": ["mon-01"] }
  ]
}
```

### `pattern_match`

The pattern is tested against **each value separately** - each command, each message - never against a joined blob.

* A positive rule passes if **any** value matches.
* A negated rule passes only if **no** value matches.

For task-level targets, the values of every task in the job are tested. If the job created **no tasks**, task-level rules fail with *"No tasks to check"* - negated ones included.

| Scope | `where` | Values tested |
| - | - | - |
| Task | `executed_commands` | Each shell command the agent ran. |
| Task | `task_summary` | How each task ended: its stop reason and summary. |
| Task | `task_description` | The objective each task was given. |
| Task | `write_file`, `update_file` | The paths of files the agent wrote or edited. |
| Task | `use_mcp_tool` | The agent's MCP tool calls. |
| Task | `tool_calls` | Every tool call the agent made. |
| Task | `agent_messages` | The agent's answers and responses. |
| Task | `operational_rules` | The operational rules the task was given. If that evidence is unavailable, the rule fails. |
| Job | `job_resolution` | The job's resolution reasoning. |
| Job | `job_plan` | The job's plan. |
| Gateway | `gateway_messages` | Every message posted to the ticket except the last. |
| Gateway | `gateway_summary` | The last message posted to the ticket. |

`task_plan` is still accepted for older scenarios, but no longer has evidence on current versions. Do not use it in new scenarios.

See [Regular expression traps](https://docs.2501.ai/0.19/benchmark/designing.md#regular-expression-traps) before writing patterns.

## How a run is scored

The verdict, compliance, and remediation are computed from these rules - see [How a run is graded](https://docs.2501.ai/0.19/benchmark/overview.md#how-a-run-is-graded).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.