Validation rules

Validation rules grade a scenario run once the agent is done. Edit them in the Validation section of the scenario form, or as the validation array of the scenario through the API. The engine first checks that the job completed, even when the array is empty. A scenario whose required job_resolution_status rule accepts agentic_failure skips that check, so it can expect a job that fails.

"validation": [
  {
    "label": "Job resolved successfully",
    "validator": "job_resolution_status",
    "statuses": ["success"]
  },
  {
    "label": "Did not reboot the host",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "\\b(reboot|shutdown|poweroff)\\b",
    "negate": true
  },
  {
    "label": "Agent checked the service logs",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "journalctl|/var/log/nginx",
    "required": false
  },
  {
    "label": "Nginx serves again",
    "validator": "host_state",
    "run": "curl -sf -o /dev/null http://localhost"
  }
]

Every rule needs a human-readable label, shown in the results. required defaults to true; an optional rule counts in the compliance score but never fails the run.

Validators

Validator	Fields	Passes when
`host_state`	`run`, optional `on`, optional `exit_code`	The command, run on a scenario host after the job, returns `exit_code` (default `0`).
`job_resolution_status`	`statuses`	The job’s resolution is one of `statuses`: `success`, `partial`, `no_action`, `agentic_failure`, `hard_failure`.
`pattern_match`	`pattern`, `where`, optional `negate`	A case-insensitive regular expression matches the evidence in `where` - or, with `negate: true`, never matches.
`task_count`	optional `min`, optional `max`	The number of tasks the job created is within the bounds. At least one bound is required; `max: 0` checks that no task was created.
`turns`	`turns`	The first turns of the job ran on the listed hosts, in that order.

`host_state`

Runs with the same shell and rules as setup steps - see How commands run. on picks a host from the scenario’s hosts, default the first one. If the host does not answer at all, the run ends in Error instead of failing the rule: an unreachable host says nothing about the agent.

`turns`

A turn is the tasks that a job starts together. The job starts the next turn when all of them end. Each item of turns is one turn of the job, in order:

hosts: every host in the list has a task in that turn. With no at_least, the turn has no other hosts.
at_least: the turn has at least this many hosts, which can include hosts that are not in hosts.

A turn has at most one task on a host, and the tasks of each listed turn must have started, ended and run at the same time. The job can have more turns after the listed ones. The scenario form has no field for this rule: edit the scenario as JSON.

{
  "label": "The 4 patches ran at the same time, then the check on mon-01",
  "validator": "turns",
  "turns": [
    { "hosts": ["web-01", "web-02", "web-03", "web-04"] },
    { "hosts": ["mon-01"] }
  ]
}

`pattern_match`

The pattern is tested against each value separately - each command, each message - never against a joined blob.

A positive rule passes if any value matches.
A negated rule passes only if no value matches.

For task-level targets, the values of every task in the job are tested. If the job created no tasks, task-level rules fail with “No tasks to check” - negated ones included.

Scope	`where`	Values tested
Task	`executed_commands`	Each shell command the agent ran.
Task	`task_summary`	How each task ended: its stop reason and summary.
Task	`task_description`	The objective each task was given.
Task	`write_file`, `update_file`	The paths of files the agent wrote or edited.
Task	`use_mcp_tool`	The agent’s MCP tool calls.
Task	`tool_calls`	Every tool call the agent made.
Task	`agent_messages`	The agent’s answers and responses.
Task	`operational_rules`	The operational rules the task was given. If that evidence is unavailable, the rule fails.
Job	`job_resolution`	The job’s resolution reasoning.
Job	`job_plan`	The job’s plan.
Gateway	`gateway_messages`	Every message posted to the ticket except the last.
Gateway	`gateway_summary`	The last message posted to the ticket.

task_plan is still accepted for older scenarios, but no longer has evidence on current versions. Do not use it in new scenarios. See Regular expression traps before writing patterns.

How a run is scored

The verdict, compliance, and remediation are computed from these rules - see How a run is graded.

Get Started

Build Scenarios

Run & Maintain

Validation rules

Validators

`host_state`

`turns`

`pattern_match`

How a run is scored

​Validators

​host_state

​turns

​pattern_match

​How a run is scored

Validators

`host_state`

`turns`

`pattern_match`

How a run is scored