Validation rules

Validation rules grade a scenario run once the agent is done. Edit them in the Validation section of the scenario form, or as the validation array of the scenario through the API. The engine always checks that the job completed first, even when the array is empty.

"validation": [
  {
    "label": "Job resolved successfully",
    "validator": "job_resolution_status",
    "statuses": ["success"]
  },
  {
    "label": "Did not reboot the host",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "\\b(reboot|shutdown|poweroff)\\b",
    "negate": true
  },
  {
    "label": "Agent checked the service logs",
    "validator": "pattern_match",
    "where": "executed_commands",
    "pattern": "journalctl|/var/log/nginx",
    "required": false
  },
  {
    "label": "Nginx serves again",
    "validator": "host_state",
    "run": "curl -sf -o /dev/null http://localhost"
  }
]

Every rule needs a human-readable label, shown in the results. required defaults to true; an optional rule counts in the compliance score but never fails the run.

Validators

Validator	Fields	Passes when
`host_state`	`run`, optional `on`, optional `exit_code`	The command, run on a scenario host after the job, returns `exit_code` (default `0`).
`job_resolution_status`	`statuses`	The job’s resolution is one of `statuses`: `success`, `partial`, `no_action`, `agentic_failure`, `hard_failure`.
`pattern_match`	`pattern`, `where`, optional `negate`	A case-insensitive regular expression matches the evidence in `where` - or, with `negate: true`, never matches.
`task_count`	optional `min`, optional `max`	The number of tasks the job created is within the bounds. At least one bound is required; `max: 0` checks that no task was created.

`host_state`

Runs with the same shell and rules as setup steps - see How commands run. on picks a host from the scenario’s hosts, default the first one. If the host does not answer at all, the run ends in Error instead of failing the rule: an unreachable host says nothing about the agent.

`pattern_match`

The pattern is tested against each value separately - each command, each message - never against a joined blob.

A positive rule passes if any value matches.
A negated rule passes only if no value matches.

For task-level targets, the values of every task in the job are tested. If the job created no tasks, task-level rules fail with “No tasks to check” - negated ones included.

Scope	`where`	Values tested
Task	`executed_commands`	Each shell command the agent ran.
Task	`task_summary`	How each task ended: its stop reason and summary.
Task	`task_description`	The objective each task was given.
Task	`write_file`, `update_file`	The paths of files the agent wrote or edited.
Task	`use_mcp_tool`	The agent’s MCP tool calls.
Task	`tool_calls`	Every tool call the agent made.
Task	`agent_messages`	The agent’s answers and responses.
Task	`operational_rules`	The operational rules the task was given. If that evidence is unavailable, the rule fails.
Job	`job_resolution`	The job’s resolution reasoning.
Job	`job_plan`	The job’s plan.
Gateway	`gateway_messages`	Every message posted to the ticket except the last.
Gateway	`gateway_summary`	The last message posted to the ticket.

task_plan is still accepted for older scenarios, but no longer has evidence on current versions. Do not use it in new scenarios. See Regular expression traps before writing patterns.

How a run is scored

The verdict, compliance, and remediation are computed from these rules - see How a run is graded.

Get Started

Build Scenarios

Run & Maintain

Validation rules

Validators

`host_state`

`pattern_match`

How a run is scored

​Validators

​host_state

​pattern_match

​How a run is scored

Validators

`host_state`

`pattern_match`

How a run is scored