Validation rules - 2501.ai

Validation rules grade a scenario run once the agent is done. Edit them in the Validation section of the scenario form, or as the validation array of the scenario through the API. The engine always checks that the job completed first, even when the array is empty.

"validation": [
  {
    "label": "Agent restarted nginx",
    "validator": "pattern_match",
    "pattern": "systemctl.*restart.*nginx",
    "where": "executed_commands"
  },
  {
    "label": "Job resolved acceptably",
    "validator": "job_resolution_status",
    "statuses": ["success", "partial"]
  },
  {
    "label": "Nginx is active",
    "validator": "host_state",
    "run": "systemctl is-active nginx"
  }
]

Validators

Validator	Fields	What it checks
`pattern_match`	`pattern`, `where`, optional `negate`	A case-insensitive regular expression against gateway, job, or task evidence.
`task_count`	optional `min`, optional `max`	The number of tasks created for the job. At least one bound is required.
`job_resolution_status`	`statuses`	The final resolution status. More than one status can be accepted.
`host_state`	`run`, optional `on`, optional `exit_code`	A shell command on a scenario host. It passes when the command returns the expected exit code, which defaults to `0`.

Every rule needs a human-readable label. required defaults to true; an optional rule still contributes to the compliance score but does not fail the run.

What `pattern_match` can inspect

Scope	Targets
Gateway	`gateway_messages`, `gateway_summary`
Job	`job_resolution`, `job_plan`
Task	`task_summary`, `task_description`, `executed_commands`, `write_file`, `update_file`, `use_mcp_tool`, `tool_calls`, `agent_messages`, `operational_rules`, `task_plan`

How a run is scored

Passed / Failed: the run passes when every required rule passes, including the built-in Job completed.
Compliance: the share of all rules that passed, required and optional, from 0 to 100.
Remediation: whether the job resolved as success. It is reported on its own and does not change the verdict.

See How a run is graded for how to read the three together.