Skip to main content
Validation rules determine whether a scenario passed. They are declared in the validation object of scenario.json and evaluated after the agent finishes executing.

​
Structure

"validation": {
  "gateway": [...],
  "job": [...],
  "tasks": [...]
}
Rules are organized into three scopes based on what they check:
ScopeWhen usedWhat it checks
gateway--from gateway onlyThe ServiceNow ticket
jobAll entry pointsThe job record (status, plan, task count)
tasksAll entry pointsPer-task data (commands, summaries, plans)
For tasks rules, each rule is checked against every task. A rule passes if it matches on at least one task.

​
Rule Structure

Every rule shares a common set of fields: 
{
  "label": "Agent restarted nginx",
  "validator": "pattern_match",
  "pattern": "systemctl.*(restart|reload|start).*nginx",
  "where": "executed_commands",
  "required": true,
  "negate": false
}
FieldDefaultDescription
label-Required. Shown in the validation report. Make it descriptive.
validator-Required. The type of check to run. See validators below.
requiredtrueWhen false, the rule is informational: it contributes to the compliance score but does not block a pass.
negatefalseInvert the result. The rule passes when the condition is NOT met.

​
Validators

​
pattern_match

Checks whether a regex pattern matches in a specific field of the job or task data. 
{
  "label": "Agent edited the nginx config",
  "validator": "pattern_match",
  "pattern": "/etc/nginx/",
  "where": "executed_commands"
}
FieldDescription
patternRegular expression. Matching is case-insensitive.
whereThe field to search.
where targets:
TargetContent
executed_commandsAll commands run by the agent, one per line
task_summaryThe agent’s summary of what it did
task_descriptionThe task description as created
task_planThe agent’s execution plan
agent_messagesFull agent reasoning history
job_resolutionThe job’s resolution summary
job_planThe job-level plan
gateway_messagesMessages posted by the gateway bot on the ticket
gateway_summaryThe gateway’s summary of ticket resolution
operational_rulesOperational constraints from the agent’s context

​
job_resolution_status

Checks the job’s final resolution status. 
{
  "label": "Job resolved successfully",
  "validator": "job_resolution_status",
  "pattern": "success"
}
Allowed values: success, agentic_failure, hard_failure, no_tasks_created.

​
ticket_status

Checks the status of the ServiceNow ticket. Only applicable with --from gateway. 
{
  "label": "Ticket was resolved",
  "validator": "ticket_status",
  "pattern": "resolved"
}

​
task_count

Verifies that the number of tasks created under the job falls within a range. 
{
  "label": "Resolved efficiently",
  "validator": "task_count",
  "min": 1,
  "max": 3
}

​
ansible

Runs an Ansible playbook and treats its exit code as pass/fail. The most reliable way to assert actual machine state. 
{
  "label": "Nginx is running and serving traffic",
  "validator": "ansible",
  "ansiblePath": "validate.yml"
}
A non-zero exit code fails the resolution gate and marks the scenario as failed. See Playbooks for how to write validate.yml.

​
Scoring Model

Compliance score: percentage of all non-Ansible rules that passed (required + optional combined). Informational. Two gates determine the actual pass/fail result: Compliance gate: passes when every required non-Ansible rule passes. Resolution gate: if an Ansible rule exists, passes when the playbook exits 0. If no Ansible rule is defined, mirrors the compliance gate result. A scenario passes only when both gates pass.