Skip to main content
Validation rules determine whether a scenario passed. They are declared in the validation object of scenario.json and evaluated after the agent finishes executing.

​
Structure

"validation": {
  "gateway": [...],
  "job": [...],
  "tasks": [...]
}
Rules are organized into three scopes based on what they check:
ScopeWhen usedWhat it checks
gateway--from gateway onlyThe ServiceNow ticket
jobAll entry pointsThe job record (status, plan, task count)
tasksAll entry pointsPer-task data (commands, summaries, plans)
For tasks rules, each rule is checked against every task. A rule passes if it matches on at least one task.

​
Rule Structure

Every rule shares a common set of fields: 
{
  "label": "Agent restarted nginx",
  "validator": "pattern_match",
  "pattern": "systemctl.*(restart|reload|start).*nginx",
  "where": "executed_commands",
  "required": true,
  "negate": false
}
FieldDefaultDescription
label-Required. Shown in the validation report. Make it descriptive.
validator-Required. The type of check to run. See validators below.
requiredtrueWhen false, the rule is informational: it contributes to the compliance score but does not block a pass.
negatefalseInvert the result. The rule passes when the condition is NOT met.

​
Validators

​
pattern_match

Checks whether a regex pattern matches in a specific field of the job or task data. 
{
  "label": "Agent edited the nginx config",
  "validator": "pattern_match",
  "pattern": "/etc/nginx/",
  "where": "executed_commands"
}
FieldDescription
patternRegular expression. Matching is case-insensitive.
whereThe field to search.
where targets:
TargetContent
executed_commandsAll commands run by the agent, one per line
task_summaryThe agent’s summary of what it did
task_descriptionThe task description as created
task_planThe agent’s execution plan
agent_messagesFull agent reasoning history
job_resolutionThe job’s resolution summary
job_planThe job-level plan
gateway_messagesMessages posted by the gateway bot on the ticket
gateway_summaryThe gateway’s summary of ticket resolution
operational_rulesOperational constraints from the agent’s context

​
job_resolution_status

Checks the job’s final resolution status. 
{
  "label": "Job resolved successfully",
  "validator": "job_resolution_status",
  "pattern": "success"
}
Allowed values: success, agentic_failure, hard_failure, partial, no_action.

​
ticket_status

Checks the status of the ServiceNow ticket. Only applicable with --from gateway. 
{
  "label": "Ticket was resolved",
  "validator": "ticket_status",
  "pattern": "resolved"
}

​
task_count

Verifies that the number of tasks created under the job falls within a range. 
{
  "label": "Resolved efficiently",
  "validator": "task_count",
  "min": 1,
  "max": 3
}

​
ansible

Runs an Ansible playbook and treats its exit code as pass/fail. The most reliable way to assert actual machine state. 
{
  "label": "Nginx is running and serving traffic",
  "validator": "ansible",
  "ansiblePath": "validate.yml"
}
A non-zero exit code fails the resolution gate and marks the scenario as failed. See Playbooks for how to write validate.yml.

​
Scoring Model

Compliance score: percentage of all non-Ansible rules that passed (required + optional combined). Informational. Two gates determine the actual pass/fail result: Compliance gate: passes when every required non-Ansible rule passes. Resolution gate: if a validate.yml Ansible rule exists, passes when the playbook exits 0. If no validate.yml rule is defined, it passes when the compliance gate passes and the compliance score is at least 80%. A scenario passes only when both gates pass.