Documentation Index
Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
Use this file to discover all available pages before exploring further.
Validation rules determine whether a scenario passed. They are declared in the validation object of scenario.json and evaluated after the agent finishes executing.
Structure
"validation": {
"gateway": [...],
"job": [...],
"tasks": [...]
}
| Scope | When used | What it checks |
|---|
gateway | --from gateway only | The ServiceNow ticket |
job | All entry points | The job record (status, plan, task count) |
tasks | All entry points | Per-task data (commands, summaries, plans) |
tasks rules, each rule is checked against every task. A rule passes if it matches on at least one task.
Rule Structure
Every rule shares a common set of fields:
{
"label": "Agent restarted nginx",
"validator": "pattern_match",
"pattern": "systemctl.*(restart|reload|start).*nginx",
"where": "executed_commands",
"required": true,
"negate": false
}
| Field | Default | Description |
|---|
label | - | Required. Shown in the validation report. Make it descriptive. |
validator | - | Required. The type of check to run. See validators below. |
required | true | When false, the rule is informational: it contributes to the compliance score but does not block a pass. |
negate | false | Invert the result. The rule passes when the condition is NOT met. |
Validators
pattern_match
Checks whether a regex pattern matches in a specific field of the job or task data.
{
"label": "Agent edited the nginx config",
"validator": "pattern_match",
"pattern": "/etc/nginx/",
"where": "executed_commands"
}
| Field | Description |
|---|
pattern | Regular expression. Matching is case-insensitive. |
where | The field to search. |
where targets:
| Target | Content |
|---|
executed_commands | All commands run by the agent, one per line |
task_summary | The agent’s summary of what it did |
task_description | The task description as created |
task_plan | The agent’s execution plan |
agent_messages | Full agent reasoning history |
job_resolution | The job’s resolution summary |
job_plan | The job-level plan |
gateway_messages | Messages posted by the gateway bot on the ticket |
gateway_summary | The gateway’s summary of ticket resolution |
operational_rules | Operational constraints from the agent’s context |
job_resolution_status
Checks the job’s final resolution status.
{
"label": "Job resolved successfully",
"validator": "job_resolution_status",
"pattern": "success"
}
success, agentic_failure, hard_failure, no_tasks_created.
ticket_status
Checks the status of the ServiceNow ticket. Only applicable with --from gateway.
{
"label": "Ticket was resolved",
"validator": "ticket_status",
"pattern": "resolved"
}
task_count
Verifies that the number of tasks created under the job falls within a range.
{
"label": "Resolved efficiently",
"validator": "task_count",
"min": 1,
"max": 3
}
ansible
Runs an Ansible playbook and treats its exit code as pass/fail. The most reliable way to assert actual machine state.
{
"label": "Nginx is running and serving traffic",
"validator": "ansible",
"ansiblePath": "validate.yml"
}
validate.yml.
Scoring Model
Compliance score: percentage of all non-Ansible rules that passed (required + optional combined). Informational.
Two gates determine the actual pass/fail result:
Compliance gate: passes when every required non-Ansible rule passes.
Resolution gate: if an Ansible rule exists, passes when the playbook exits 0. If no Ansible rule is defined, mirrors the compliance gate result.
A scenario passes only when both gates pass.
