> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Validation

> Validation rules, validators, and scoring model

Validation rules determine whether a scenario passed. They are declared in the `validation` object of `scenario.json` and evaluated after the agent finishes executing.

***

## Structure

```json theme={null}
"validation": {
  "gateway": [...],
  "job": [...],
  "tasks": [...]
}
```

Rules are organized into three scopes based on what they check:

| Scope     | When used             | What it checks                             |
| --------- | --------------------- | ------------------------------------------ |
| `gateway` | `--from gateway` only | The ServiceNow ticket                      |
| `job`     | All entry points      | The job record (status, plan, task count)  |
| `tasks`   | All entry points      | Per-task data (commands, summaries, plans) |

For `tasks` rules, each rule is checked against every task. A rule passes if it matches on at least one task.

***

## Rule Structure

Every rule shares a common set of fields:

```json theme={null}
{
  "label": "Agent restarted nginx",
  "validator": "pattern_match",
  "pattern": "systemctl.*(restart|reload|start).*nginx",
  "where": "executed_commands",
  "required": true,
  "negate": false
}
```

| Field       | Default | Description                                                                                                |
| ----------- | ------- | ---------------------------------------------------------------------------------------------------------- |
| `label`     | -       | Required. Shown in the validation report. Make it descriptive.                                             |
| `validator` | -       | Required. The type of check to run. See validators below.                                                  |
| `required`  | `true`  | When `false`, the rule is informational: it contributes to the compliance score but does not block a pass. |
| `negate`    | `false` | Invert the result. The rule passes when the condition is NOT met.                                          |

***

## Validators

### `pattern_match`

Checks whether a regex pattern matches in a specific field of the job or task data.

```json theme={null}
{
  "label": "Agent edited the nginx config",
  "validator": "pattern_match",
  "pattern": "/etc/nginx/",
  "where": "executed_commands"
}
```

| Field     | Description                                       |
| --------- | ------------------------------------------------- |
| `pattern` | Regular expression. Matching is case-insensitive. |
| `where`   | The field to search.                              |

**`where` targets:**

| Target              | Content                                          |
| ------------------- | ------------------------------------------------ |
| `executed_commands` | All commands run by the agent, one per line      |
| `task_summary`      | The agent's summary of what it did               |
| `task_description`  | The task description as created                  |
| `task_plan`         | The agent's execution plan                       |
| `agent_messages`    | Full agent reasoning history                     |
| `job_resolution`    | The job's resolution summary                     |
| `job_plan`          | The job-level plan                               |
| `gateway_messages`  | Messages posted by the gateway bot on the ticket |
| `gateway_summary`   | The gateway's summary of ticket resolution       |
| `operational_rules` | Operational constraints from the agent's context |

***

### `job_resolution_status`

Checks the job's final resolution status.

```json theme={null}
{
  "label": "Job resolved successfully",
  "validator": "job_resolution_status",
  "pattern": "success"
}
```

Allowed values: `success`, `agentic_failure`, `hard_failure`, `no_tasks_created`.

***

### `ticket_status`

Checks the status of the ServiceNow ticket. Only applicable with `--from gateway`.

```json theme={null}
{
  "label": "Ticket was resolved",
  "validator": "ticket_status",
  "pattern": "resolved"
}
```

***

### `task_count`

Verifies that the number of tasks created under the job falls within a range.

```json theme={null}
{
  "label": "Resolved efficiently",
  "validator": "task_count",
  "min": 1,
  "max": 3
}
```

***

### `ansible`

Runs an Ansible playbook and treats its exit code as pass/fail. The most reliable way to assert actual machine state.

```json theme={null}
{
  "label": "Nginx is running and serving traffic",
  "validator": "ansible",
  "ansiblePath": "validate.yml"
}
```

A non-zero exit code fails the resolution gate and marks the scenario as failed. See [Playbooks](/0.4/scenario-runner/playbooks) for how to write `validate.yml`.

***

## Scoring Model

**Compliance score**: percentage of all non-Ansible rules that passed (required + optional combined). Informational.

Two gates determine the actual pass/fail result:

**Compliance gate**: passes when every `required` non-Ansible rule passes.

**Resolution gate**: if an Ansible rule exists, passes when the playbook exits 0. If no Ansible rule is defined, mirrors the compliance gate result.

**A scenario passes only when both gates pass.**