> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Why Benchmark

> Measure how your agents handle real incidents on your own machines, before they meet production tickets

**Benchmark** puts your agents in front of incidents you stage yourself - a full disk, a stopped service, a blocked port - on real machines you can afford to break, and grades what they did: did the agent **fix it**, and did it **fix it the right way**.

Everything happens in **Command Center → Benchmarks** or through the [API](https://docs.2501.ai/0.20/benchmark/api.md). You write the scenario, test it on the machine, run it against the models you want, and read a graded result.

![Benchmarks](https://mintcdn.com/2501/Nw-5R3i4HUHiTxUb/images/benchmarks_list_016.png?fit=max\&auto=format\&n=Nw-5R3i4HUHiTxUb\&q=85\&s=00e9c7f18cb7c6c3affc1049f88fef11)

Benchmarks: every scenario's pass rate and trend over time.

## Why benchmark

> [card] Validate new ticket types
>
> Before letting a specialty handle a new kind of ticket, stage that failure on a test machine and watch how the agent handles it across many runs.

> [card] Catch regressions
>
> A scenario that used to pass starts failing after a specialty, rule, or model change. The trend line shows it the same day.

> [card] Compare models
>
> Run the same scenarios with a different main engine. The scores tell you which model behaves better, not just which is faster.

> [card] Prove compliance
>
> Every run keeps a record: the commands the agent ran, the operational rules it was given, and which of your rules passed.

## The two concepts

| Concept | What it is |
| - | - |
| **Scenario** | One reproducible incident: the commands that break the machine (**Target Setup**), the ticket the agent receives, the rules that grade the run (**Validation**), and the commands that put the machine back (**Target Restore**). |
| **Benchmark** | One or more scenarios launched together with the same gateway and models. Each scenario in it produces a **scenario run** with its own result. |

A scenario targets real hosts and agents from your organization. There are no special benchmark machines: any host with an agent can be a target, so pick hosts you can afford to break.

## How a scenario runs

1. **Target Setup** runs your setup commands in order. If one fails, no ticket is sent, the run ends in **Error**, and restore still runs.
2. **Flush** clears the evaluated agents' memory, so an earlier run cannot leak into this one. The tasks of other jobs still running on those agents stay in memory, so a live job is never disturbed.
3. **Dispatch** files the ticket. With the **Runner** gateway, the ticket is created inside 2501. With **ServiceNow**, a real incident is created on your instance and 2501 waits up to 10 minutes for the gateway to pick it up. With **Ecritel**, a real monitoring alert is raised on the scenario's host and 2501 waits for the gateway to turn it into a job.
4. **The agent works** the ticket exactly as it would a real one. The run waits up to 45 minutes for the job to end.
5. **Collect** gathers the commands, messages and job resolution, and **Evaluate** applies the [validation rules](https://docs.2501.ai/0.20/benchmark/validation.md) to them, including commands on the host to check its final state.
6. **Target Restore** runs your restore commands. It runs on every path - success, failure, or stop - so the next run starts from a clean machine. If restore fails, the run says **Restore failed**: the machine may still be broken.

Two things are frozen when you launch: the **scenario version** (editing a scenario never changes a run already queued) and the **models**. A run is always graded against the definition and engines it started with.

## How a run is graded

Grading is deterministic: no model judges the run. A built-in rule, **Job completed**, is checked first, then your rules. A scenario whose required `job_resolution_status` rule accepts `agentic_failure` replaces Job completed with that rule, so it can expect a job that fails. Each scenario run reports three results:

| Result | What it means |
| - | - |
| **Pass / Fail** | The verdict. A run passes when every required rule passes, including Job completed when it applies. |
| **Compliance** | The share of rules that passed - required, optional, and Job completed when it applies - from 0 to 100. It tells you how close a failing run got. |
| **Remediation** | Whether the job resolved as `success`. It is reported on its own and does not change the verdict. |

A run that never reached grading - its setup failed, or the job died for a technical reason - shows **Error**, not Fail. An Error means the scenario or the platform needs fixing; a Fail is a real result about the agent. See [Reading results](https://docs.2501.ai/0.20/benchmark/results.md).

## Prerequisites

* **A Runner gateway.** Create one in **Gateways → New Gateway**, type **Runner**. It needs no configuration. To run through ServiceNow or Ecritel instead, see [Running through ServiceNow](https://docs.2501.ai/0.20/benchmark/command-center.md#running-through-servicenow) and [Running through Ecritel](https://docs.2501.ai/0.20/benchmark/command-center.md#running-through-ecritel).
* **A remote-execution agent on every target host.** Setup, restore, and `host_state` checks reuse that agent's connection: its SSH or WinRM credentials, jump hosts, shell, environment, and user. Linux and Windows hosts are supported; network devices are not.
* **Hosts you can afford to break.** Setup commands change the machine for real, usually as root.
* **A working model configuration.** The models are checked and frozen at launch.

## Who can do what

| Role | Can |
| - | - |
| **Admin** | Create, edit, publish, disable, and delete scenarios, plus everything a User can do. |
| **User** | Run scenarios and benchmarks (including Target Setup and Target Restore on their own), stop them, and read results. |
| **Auditor** | Read scenarios and results. |

## What to read next

> [card] [Quickstart](https://docs.2501.ai/0.20/benchmark/quickstart.md)
>
> Build, test, and run your first scenario in Command Center.

> [card] [Designing scenarios](https://docs.2501.ai/0.20/benchmark/designing.md)
>
> The rules that make a scenario reliable, and how to grade it fairly.

> [card] [Scenario reference](https://docs.2501.ai/0.20/benchmark/scenario.md)
>
> Every field, step type, and shell rule.

> [card] [Keeping scenarios healthy](https://docs.2501.ai/0.20/benchmark/maintaining.md)
>
> Stop drift from quietly corrupting your results.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.