> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Why Benchmark

> Validate agent behavior on realistic scenarios before exposing it to production

Production is the wrong place to test a new agent. **Benchmark** runs your agents through realistic, reproducible scenarios in a sandbox - a broken nginx, a CrashLoopBackOff pod, a filling disk - and gives you two scores: did the agent **fix it**, and did it **fix it the right way**.

Scenarios can be authored two ways: as files in a repository, run with the [CLI runner](/0.16/benchmark/quickstart) and Ansible playbooks; or created, run, and reviewed entirely in Command Center, with setup, validation, and restore steps written as plain shell commands. See [Author in Command Center](/0.16/benchmark/command-center) for the UI-native path.

## Why benchmark

<CardGroup cols={2}>
  <Card title="Validate new ticket typologies" icon="vial">
    Before letting a new specialty handle real tickets, replicate the failure mode in a sandbox and see how the agent handles it across dozens of runs.
  </Card>

  <Card title="Catch regressions" icon="chart-line">
    A previously-passing scenario starts failing after a specialty edit. Trend lines surface it immediately.
  </Card>

  <Card title="Compare models or prompts" icon="scale-balanced">
    Same scenario, different `--main-engine`. Same scenario, different specialty. The compliance score tells you which behaves better, not just which is faster.
  </Card>

  <Card title="Prove compliance to your team" icon="clipboard-check">
    Auditors get a record: which actions the agent took, which operational rules it followed, which commands it avoided.
  </Card>
</CardGroup>

## The two scores

Every scenario produces two **independent** scores.

| Score | Question | How it's measured |
| - | - | - |
| **Pass rate** | Did the agent actually fix the problem? | Ground-truth check against the host's state after the agent finishes (typically an Ansible playbook). |
| **Compliance** | Did the agent follow your process? | Pattern-matching against executed commands, task summaries, the agent's reasoning, injected rules. |

A scenario **passes only when both gates pass**. The split is intentional:

* An agent can fix a problem in a way that **violates your processes** - passes task validation, fails compliance.
* An agent can follow every right step and **still leave the system broken** - passes compliance, fails task validation.

You want both green.

## How a scenario runs

```
Pre-flight ─▶ Provision ─▶ Prepare ─▶ Execute ─▶ Validate ─▶ Restore ─▶ Report
   │            │            │           │           │           │           │
   verify       resolve      run         dispatch    score the   reset the   write
   env + DB     host +       prepare.yml ticket to   run         host or     ScenarioReport
                agent        (introduce  the agent              tear down   to the DB
                             the failure)                        the VM
```

The runner mirrors how real tickets flow through 2501 - same gateway, same agent, same orchestrator - but with a controlled environment and a scoring harness around it. See [Playbooks](/0.16/benchmark/playbooks) for the execution diagram in full.

## Where benchmarks run

| Mode | When to use |
| - | - |
| **`--mode host`** (default) | Pre-provisioned VMs you maintain. Simplest setup, most production-like. |
| **`--mode lima`** | macOS sandbox. The runner provisions a fresh Lima VM per run, destroys it after. |
| **`--mode incus`** | Linux sandbox. Same idea as lima but with Incus. |

Both VM modes let you run scenarios `--parallel` for throughput, and guarantee every run starts from an identical, known-good state. See [VM Sandbox](/0.16/benchmark/sandbox) for setup.

## Prerequisites

* **A running 2501 instance** reachable from the runner machine (the runner connects to the database via `DATABASE_URL` and optionally to a gateway or the engine API).
* **Ansible** installed and on PATH - playbooks drive prepare / validate / restore.
* **SSH access** from the runner machine to your sandbox hosts (or Lima/Incus installed for VM modes).
* **Scenarios** in a directory the runner can find (auto-detected, or set with `-p`).

## What to read next

<CardGroup cols={2}>
  <Card title="Quickstart" icon="play" href="/0.16/benchmark/quickstart">
    Run your first scenario in 5 minutes.
  </Card>

  <Card title="Authoring scenarios" icon="pen-ruler" href="/0.16/benchmark/scenario">
    Write `scenario.json`, hosts, agents, playbooks, validators.
  </Card>

  <Card title="Running" icon="terminal" href="/0.16/benchmark/start">
    The `2501 runner start / validate / flush` reference.
  </Card>

  <Card title="VM Sandbox" icon="cube" href="/0.16/benchmark/sandbox">
    Ephemeral VMs for fully reproducible runs.
  </Card>

  <Card title="Author in Command Center" icon="window-maximize" href="/0.16/benchmark/command-center">
    Create, run, and review scenarios from the UI - no files or Ansible.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.