> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarking for Risk

> Validate new use cases on test machines before exposing them to production

When a new ticket type arrives, you don't have to test it in production. Stage the incident on a test machine you can afford to break, let the agent work it, and grade what it did - as many times as you need to trust it.

## What gets measured

Each scenario run produces three results:

| Result | Question |
| - | - |
| **Pass / Fail** | Did the run meet every required rule? |
| **Compliance** | How many of your rules passed, required and optional? |
| **Remediation** | Did the job resolve as `success`? |

Your rules decide what "right" means. Two kinds matter most for risk:

### The outcome

A `host_state` rule runs a command on the machine after the agent is done - is the service up, is the port reachable, is the file gone - and checks its exit code. It verifies the end result regardless of how the agent got there, and it never takes the agent's word for it.

For diagnose-only tickets, the same check proves the agent **changed nothing**: the fault must still be there.

### The behavior

Rules on what the agent did:

* which commands it ran, and which it must never run (reboot, flush a firewall, delete data),
* which tools it used,
* what its summary and ticket reply say,
* which operational rules it was given,
* how many tasks the job needed.

Making the "never do" rules required means a run that fixes the problem the wrong way still fails. Compliance gives auditors a record that agents follow company practice - useful in regulated industries.

## Automation

Benchmarks run on demand from Command Center, or from the [API](/0.17/benchmark/api), so you can trigger them from your own scheduler or CI. Results appear in **Command Center → Benchmarks** with pass rate, compliance, and trends over time, so you can spot regressions early.

Recommended cadence: a **weekly or nightly run** of your full scenario suite, plus a run of the scenarios touching a specialty or rule **before you change it** in production.

## What benchmarking is good for

| Use case | Why |
| - | - |
| Validating a new ticket type before exposing it to production | Stage the failure, see how the agent handles it across many runs |
| Comparing two specialties | Same scenarios, a different specialty on the agents, see which passes more often and more cleanly |
| Catching regressions after a rule edit | Trend lines show the moment a previously passing scenario starts failing |
| Choosing a model | Same scenarios, a different **Main engine** at launch |

See [Benchmark](/0.17/benchmark/overview) to get started, and [Designing scenarios](/0.17/benchmark/designing) to build scenarios you can trust.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.