Skip to main content
When a new ticket type arrives, you don’t have to test it in production. Stage the incident on a test machine you can afford to break, let the agent work it, and grade what it did - as many times as you need to trust it.

​
What gets measured

Each scenario run produces three results: Your rules decide what “right” means. Two kinds matter most for risk:

​
The outcome

A host_state rule runs a command on the machine after the agent is done - is the service up, is the port reachable, is the file gone - and checks its exit code. It verifies the end result regardless of how the agent got there, and it never takes the agent’s word for it. For diagnose-only tickets, the same check proves the agent changed nothing: the fault must still be there.

​
The behavior

Rules on what the agent did:
  • which commands it ran, and which it must never run (reboot, flush a firewall, delete data),
  • which tools it used,
  • what its summary and ticket reply say,
  • which operational rules it was given,
  • how many tasks the job needed.
Making the “never do” rules required means a run that fixes the problem the wrong way still fails. Compliance gives auditors a record that agents follow company practice - useful in regulated industries.

​
Automation

Benchmarks run on demand from Command Center, or from the API, so you can trigger them from your own scheduler or CI. Results appear in Command Center → Benchmarks with pass rate, compliance, and trends over time, so you can spot regressions early. Recommended cadence: a weekly or nightly run of your full scenario suite, plus a run of the scenarios touching a specialty or rule before you change it in production.

​
What benchmarking is good for

See Benchmark to get started, and Designing scenarios to build scenarios you can trust.