Skip to main content
Production is the wrong place to test a new agent. Benchmark runs your agents through realistic, reproducible scenarios in a sandbox — a broken nginx, a CrashLoopBackOff pod, a filling disk — and gives you two scores: did the agent fix it, and did it fix it the right way.

​
Why benchmark

Validate new ticket typologies

Before letting a new specialty handle real tickets, replicate the failure mode in a sandbox and see how the agent handles it across dozens of runs.

Catch regressions

A previously-passing scenario starts failing after a specialty edit. Trend lines surface it immediately.

Compare models or prompts

Same scenario, different --main-engine. Same scenario, different specialty. The compliance score tells you which behaves better, not just which is faster.

Prove compliance to your team

Auditors get a record: which actions the agent took, which operational rules it followed, which commands it avoided.

​
The two scores

Every scenario produces two independent scores. A scenario passes only when both gates pass. The split is intentional:
  • An agent can fix a problem in a way that violates your processes — passes task validation, fails compliance.
  • An agent can follow every right step and still leave the system broken — passes compliance, fails task validation.
You want both green.

​
How a scenario runs

The runner mirrors how real tickets flow through 2501 — same gateway, same agent, same orchestrator — but with a controlled environment and a scoring harness around it. See Playbooks for the execution diagram in full.

​
Where benchmarks run

Both VM modes let you run scenarios --parallel for throughput, and guarantee every run starts from an identical, known-good state. See VM Sandbox for setup.

​
Prerequisites

  • A running 2501 instance reachable from the runner machine (the runner connects to the database via DATABASE_URL and optionally to a gateway or the engine API).
  • Ansible installed and on PATH — playbooks drive prepare / validate / restore.
  • SSH access from the runner machine to your sandbox hosts (or Lima/Incus installed for VM modes).
  • Scenarios in a directory the runner can find (auto-detected, or set with -p).

Quickstart

Run your first scenario in 5 minutes.

Authoring scenarios

Write scenario.json, hosts, agents, playbooks, validators.

Running

The 2501 runner start / validate / flush reference.

VM Sandbox

Ephemeral VMs for fully reproducible runs.