Skip to main content
Goal: run a single nginx-broken-config scenario against an agent, see the result, and read the report. ~5 minutes once the prerequisites are in place.

Prerequisites

You needHow
A running 2501 instance with at least one host + agentSee Quickstart
2501 CLI installed and signed in (2501 status works)See CLI
Ansible on PATHbrew install ansible / apt install ansible
A sandbox host the agent can SSH intoA spare VM, container, or Lima/Incus instance
If you’d rather have the runner provision an ephemeral VM for you, jump to VM Sandbox instead — the rest of this page assumes --mode host with a pre-provisioned target.

Step 1 — get a scenarios directory

The runner reads scenarios from a directory. The fastest path is to clone the 2501 scenarios examples repo (your account team can share the URL), or write your own.

Step 2 — write scenario.json

scenario.json
Replace the two IDs with real ones from your tenant (find them in Command Center → Hosts and → Agents).

Step 3 — write the playbooks

The runner needs three Ansible playbooks: prepare.yml introduces the failure, validate.yml checks the fix worked, restore.yml resets the host.
prepare.yml
validate.yml
restore.yml
Add an inventory.ini so Ansible knows how to reach the host:
inventory.ini

Step 4 — run it

From the repo root that contains ./scenarios/:
The runner walks through Provision → Prepare → Execute → Validate → Restore, prints a summary table, and writes a ScenarioReport to the database.

Step 5 — read the report

Two places:
  • Terminal output — pass/fail per rule, plus the two aggregate scores.
  • Command Center → Benchmarks — the same run, charted over time alongside others.
Pass an iteration count if you want more confidence than a single run gives you:
10 runs is usually enough to surface flakes and gauge consistency. For full confidence, run 100+ iterations as part of a CI job.

Scenario structure

Every field of scenario.json, with examples.

Validation rules

Every validator, every where: target, the scoring model.

Playbooks

What prepare / validate / restore each do, and when they run.

VM Sandbox

Skip the manual host setup — let the runner provision a fresh VM per run.