Quickstart

This walkthrough builds a small scenario end to end: stop nginx on a test host, ask the agent to fix it, check that nginx serves again, and put the host back. It takes about 15 minutes. To follow it, you need:

• an Admin role,
• a Linux host running nginx that you can afford to break - an existing test VM, or a new one you spin up for the occasion (apt install nginx is enough),
• an agent on that host, so 2501 can reach it.

​

1. Create a Runner gateway

The ticket reaches the agent through a gateway. The Runner gateway creates it inside 2501, so nothing touches your ticketing system. Go to Gateways → New Gateway, pick type Runner, and save. You only do this once per organization.

​

2. Create the scenario

Go to Benchmarks → New Scenario and fill in the form. Scenario Definition Write the ticket the way a user or a monitoring alert would: symptoms only, never the cause or the fix. Evaluated Agents - add the agent on your test host. Its host becomes the target of every command below. Target Setup - the commands that break the host. Add three steps: The first step makes sure the host starts from a known state even if an earlier run was interrupted. The last step proves the incident is really there: if nginx is somehow still running, setup fails instead of grading the agent against a healthy host. Validation - the rules that grade the run. Add three:

• Job resolved successfully - validator job_resolution_status, statuses success.
• Did not reboot the host - validator pattern_match, where executed_commands, pattern \b(reboot|shutdown|poweroff)\b, negate on.
• Nginx serves again - validator host_state, run curl -sf -o /dev/null http://localhost.

The host_state rule is the important one: it checks the machine itself instead of trusting what the agent says it did. Target Restore - the commands that put the host back: Click Save. The scenario is created as a draft.

​

3. Test setup and restore without an agent

On the scenario’s page, click Target Setup. It runs only the setup commands, with each step’s status and output - no ticket, no grading. Check that every step passed and that nginx is really stopped on the host. Then click Target Restore and check that nginx is back. Run both a second time. The second setup proves your scenario recovers from a host an earlier run left behind. Fix any failing step and repeat until both pass twice in a row.

​

4. Run it

Click Run now. Keep the defaults - Runner gateway and your current engines - and launch. The benchmark page shows the run moving through its stages: target setup, clearing the agents’ memory, handing over the ticket, the agentic lifecycle, collecting, evaluation, and target restore.

​

5. Read the result

When the run ends, open it. You get:

• the verdict (Pass or Fail),
• the compliance score,
• remediation (did the job resolve as success),
• each validation check with what it inspected.

If the run shows Error instead of a verdict, the scenario or the platform broke before grading - see Reading results to find out which.

​

6. Run it again, then publish

One run of an agent proves nothing: models are not deterministic. Run the scenario several times, or put it in a benchmark with others, and look at the pass rate across runs. When you trust the scenario, edit it and set its status to published. Published and draft scenarios both run; publishing tells your team the scenario is ready.

​

Next steps