Skip to main content

​
Execution Flow

Understanding what the runner does on each invocation helps when writing scenarios and debugging failures. Every 2501-runner run call goes through these phases:

​
1. Pre-flight

Before any scenario executes, the runner validates the full environment:
  • Connects to the database and verifies ORG_ID, TENANT_ID, USER_ID exist
  • If --from gateway: tests ServiceNow API connectivity
  • If --from ticket: tests engine API connectivity
  • Verifies ansible-playbook is available in PATH Use --check to run only this step without executing anything.

​
2. Scenario Discovery

The runner scans the scenarios directory, loads all scenario.json files, and resolves the -s argument against available keys and tags.

​
3. Per-Scenario Loop

For each matched scenario (and each iteration when -i > 1): Provision Resolve host and agent records from the database by their IDs, flush agent memory. Prepare
  • Run a silent, error-suppressed restore.yml first (to clear stale state from any previous failed run)
  • Run prepare.yml with the Ansible inventory; abort the scenario if it fails
Execute
  • Dispatch the scenario through the selected entry point (see Entry Points below)
  • Poll for job or task completion; check allowedAgents/allowedHosts on every poll cycle
Validate
  • Evaluate all validation rules in order
  • Run the Ansible validate.yml playbook if declared
  • Compute compliance score and pass/fail result
Restore
  • Run restore.yml to reset the host to baseline
  • Restore failures are non-fatal: a warning is logged and execution continues

​
4. Report

After all scenarios complete, the runner:
  • Prints a summary table (pass/fail, duration, token usage, validation details per rule)
  • Persists a ScenarioReport to the database
  • Exits 0 if all passed, or 1 if any failed and --fail-on-error is set

​
Environment Setup

Environment variables are loaded automatically from /etc/2501/env.runner. Override the path with --env-file.

​
Required

​
ServiceNow (--from gateway)

​
Engine API (--from ticket)

​
The run Command

​
Options

​
Selecting Scenarios

The -s flag accepts scenario keys or tags, comma-separated. To run all your scenarios at once, tag them with a common tag (e.g. all) and use that.

​
Entry Points

The --from flag controls how each scenario is dispatched. Each entry point exercises a different layer of the stack.

​
--from gateway (default)

Creates a ticket in ServiceNow. The gateway bot processes it, creates a job, and the agent resolves it. The bot then marks the ticket resolved. This is the most complete end-to-end path: it exercises the full integration between your ticketing system and your 2501 deployment.
Because the engine selects agents automatically, use allowedAgents in validation if you need to assert which agent was chosen. If an unauthorized agent is used, the runner kills the job immediately. Requires: --gateway servicenow and the SERVICENOW_* env vars.

​
--from task

Creates a task directly for a single agent, bypassing the job router entirely. This is the fastest and most direct path.
Requirements:
  • The scenario must define exactly one agent and one host
  • The agent must be referenced by agent_id
Use this when you want to benchmark a specific agent’s response to an instruction without involving the gateway or job orchestration layer.

​
--from ticket

POSTs directly to the 2501 engine’s internal ticket endpoint. The engine creates a job and routes it to agents internally.
This exercises the engine’s internal ticket-to-job flow without going through an external gateway. Requires: ENGINE_API_URL and ENGINE_API_KEY env vars.

​
Preflight Check

Run --check to validate your full configuration before executing any scenarios. Useful after environment changes or before a large batch run.
The runner validates database connectivity, org/tenant/user IDs, gateway or engine API reachability, specialty keys, and Ansible availability. No scenarios are executed.

​
Verbosity

​
Iterating

Run each scenario multiple times to check for consistency and surface flaky behavior:
The full prepare → execute → validate → restore cycle runs for each iteration.

​
Overriding Engines

Override the LLM engines for all agents across all scenarios in a run. Useful for comparing how different models perform on the same scenario set.

​
CI Integration

Use --fail-on-error to exit non-zero when any scenario fails:
Multiple iterations for regression detection:

​
The validate Command

Re-runs the validation rules for a scenario against an existing job or task, without re-executing the scenario. Use this when iterating on validation rules and you don’t want to wait for another full agent run.

​
The restore Command

Re-runs the restore.yml playbook for a scenario. Use this to manually reset a host that was left in a dirty state after a failed or interrupted run.