Skip to main content
2501-runner run -s <scenarios> [options]

​
Execution Flow

Every 2501-runner run call goes through these phases: Pre-flight: validates the environment: configuration load, DB connectivity, org/tenant/user IDs, gateway connectivity, and Ansible availability. Scenario Discovery: scans the scenarios directory, loads all scenario.json files, resolves the -s argument against available keys and tags. Per-Scenario Loop: for each matched scenario (and each iteration when -i > 1):
  • Provision: resolve host and agent from the DB (or boot a VM clone in incus/lima mode)
  • Prepare: run restore.yml silently to clear stale state, then run prepare.yml (abort if it fails)
  • Flush: flush memory for the scenario’s agents
  • Execute: create a ticket through the selected gateway, then poll until the job is found and completes
  • Validate: evaluate the validation rules declared in scenario.json, and run validate.yml if the scenario declares it
  • Restore / Teardown: in host mode run restore.yml to reset the host (non-fatal if it fails); in VM modes destroy the clone
Report: prints a summary table and persists a Benchmark plus per-scenario ScenarioReport rows to the DB. The process exits 0 unless a pre-flight or infrastructure error occurs, or --fail-on-error is set and at least one scenario failed.

​
Options

OptionDefaultDescription
-s, --scenarios <list>requiredScenario keys or tags, comma-separated. Mix freely.
--source <source>gitWhere scenarios are loaded from: git | fs
-m, --mode <mode>hostExecution mode: host | incus | lima
-g, --gateway <type>runnerGateway used to submit tickets: runner | servicenow
-i, --iter <n>1Number of iterations per scenario
-p, --scenarios-path <path>auto-detectedPath to the scenarios root directory
--vm-templates-path <path>auto-detectedPath to the VM templates directory (incus/lima modes)
--env-file <path>auto-detectedPath to the env file (falls back to process.env)
--main-engine <engine>Override the main engine for all agents in this run
--secondary-engine <engine>Override the secondary engine for all agents in this run
--specialty <key>Override the specialty for all agents in this run
--gateway-engine <model>Override the tenant llm_model for all tickets submitted via the runner gateway
--gateway-multimodal-engine <model>Override the tenant multimodal_llm_model for all tickets submitted via the runner gateway
--parallelRun scenarios concurrently using available RAM. Requires --mode incus or --mode lima.
--parallel-ram-cap <gb>Cap the RAM budget for parallel execution in GB. Requires --parallel.
--fail-on-errorExit with code 1 if any scenario fails
--log-file <path>Mirror all output (ANSI-stripped) to a file
-v / -vv-v: show failed checks. -vv: show all checks with full debug content.

​
Gateways

The -g, --gateway flag controls how each scenario’s ticket is submitted. Every scenario is dispatched by creating a ticket; the runner then waits for the resulting job to complete. --gateway runner (default): POSTs directly to the engine’s runner gateway endpoint. Requires ENGINE_API_URL and ENGINE_API_KEY. 
2501-runner run -s nginx/001-broken-config --gateway runner
--gateway servicenow: creates an incident in ServiceNow, exercising the full integration (gateway to job to agent to ticket resolution). Requires the SERVICENOW_* env vars. 
2501-runner run -s nginx/001-broken-config --gateway servicenow

​
Scenario Source

The --source flag controls where scenario files come from. --source git (default): clones the repository given by GIT_RUNNER_REPOSITORY (branch GIT_REF, default main) using GIT_RUNNER_PAT, then reads scenarios from the clone. With --scenarios-path, the path is resolved relative to the repo root. --source fs: reads scenarios from the local filesystem at --scenarios-path (or an auto-detected default).

​
Environment Variables

VariableRequiredDescription
DATABASE_URLalwaysPostgreSQL connection string
ORG_IDalwaysYour organization ID
TENANT_IDalwaysYour tenant ID
USER_IDalwaysThe user ID under which scenarios run
ENGINE_API_URL--gateway runnerBase URL of the 2501 engine API
ENGINE_API_KEY--gateway runnerAPI key for engine authentication
SERVICENOW_API_URL--gateway servicenowServiceNow instance URL
SERVICENOW_USERNAME--gateway servicenowServiceNow username
SERVICENOW_PASSWORD--gateway servicenowServiceNow password
SERVICENOW_ASSIGNMENT_GROUP_ID--gateway servicenowAssignment group for created incidents
SERVICENOW_CALLER_ID--gateway servicenowCaller ID for created incidents
GIT_RUNNER_REPOSITORY--source gitGit URL of the scenarios repository
GIT_RUNNER_PAT--source gitPersonal access token used to clone the repository
GIT_REF--source gitBranch to clone (defaults to main)

​
Common Patterns

# Run a single scenario
2501-runner run -s nginx/001-broken-config

# Run all scenarios with a tag
2501-runner run -s nginx

# Mix tags and explicit keys
2501-runner run -s nginx,disk/001-cleanup

# 5 iterations
2501-runner run -s nginx/001-broken-config -i 5

# Submit through ServiceNow instead of the runner gateway
2501-runner run -s nginx/001-broken-config --gateway servicenow

# Run scenarios from the local filesystem with a log file
2501-runner run -s regression-suite --source fs -i 3 --log-file ./runner.log

# Parallel VM runs
2501-runner run -s nginx --mode lima --parallel --parallel-ram-cap 16