start

2501 runner start -s <scenarios> [options]

Execution Flow

Every 2501 runner start call goes through these phases: Pre-flight: validates the environment: configuration load, DB connectivity, org/tenant/user IDs, gateway connectivity, and Ansible availability. Scenario Discovery: scans the scenarios directory, loads all scenario.json files, resolves the -s argument against available keys and tags. Per-Scenario Loop: for each matched scenario (and each iteration when -i > 1):

Provision: resolve host and agent from the DB (or boot a VM clone in incus/lima mode)
Prepare: run restore.yml silently to clear stale state, then run prepare.yml (abort if it fails)
Flush: flush memory for the scenario’s agents
Execute: create a ticket through the selected gateway, then poll until the job is found and completes
Validate: evaluate the validation rules declared in scenario.json, and run validate.yml if the scenario declares it
Restore / Teardown: in host mode run restore.yml to reset the host (non-fatal if it fails); in VM modes destroy the clone

Report: prints a summary table and persists a Benchmark plus per-scenario ScenarioReport rows to the DB. The process exits 0 unless a pre-flight or infrastructure error occurs, or --fail-on-error is set and at least one scenario failed.

Options

Option	Default	Description
`-s, --scenarios <list>`	required	Scenario keys or tags, comma-separated. Mix freely.
`--source <source>`	`git`	Where scenarios are loaded from: `git` \| `fs`
`-m, --mode <mode>`	`host`	Execution mode: `host` \| `incus` \| `lima`
`-g, --gateway <type>`	`runner`	Gateway used to submit tickets: `runner` \| `servicenow`
`-i, --iter <n>`	`1`	Number of iterations per scenario
`-p, --scenarios-path <path>`	auto-detected	Path to the scenarios root directory
`--vm-templates-path <path>`	auto-detected	Path to the VM templates directory (`incus`/`lima` modes)
`--env-file <path>`	auto-detected	Path to the env file (falls back to `process.env`)
`--main-engine <engine>`		Override the main engine for all agents in this run
`--secondary-engine <engine>`		Override the secondary engine for all agents in this run
`--specialty <key>`		Override the specialty for all agents in this run
`--gateway-engine <model>`		Override the tenant `llm_model` for all tickets submitted via the runner gateway
`--gateway-multimodal-engine <model>`		Override the tenant `multimodal_llm_model` for all tickets submitted via the runner gateway
`--parallel`		Run scenarios concurrently using available RAM. Requires `--mode incus` or `--mode lima`.
`--parallel-ram-cap <gb>`		Cap the RAM budget for parallel execution in GB. Requires `--parallel`.
`--fail-on-error`		Exit with code 1 if any scenario fails
`--log-file <path>`		Mirror all output (ANSI-stripped) to a file
`-v` / `-vv`		`-v`: show failed checks. `-vv`: show all checks with full debug content.

Gateways

The -g, --gateway flag controls how each scenario’s ticket is submitted. Every scenario is dispatched by creating a ticket; the runner then waits for the resulting job to complete. --gateway runner (default): POSTs directly to the engine’s runner gateway endpoint. Requires ENGINE_API_URL and ENGINE_API_KEY.

2501 runner start -s nginx/001-broken-config --gateway runner

--gateway servicenow: creates an incident in ServiceNow, exercising the full integration (gateway to job to agent to ticket resolution). Requires the SERVICENOW_* env vars.

2501 runner start -s nginx/001-broken-config --gateway servicenow

Scenario Source

The --source flag controls where scenario files come from. --source git (default): clones the repository given by GIT_RUNNER_REPOSITORY (branch GIT_REF, default main) using GIT_RUNNER_PAT, then reads scenarios from the clone. With --scenarios-path, the path is resolved relative to the repo root. --source fs: reads scenarios from the local filesystem at --scenarios-path (or an auto-detected default).

Environment Variables

Variable	Required	Description
`DATABASE_URL`	always	PostgreSQL connection string
`ORG_ID`	always	Your organization ID
`TENANT_ID`	always	Your tenant ID
`USER_ID`	always	The user ID under which scenarios run
`ENGINE_API_URL`	`--gateway runner`	Base URL of the 2501 engine API
`ENGINE_API_KEY`	`--gateway runner`	API key for engine authentication
`SERVICENOW_API_URL`	`--gateway servicenow`	ServiceNow instance URL
`SERVICENOW_USERNAME`	`--gateway servicenow`	ServiceNow username
`SERVICENOW_PASSWORD`	`--gateway servicenow`	ServiceNow password
`SERVICENOW_ASSIGNMENT_GROUP_ID`	`--gateway servicenow`	Assignment group for created incidents
`SERVICENOW_CALLER_ID`	`--gateway servicenow`	Caller ID for created incidents
`GIT_RUNNER_REPOSITORY`	`--source git`	Git URL of the scenarios repository
`GIT_RUNNER_PAT`	`--source git`	Personal access token used to clone the repository
`GIT_REF`	`--source git`	Branch to clone (defaults to `main`)

Common Patterns

# Run a single scenario
2501 runner start -s nginx/001-broken-config

# Run all scenarios with a tag
2501 runner start -s nginx

# Mix tags and explicit keys
2501 runner start -s nginx,disk/001-cleanup

# 5 iterations
2501 runner start -s nginx/001-broken-config -i 5

# Submit through ServiceNow instead of the runner gateway
2501 runner start -s nginx/001-broken-config --gateway servicenow

# Run scenarios from the local filesystem with a log file
2501 runner start -s regression-suite --source fs -i 3 --log-file ./runner.log

# Parallel VM runs
2501 runner start -s nginx --mode lima --parallel --parallel-ram-cap 16

Get Started

Authoring

Running

VM Sandbox

Execution Flow

Options

Gateways

Scenario Source

Environment Variables

Common Patterns

​Execution Flow

​Options

​Gateways

​Scenario Source

​Environment Variables

​Common Patterns

Execution Flow

Options

Gateways

Scenario Source

Environment Variables

Common Patterns