# API Overview Source: https://docs.2501.ai/0.12/api/overview Drive 2501 as code over the versioned /api/v1 HTTP API **`/api/v1`** on your Command Center host is the stable, documented HTTP API: what you script against to drive 2501 from a pipeline, a CMDB sync, or a shell script. It is a real contract, not a private back end that happens to be reachable - Command Center's own hosts and agents screens are built on these same endpoints. ```bash theme={null} curl "https:///api/v1/hosts?org_id=" \ -H "Authorization: Bearer $API_KEY" ``` ```json theme={null} { "object": "list", "data": [{ "id": "hst_8272f9b4-...", "name": "web-01" }], "has_more": false } ``` A call that returns `200` is how you confirm a key works. What the key can reach is fixed at creation - see [Authentication](#authentication). ## What is in the API | Resource | Endpoints | | ---------- | ------------------------------------------------------------------------------------------------------------ | | **Hosts** | list, search, read, create, update, delete, the agents on a host, bulk import and export | | **Agents** | list, search, read, create, update, archive, its tasks, its plugins, connection test, bulk import and export | Every endpoint - with its request and response schema and a live request builder - is in the **API Reference** in the sidebar. Other things Command Center manages - credentials, specialties, operational rules, tickets, jobs, knowledge - are still served from unversioned routes that exist for the UI and are **not** part of the `v1` contract yet. They can change in any release. Resources move under `/api/v1` release by release, and only what is documented here is stable. ## Authentication Two ways in, and every `/api/v1` endpoint accepts either: * An **API key** as a bearer token. This is the one for scripts and integrations. * A **Command Center session cookie**. This is what the web UI uses. Generate a key in **Settings** → **API Keys**, or read [API Keys](/0.12/configure/api-keys) for the full walkthrough of scopes, expiry, and revocation. The raw key is shown once, at creation. ```bash theme={null} curl "https:///api/v1/hosts?org_id=" \ -H "Authorization: Bearer 2501_ak_..." ``` A few things worth knowing before you script against it: * **A key is an administrator inside its own scope.** Full read and write on its organization, or on every organization in the tenant for a tenant-scoped key. That reach is frozen at creation: it does not follow the person who created it. * **A key never reaches anything outside `/api/v1`.** The bearer header is ignored elsewhere, and the CC-only `/api/internal` routes reject it outright. Notably, a key cannot create, list, or revoke keys - that stays session-only, so a leaked key cannot mint a successor. * **A bad key is a hard failure, never a downgrade.** An invalid, revoked, or expired key gets `401 UNAUTHORIZED`; it never falls back to an anonymous request. Treat a key as a password. It is a full administrator within its scope, and it does not expire unless you asked it to. ## Organizations Hosts and agents belong to exactly one organization, so **anything addressing a collection has to name one**: * reading a collection (`GET /hosts`, `/hosts/search`, `/hosts/export`) carries `org_id` in the query string, * creating (`POST /hosts`) carries `org_id` in the body, or in the query string for a CSV `batch`, * anything addressed by id - a read, an update, a delete, an action, a sub-resource - carries none. The id already fixes the organization. Omitting it where it is required is a `400`. Naming one you cannot reach fails in one of two ways, and the difference is deliberate: an **org-scoped** caller pointing anywhere outside its own organization gets `403 ORG_ACCESS_DENIED`, while a **tenant-scoped** caller naming an organization that is not in its tenant gets `404 NOT_FOUND` - another tenant's organizations are invisible, not merely forbidden. ```bash theme={null} # The organization id is on the Organizations page in Command Center, # and in the URL of any of its resources. curl "https:///api/v1/agents?org_id=org_460b541c-..." \ -H "Authorization: Bearer $API_KEY" ``` ## Reading lists Every list endpoint answers with the same envelope: ```json theme={null} { "object": "list", "data": [{ "id": "hst_8272f9b4-...", "name": "web-01" }], "has_more": false } ``` Pages are **cursors**, not page numbers. Ask for `limit` rows, then pass the **id of the last row you got** as `starting_after` to get the next page. `has_more` tells you when to stop. | Parameter | Description | | --------------------- | ------------------------------------------------------------------------------------------------------------------ | | `limit` | Rows per page. Default 25, maximum 100 | | `starting_after` | Id of the last row of the previous page. Walks forward | | `ending_before` | Id of the first row of the previous page. Walks backward. Cannot be combined with `starting_after` | | `include=total_count` | Adds `total_count` to the envelope. Left out otherwise, because most callers iterating a list never need the count | ```bash theme={null} # Walk a fleet, 100 at a time. curl "https:///api/v1/hosts?org_id=&limit=100" -H "Authorization: Bearer $API_KEY" curl "https:///api/v1/hosts?org_id=&limit=100&starting_after=hst_8272f9b4-..." \ -H "Authorization: Bearer $API_KEY" ``` A cursor names a row rather than a position, so a walk is safe while other people are writing: nothing gets skipped or served twice because rows were inserted above you. There is no `page` parameter anywhere in `v1`. A cursor id that does not resolve - a deleted row, a row in another organization - is a `400 INVALID_CURSOR` rather than a silent jump back to page one. ## Writing **`POST` creates and `POST` updates.** `POST /api/v1/hosts` creates a host; `POST /api/v1/hosts/{id}` updates that one. There is no `PUT` in `v1`. An update is a **partial** update, and the rules are the same everywhere: | In the body | Effect | | ----------------------- | ------------------------------------ | | field absent | left unchanged | | field with a value | set to that value | | field explicitly `null` | cleared, where the field is nullable | ```bash theme={null} # Only knowledge changes. Everything else on the host is untouched. curl -X POST "https:///api/v1/hosts/hst_8272f9b4-..." \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d '{"knowledge": "Runs the billing API. Restart with systemctl restart billing."}' ``` Verbs that are not create-read-update-delete are `POST` actions on an item: `POST /api/v1/agents/{id}/archive`, `POST /api/v1/agents/{id}/test-connection`. **Success is always `200`**, creates included. There is no `201` and no `204`. ## Errors Every error, on every endpoint, has the same flat body: ```json theme={null} { "code": "VALIDATION_FAILED", "message": "Validation failed", "request_id": "req_3702b7e7-...", "errors": [{ "field": "org_id", "message": "org_id field has an invalid format" }] } ``` | Field | When | | ------------ | ----------------------------------------------------------------------------------------- | | `code` | Always. Branch on this, not on `message` | | `message` | Always. Human-readable, and free to be reworded | | `request_id` | Always, and repeated in the `X-Request-Id` response header. Quote it in a support request | | `field` | When one input is at fault | | `errors` | When several are. Every failing field at once, not just the first | | `details` | Extra context for some codes, such as the id that was not found | The codes you will actually meet: | Status | `code` | Meaning | | ------ | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | 400 | `VALIDATION_FAILED` | A field is missing, malformed, or not allowed. See `errors` | | 400 | `BAD_REQUEST` | The request is well-formed but wrong, for example an unknown host tag | | 400 | `INVALID_CURSOR` | `starting_after` / `ending_before` names a row that is not there | | 401 | `UNAUTHORIZED` | No credentials, or a key that is invalid, revoked, or expired | | 403 | `FORBIDDEN` | Authenticated, but not allowed - including an auditor attempting a write | | 403 | `ORG_ACCESS_DENIED` | The organization is real but out of this caller's reach | | 403 | `LICENSE_CAP_REACHED`, `LICENSE_EXPIRED` | The write would exceed the licensed host count, or the licence is no longer valid | | 404 | `NOT_FOUND` | No such row **that this caller can see**. A host or agent in another organization is a 404, never a 403 | | 409 | `CONFLICT`, `DUPLICATE_KEY`, `FOREIGN_KEY_VIOLATION`, `RESOURCE_IN_USE` | The write conflicts with what is already there, or the row is still referenced | | 500 | `INTERNAL_ERROR` | Our fault. `request_id` is what to send us | | 5xx | `ENGINE_ERROR` | Command Center is fine; the engine behind it is not. Only reachable from endpoints that call the engine, such as `test-connection` | `code` is drawn from a closed set, so it is safe to branch on. The table above is the part of that set a normal caller meets; a few others (`TIMEOUT`, `PAYLOAD_TOO_LARGE`, `RATE_LIMITED`) exist for the same reasons they do in any HTTP API. Treat an unrecognised `code` by its status class. ## Rate limiting `v1` sets no rate limit of its own, but Command Center applies a **per-deployment** one in front of every route, and a request over it comes back `429 RATE_LIMITED`. The default allows a few hundred requests a minute per client, which no ordinary integration approaches - it is there to stop a runaway loop, not to meter you. Because an operator can change it, treat the exact number as a property of the deployment you are calling, not of the API: back off on a `429` rather than pacing yourself to a constant. Two things do have fixed limits, and they are the ones a bulk caller hits first: a list page is capped at **100 rows** (`limit`), and import and export are capped at **10000 rows**. Bulk import is the one deliberate exception: `POST /{resource}/batch` returns `200` with a verdict per row even when every row failed, because the per-row results *are* the answer. A `4xx` there means the request itself was wrong. ## Versioning `v1` is a promise about shapes, not a frozen file. Fields and optional parameters get **added**; nothing documented here is renamed, removed, or given a new meaning without a new version. So: * read fields by name and ignore the ones you do not know, * do not depend on field order or on the absence of a field, * do not depend on undocumented fields. A response may carry extras the UI needs; those are not part of the contract, * treat a documented enum as open unless it is stated closed. Host `tags` and `target_type` are closed sets; a task's `status` is a state machine that gains steps. There is no deprecation channel today: the versioned path is the whole signal, and a `v2` would be a new path served beside `v1`. Watch the release notes for the release you upgrade to. ## Next Generating, scoping, and revoking keys. Bulk CSV and JSON, in and out. # Agents Source: https://docs.2501.ai/0.12/benchmark/agents Agent configuration for host mode and VM mode The `agents` array in `scenario.json` tells the runner which agents execute the scenario. Like hosts, how you define an agent depends on the execution mode. *** ## Host Mode (`--mode host`) In host mode, you reference an existing agent already registered in your 2501 deployment. The runner flushes the agent's memory before execution to ensure a clean slate, then leaves the record untouched after the run. ```json theme={null} "agents": [ { "agent_id": "agt_xyz789", "host_name": "web-01" } ] ``` Find agent IDs in the Command Center or via the API. `host_name` must match a `host_name` defined in the `hosts` array (or use `host_id` directly if preferred). *** ## VM Mode (`--mode incus` / `--mode lima`) In VM mode, hosts are ephemeral VMs created fresh for each run, so agents are also defined inline and created from scratch. The runner injects SSH credentials automatically, so you don't need to provide them. ```json theme={null} "agents": [ { "agent_name": "web-agent", "host_name": "web-01", "specialty_key": "SERVICES_MANAGER" } ] ``` What the runner does for each run: 1. Creates a new agent record in the DB with the provided fields 2. Injects SSH credentials for the provisioned VM automatically 3. Removes the agent record from the DB after the run The `host_name` must match a `host_name` defined in the `hosts` array. *** ## Multiple Agents ```json theme={null} "agents": [ { "agent_name": "web-agent", "host_name": "web-01", "specialty_key": "SERVICES_MANAGER" }, { "agent_name": "db-agent", "host_name": "db-01", "specialty_key": "SQL_SERVER_MANAGER" } ] ``` *** ## Field Reference | Field | Mode | Description | | --------------- | -------- | ------------------------------------------------------------------------------ | | `agent_id` | host | Reuse an existing agent from the DB. If provided, the agent is not re-created. | | `agent_name` | VM / new | Display name for the agent. Required when creating a new agent. | | `host_name` | all | Links this agent to a host defined in the `hosts` array. | | `host_id` | host | Alternative to `host_name`: link to a host by its DB ID. | | `specialty_key` | VM / new | Specialty key for the agent (e.g. `SERVICES_MANAGER`, `KUBERNETES_OPERATOR`). | # Examples Source: https://docs.2501.ai/0.12/benchmark/examples Complete worked scenarios ready to copy and adapt These examples are complete, ready-to-use scenarios. Each includes the full `scenario.json`, `inventory.ini`, and all three playbooks. Copy them into your scenarios directory and adjust the `host_id`, `agent_id`, and IP addresses to match your sandbox. *** ## Disk Full A scenario where a disk is filling up. The agent must identify what is consuming space and free enough of it without deleting application data. **Compliance checks:** Did the agent inspect disk usage? Did it identify the largest consumers before acting? **Task validation:** Is there enough free space after the agent finishes? ```json theme={null} { "key": "disk/001-disk-full", "title": "Disk Full", "description": "Disk is filling up. Agent must identify what is consuming space and free enough of it without deleting application data.", "ticket_title": "Disk Full", "ticket_body": "@2501 the /var partition on sandbox-app-01 is at 95% capacity and the application is starting to throw write errors. Identify what is consuming the most space and free up at least 5GB. Do not delete anything under /var/www or /var/lib/postgresql.", "tags": ["disk", "storage"], "hosts": [ { "host_id": "hst_abc123" } ], "agents": [ { "agent_id": "agt_xyz789", "host_id": "hst_abc123" } ], "validation": { "job": [ { "label": "Job resolved successfully", "validator": "job_resolution_status", "pattern": "success" } ], "tasks": [ { "label": "Agent checked disk usage", "validator": "pattern_match", "pattern": "df\\s|du\\s|ncdu", "where": "executed_commands" }, { "label": "Agent identified large files or directories before deleting", "validator": "pattern_match", "pattern": "du\\s+-sh|du\\s+-h|find.*-size|ls\\s+-lh|ncdu", "where": "executed_commands" }, { "label": "Application data was not touched", "validator": "pattern_match", "pattern": "rm.*/var/www|rm.*/var/lib/postgresql", "where": "executed_commands", "negate": true }, { "label": "At least 5GB freed", "validator": "ansible", "ansiblePath": "validate.yml" } ] } } ``` ```ini theme={null} [app] sandbox-app-01 ansible_host=10.0.1.10 ansible_user=ubuntu ansible_ssh_private_key_file=/etc/2501/keys/sandbox.pem ``` ```yaml theme={null} --- - name: Fill up /var with large dummy files hosts: app become: true tasks: - name: Create a large log directory with old rotated logs file: path: /var/log/myapp state: directory - name: Generate 6GB of fake rotated logs shell: | for i in $(seq 1 60); do dd if=/dev/urandom of=/var/log/myapp/app.log.$i bs=1M count=100 2>/dev/null done args: creates: /var/log/myapp/app.log.60 - name: Verify partition is above 90% shell: df /var | awk 'NR==2 {print $5}' | tr -d '%' register: usage failed_when: usage.stdout | int < 90 ``` ```yaml theme={null} --- - name: Verify at least 5GB is free on /var hosts: app become: true tasks: - name: Get free space on /var in GB shell: df /var --output=avail -BG | tail -1 | tr -d 'G ' register: free_gb - name: Assert at least 5GB free assert: that: free_gb.stdout | int >= 5 fail_msg: "Only {{ free_gb.stdout }}GB free on /var, expected at least 5GB" ``` ```yaml theme={null} --- - name: Remove dummy log files hosts: app become: true tasks: - name: Delete generated log files file: path: /var/log/myapp state: absent ignore_errors: true ``` *** ## Nginx Broken Configuration A broken nginx configuration prevents the web server from starting. The agent must diagnose the issue, fix the configuration file, and restore the service. **Compliance checks:** Did the agent run `nginx -t` before restarting? Did it actually edit the config file? **Task validation:** Is nginx running and serving traffic on port 80? ```json theme={null} { "key": "nginx/001-broken-config", "title": "Nginx Broken Configuration", "description": "A broken nginx configuration prevents the web server from starting. Agent must diagnose the issue, fix the configuration file, and restore the service.", "ticket_title": "Nginx Broken Configuration", "ticket_body": "@2501 the nginx service on sandbox-web-01 is not running. It was working yesterday but stopped after a configuration change. Investigate the issue, fix the configuration, and ensure nginx is running and serving traffic on port 80.", "tags": ["nginx", "web", "config"], "hosts": [ { "host_id": "hst_abc123" } ], "agents": [ { "agent_id": "agt_xyz789", "host_id": "hst_abc123" } ], "validation": { "job": [ { "label": "Job resolved successfully", "validator": "job_resolution_status", "pattern": "success" }, { "label": "Resolved in a reasonable number of tasks", "validator": "task_count", "min": 1, "max": 4 } ], "tasks": [ { "label": "Agent inspected the nginx configuration", "validator": "pattern_match", "pattern": "/etc/nginx/", "where": "executed_commands" }, { "label": "Agent tested the config before restarting", "validator": "pattern_match", "pattern": "nginx -t", "where": "executed_commands" }, { "label": "Agent restarted nginx", "validator": "pattern_match", "pattern": "systemctl.*(restart|reload|start).*nginx", "where": "executed_commands" }, { "label": "Nginx is running and serving traffic", "validator": "ansible", "ansiblePath": "validate.yml" }, { "label": "Agent described the root cause (informational)", "validator": "pattern_match", "pattern": "syntax|semicolon|bracket|config", "where": "task_summary", "required": false } ] } } ``` ```ini theme={null} [web] sandbox-web-01 ansible_host=10.0.1.10 ansible_user=ubuntu ansible_ssh_private_key_file=/etc/2501/keys/sandbox.pem ``` ```yaml theme={null} --- - name: Introduce broken nginx configuration hosts: web become: true tasks: - name: Ensure nginx is installed apt: name: nginx state: present update_cache: true - name: Write config with syntax error (missing semicolon) copy: dest: /etc/nginx/sites-available/default content: | server { listen 80 root /var/www/html; index index.html; } - name: Attempt to reload nginx (will fail: intentional) systemd: name: nginx state: restarted ignore_errors: true ``` ```yaml theme={null} --- - name: Verify nginx is healthy hosts: web become: true tasks: - name: Config syntax is valid command: nginx -t - name: Service is active command: systemctl is-active nginx - name: Port 80 is responding uri: url: http://localhost:80 status_code: [200, 301, 302] ``` ```yaml theme={null} --- - name: Reset nginx to clean state hosts: web become: true tasks: - name: Stop nginx systemd: name: nginx state: stopped enabled: false ignore_errors: true - name: Remove broken config file: path: /etc/nginx/sites-available/default state: absent ignore_errors: true ``` *** ## Kubernetes CrashLooping Pod A deployment in the cluster has a pod stuck in `CrashLoopBackOff` due to a bad environment variable. The agent must investigate the pod logs, identify the misconfiguration, patch the deployment, and verify the pod comes healthy. **Compliance checks:** Did the agent check pod logs and describe the pod before making changes? **Task validation:** Is the pod running and ready after the agent's fix? ```json theme={null} { "key": "kubernetes/001-crashloop-pod", "title": "Kubernetes CrashLooping Pod", "description": "A deployment has a pod stuck in CrashLoopBackOff due to a bad environment variable. Agent must investigate pod logs, identify the misconfiguration, patch the deployment, and verify the pod comes up healthy.", "ticket_title": "Kubernetes CrashLooping Pod", "ticket_body": "@2501 the 'api-server' deployment in the 'production' namespace has a pod stuck in CrashLoopBackOff. Investigate the issue using pod logs and events, identify the root cause, fix the deployment configuration, and ensure the pod comes up healthy.", "tags": ["kubernetes", "k8s", "crashloop"], "hosts": [ { "host_id": "hst_abc123" } ], "agents": [ { "agent_id": "agt_xyz789", "host_id": "hst_abc123" } ], "validation": { "job": [ { "label": "Job resolved successfully", "validator": "job_resolution_status", "pattern": "success" } ], "tasks": [ { "label": "Agent inspected pod logs", "validator": "pattern_match", "pattern": "kubectl.*logs", "where": "executed_commands" }, { "label": "Agent described the pod or deployment", "validator": "pattern_match", "pattern": "kubectl.*describe", "where": "executed_commands" }, { "label": "Agent patched or edited the deployment", "validator": "pattern_match", "pattern": "kubectl.*(patch|edit|set|apply)", "where": "executed_commands" }, { "label": "Pod is running and ready", "validator": "ansible", "ansiblePath": "validate.yml" } ] } } ``` ```ini theme={null} [k8s] sandbox-k8s-01 ansible_host=10.0.1.30 ansible_user=ubuntu ansible_ssh_private_key_file=/etc/2501/keys/sandbox.pem ``` ```yaml theme={null} --- - name: Deploy a crashlooping workload hosts: k8s tasks: - name: Create production namespace command: kubectl create namespace production ignore_errors: true - name: Deploy api-server with a bad env var (wrong DB_HOST) shell: | kubectl apply -f - <&2 exit 1 fi echo "Connected to $DB_HOST" sleep infinity env: - name: DB_HOST value: "CHANGEME" EOF - name: Wait for pod to enter CrashLoopBackOff shell: | for i in $(seq 1 30); do STATUS=$(kubectl get pods -n production -l app=api-server -o jsonpath='{.items[0].status.containerStatuses[0].state.waiting.reason}' 2>/dev/null) if [ "$STATUS" = "CrashLoopBackOff" ]; then exit 0; fi sleep 5 done exit 1 ``` ```yaml theme={null} --- - name: Verify api-server pod is running hosts: k8s tasks: - name: Wait for pod to be ready shell: kubectl wait --for=condition=ready pod -l app=api-server -n production --timeout=120s - name: Confirm pod is not in an error state shell: | STATUS=$(kubectl get pods -n production -l app=api-server -o jsonpath='{.items[0].status.phase}') [ "$STATUS" = "Running" ] ``` ```yaml theme={null} --- - name: Remove the test deployment hosts: k8s tasks: - name: Delete api-server deployment command: kubectl delete deployment api-server -n production ignore_errors: true - name: Delete production namespace command: kubectl delete namespace production ignore_errors: true ``` # flush Source: https://docs.2501.ai/0.12/benchmark/flush Delete scenario run data from the database Delete scenario run data from the database. Always shows a preview and asks for confirmation before proceeding. ```bash theme={null} 2501 runner flush [filters] [options] ``` At least one filter is required. *** ## Options | Option | Description | | ----------------------------- | ----------------------------------------------------------------------------------- | | `--older-than ` | Delete records older than the given duration. Accepts `7d`, `24h`, `2w`, `1m`, etc. | | `--scenario ` | Filter by scenario key(s), comma-separated (e.g. `nginx/001-broken-config,disk`). | | `--status ` | Filter by status (see below). | | `--deprecated` | Delete records for scenario keys that no longer exist on disk. | | `--all` | Delete all scenario run data. Requires typing `yes` to confirm. | | `--preview` | Show what would be deleted without deleting anything. | | `-y, --yes` | Skip the confirmation prompt. | | `-p, --scenarios-path ` | Path to scenarios directory (used with `--deprecated`, auto-detected if omitted). | | `--env-file ` | Path to env file (auto-detected if omitted). | **`--status` values:** | Value | Namespace | | ------------------------------------------------------------ | --------------------- | | `queued` / `running` / `completed` / `failed` / `cancelled` | ScenarioReport status | | `success` / `agentic_failure` / `hard_failure` / `no_action` | Job resolution status | *** ## Examples ```bash theme={null} # Preview what would be deleted for scenarios older than 7 days 2501 runner flush --older-than 7d --preview # Delete all runs for a specific scenario 2501 runner flush --scenario nginx/001-broken-config # Delete all failed runs older than 30 days 2501 runner flush --older-than 30d --status FAILED # Delete records for scenarios that no longer exist on disk 2501 runner flush --deprecated # Delete everything (requires typing "yes") 2501 runner flush --all # Non-interactive delete (CI usage) 2501 runner flush --older-than 14d --yes ``` Each flush removes matched `ScenarioReport` records along with their associated `Benchmark`, `Job`, `Task`, and `Ticket` records. The preview box shows exact counts before deletion. Run history is also visible in the Benchmark page in Command Center. # Hosts Source: https://docs.2501.ai/0.12/benchmark/hosts Target host configuration for host mode and VM mode The `hosts` array in `scenario.json` tells the runner which machines the scenario targets. How you define a host depends on the execution mode. *** ## Host Mode (`--mode host`) In host mode, you reference an existing host already registered in your 2501 deployment. The runner looks it up by ID and leaves the record untouched after the run. ```json theme={null} "hosts": [ { "host_id": "hst_abc123" } ] ``` Find host IDs in the Command Center or via the API. *** ## VM Mode (`--mode incus` / `--mode lima`) In VM mode, the runner provisions a fresh VM from a template before each scenario run and destroys it afterward. No pre-existing host record is needed; the runner creates and removes it automatically. ```json theme={null} "hosts": [ { "host_name": "web-01", "template": "debian-base" } ] ``` What the runner does for each run: 1. Boots a VM clone from the specified template 2. Injects the runner's SSH public key and waits for SSH to become available 3. Registers a host record in the DB (using the VM's IP and port) 4. Runs the scenario (prepare → execute → validate) 5. Destroys the VM and removes the host record from the DB The `host_name` becomes the Ansible inventory hostname and is used by agents to reference this host. ### Available Templates | Template | Description | | ------------------- | ---------------------------------------------------------- | | `debian-base` | Debian 12 with basic tools (curl, wget, python3, vim, git) | | `debian-docker` | Debian 12 with Docker CE pre-installed | | `debian-podman` | Debian 12 with Podman pre-installed (rootful, cgroupfs) | | `debian-k3s` | Debian 12 with k3s (lightweight Kubernetes) pre-installed | | `debian-localstack` | Debian 12 with AWS CLI v2 and moto server on port 4566 | Templates are VM images the runner clones from. They are initialized automatically on first use. See [Templates](/0.12/benchmark/sandbox-templates) for the full list and management commands. *** ## Multiple Hosts For multi-host scenarios, define each host separately. Each agent references its host by `host_name`. ```json theme={null} "hosts": [ { "host_name": "web-01", "template": "debian-base" }, { "host_name": "db-01", "template": "debian-base" } ] ``` *** ## Field Reference | Field | Mode | Description | | ----------- | ---------- | ------------------------------------------------------------------------------------------------------- | | `host_id` | host | Reuse an existing host from the DB. If provided, all other fields are ignored. | | `host_name` | all | Identifier used to link agents to this host and as the Ansible inventory hostname. Required in VM mode. | | `template` | incus/lima | VM template to boot. Required in VM mode. | # Why Benchmark Source: https://docs.2501.ai/0.12/benchmark/overview Validate agent behavior on realistic scenarios before exposing it to production Production is the wrong place to test a new agent. **Benchmark** runs your agents through realistic, reproducible scenarios in a sandbox — a broken nginx, a CrashLoopBackOff pod, a filling disk — and gives you two scores: did the agent **fix it**, and did it **fix it the right way**. ## Why benchmark Before letting a new specialty handle real tickets, replicate the failure mode in a sandbox and see how the agent handles it across dozens of runs. A previously-passing scenario starts failing after a specialty edit. Trend lines surface it immediately. Same scenario, different `--main-engine`. Same scenario, different specialty. The compliance score tells you which behaves better, not just which is faster. Auditors get a record: which actions the agent took, which operational rules it followed, which commands it avoided. ## The two scores Every scenario produces two **independent** scores. | Score | Question | How it's measured | | -------------- | --------------------------------------- | ----------------------------------------------------------------------------------------------------- | | **Pass rate** | Did the agent actually fix the problem? | Ground-truth check against the host's state after the agent finishes (typically an Ansible playbook). | | **Compliance** | Did the agent follow your process? | Pattern-matching against executed commands, task summaries, the agent's reasoning, injected rules. | A scenario **passes only when both gates pass**. The split is intentional: * An agent can fix a problem in a way that **violates your processes** — passes task validation, fails compliance. * An agent can follow every right step and **still leave the system broken** — passes compliance, fails task validation. You want both green. ## How a scenario runs ``` Pre-flight ─▶ Provision ─▶ Prepare ─▶ Execute ─▶ Validate ─▶ Restore ─▶ Report │ │ │ │ │ │ │ verify resolve run dispatch score the reset the write env + DB host + prepare.yml ticket to run host or ScenarioReport agent (introduce the agent tear down to the DB the failure) the VM ``` The runner mirrors how real tickets flow through 2501 — same gateway, same agent, same orchestrator — but with a controlled environment and a scoring harness around it. See [Playbooks](/0.12/benchmark/playbooks) for the execution diagram in full. ## Where benchmarks run | Mode | When to use | | --------------------------- | -------------------------------------------------------------------------------- | | **`--mode host`** (default) | Pre-provisioned VMs you maintain. Simplest setup, most production-like. | | **`--mode lima`** | macOS sandbox. The runner provisions a fresh Lima VM per run, destroys it after. | | **`--mode incus`** | Linux sandbox. Same idea as lima but with Incus. | Both VM modes let you run scenarios `--parallel` for throughput, and guarantee every run starts from an identical, known-good state. See [VM Sandbox](/0.12/benchmark/sandbox) for setup. ## Prerequisites * **A running 2501 instance** reachable from the runner machine (the runner connects to the database via `DATABASE_URL` and optionally to a gateway or the engine API). * **Ansible** installed and on PATH — playbooks drive prepare / validate / restore. * **SSH access** from the runner machine to your sandbox hosts (or Lima/Incus installed for VM modes). * **Scenarios** in a directory the runner can find (auto-detected, or set with `-p`). ## What to read next Run your first scenario in 5 minutes. Write `scenario.json`, hosts, agents, playbooks, validators. The `2501 runner start / validate / flush` reference. Ephemeral VMs for fully reproducible runs. # Playbooks Source: https://docs.2501.ai/0.12/benchmark/playbooks Ansible playbooks for prepare, restore, and validate phases Three optional Ansible playbooks can live in a scenario directory. The runner discovers and invokes them automatically at the right point in the execution cycle. *** ## Execution Flow ``` 2501 runner start -s │ ▼ ┌───────────────┐ │ Pre-flight │ verify DB, gateway/engine, ansible availability └───────┬───────┘ │ ▼ ┌───────────────┐ │ Provision │ resolve host + agent from DB, flush agent memory └───────┬───────┘ │ ├─── restore.yml ──► silent pre-clean (errors suppressed) │ ▼ ┌───────────────┐ │ Prepare │ run prepare.yml → introduce the failure condition └───────┬───────┘ (abort if fails) │ ▼ ┌───────────────┐ │ Execute │ dispatch task via gateway / job / task / ticket │ │ agent investigates and acts on the target host └───────┬───────┘ │ ▼ ┌───────────────┐ │ Validate │ evaluate pattern_match / task_count / status rules │ │ run validate.yml → check actual machine state └───────┬───────┘ │ ▼ ┌───────────────┐ │ Restore │ run restore.yml → reset host to clean baseline └───────┬───────┘ (non-fatal if fails) │ ▼ ┌───────────────┐ │ Report │ print summary, persist ScenarioReport to DB └───────────────┘ ``` *** ## prepare.yml Runs **before** the agent executes. Use it to introduce the failure condition the scenario is benchmarking. Before `prepare.yml` runs, the runner silently executes `restore.yml` with errors suppressed. This clears leftover state from any previous failed run so each execution starts from a known baseline. If `prepare.yml` itself fails, execution is aborted and the scenario is marked as failed. ```yaml theme={null} --- - name: Deploy broken nginx configuration hosts: all become: true tasks: - name: Ensure nginx is installed apt: name: nginx state: present update_cache: true - name: Write a config with a syntax error (missing semicolon after listen 80) copy: dest: /etc/nginx/sites-available/default content: | server { listen 80 root /var/www/html; index index.html; } - name: Attempt reload: this will fail, which is intentional systemd: name: nginx state: restarted ignore_errors: true ``` *** ## restore.yml Runs **after** validation, during cleanup. Use it to reset the host to a clean baseline. Restore failures are **non-fatal**: the runner logs a warning and continues. Write restore playbooks defensively with `ignore_errors: true` on steps that may fail on an already-clean host. ```yaml theme={null} --- - name: Reset nginx to clean state hosts: all become: true tasks: - name: Stop nginx systemd: name: nginx state: stopped enabled: false ignore_errors: true - name: Remove injected config file: path: /etc/nginx/sites-available/default state: absent ignore_errors: true ``` *** ## validate.yml Runs **after** execution as part of the validation phase. Use it to verify that the agent's actions actually worked: service status, file contents, port availability, process list. Declared in `scenario.json` as an `ansible` validator: ```json theme={null} { "label": "Nginx is healthy", "validator": "ansible", "ansiblePath": "validate.yml" } ``` A non-zero exit code fails the resolution gate and marks the scenario as failed. ```yaml theme={null} --- - name: Verify nginx is healthy hosts: all become: true tasks: - name: Config syntax is valid command: nginx -t - name: Service is active command: systemctl is-active nginx - name: Port 80 is responding uri: url: http://localhost:80 status_code: [200, 301, 302] ``` *** ## Using Environment Variables Variables from the env file are available in all playbooks via `lookup('env', ...)`: ```yaml theme={null} - name: Clone a private repository git: repo: "https://{{ lookup('env', 'GH_TOKEN') }}@github.com/your-org/fixtures.git" dest: /opt/fixtures ``` *** ## inventory.ini The runner needs an Ansible inventory to know which hosts to target and how to reach them. **Host mode**: place an `inventory.ini` in the scenario directory. The runner detects it automatically and passes it to every `ansible-playbook` call. **VM mode (incus/lima)**: the runner generates the inventory automatically from the provisioned VM's IP, port, and SSH key. You do not need an `inventory.ini`. ```ini theme={null} [web] sandbox-web-01 ansible_host=10.0.1.10 ansible_user=ubuntu ansible_port=22 [all:vars] ansible_ssh_private_key_file=/etc/2501/keys/sandbox.pem ``` The hostnames must match the `host_name` values used in `scenario.json`. If no `inventory.ini` is present in host mode, `ansible-playbook` runs without an explicit inventory. Your playbooks won't be able to reach any hosts. Always include `inventory.ini` when your scenario has playbooks. # Quickstart Source: https://docs.2501.ai/0.12/benchmark/quickstart Run your first benchmark scenario end to end Goal: run a single nginx-broken-config scenario against an agent, see the result, and read the report. \~5 minutes once the prerequisites are in place. ## Prerequisites | You need | How | | -------------------------------------------------------- | -------------------------------------------------- | | A running 2501 instance with at least one host + agent | See [Quickstart](/0.12/getting-started/quickstart) | | `2501` CLI installed and signed in (`2501 status` works) | See [CLI](/0.12/cli/overview) | | Ansible on PATH | `brew install ansible` / `apt install ansible` | | A sandbox host the agent can SSH into | A spare VM, container, or Lima/Incus instance | If you'd rather have the runner provision an ephemeral VM for you, jump to [VM Sandbox](/0.12/benchmark/sandbox) instead — the rest of this page assumes `--mode host` with a pre-provisioned target. ## Step 1 — get a scenarios directory The runner reads scenarios from a directory. The fastest path is to clone the 2501 scenarios examples repo (your account team can share the URL), or write your own. ```bash theme={null} mkdir -p ./scenarios/nginx/001-broken-config cd ./scenarios/nginx/001-broken-config ``` ## Step 2 — write `scenario.json` ```json scenario.json theme={null} { "key": "nginx/001-broken-config", "title": "Nginx fails to start due to broken config", "description": "Nginx config has a syntax error. Agent must find it, fix it, and bring the service back up on port 80.", "ticket_title": "Nginx not responding", "ticket_body": "@2501 the nginx service on sandbox-web-01 is down. It was working yesterday but stopped after a config change. Investigate, fix the config, ensure nginx is running and serving traffic on port 80.", "tags": ["nginx", "web"], "hosts": [ { "host_id": "hst_REPLACE_WITH_YOUR_HOST_ID" } ], "agents": [ { "agent_id": "agt_REPLACE_WITH_YOUR_AGENT_ID", "host_id": "hst_REPLACE_WITH_YOUR_HOST_ID" } ], "validation": { "job": [ { "label": "Job resolved successfully", "validator": "job_resolution_status", "pattern": "success" } ], "tasks": [ { "label": "Agent tested config before restarting", "validator": "pattern_match", "pattern": "nginx -t", "where": "executed_commands" }, { "label": "Agent restarted nginx", "validator": "pattern_match", "pattern": "systemctl.*(restart|reload|start).*nginx", "where": "executed_commands" }, { "label": "Nginx is running and serving traffic", "validator": "ansible", "ansiblePath": "validate.yml" } ] } } ``` Replace the two IDs with real ones from your tenant (find them in Command Center → Hosts and → Agents). ## Step 3 — write the playbooks The runner needs three Ansible playbooks: `prepare.yml` introduces the failure, `validate.yml` checks the fix worked, `restore.yml` resets the host. ```yaml prepare.yml theme={null} - name: Break nginx config hosts: web become: true tasks: - name: Ensure nginx is installed apt: { name: nginx, state: present, update_cache: true } - name: Write a config with a missing semicolon copy: dest: /etc/nginx/sites-available/default content: | server { listen 80 root /var/www/html; index index.html; } - name: Attempt reload (will fail, intentional) systemd: { name: nginx, state: restarted } ignore_errors: true ``` ```yaml validate.yml theme={null} - name: Verify nginx is healthy hosts: web become: true tasks: - name: Config syntax is valid command: nginx -t - name: Service is active command: systemctl is-active nginx - name: Port 80 responds uri: { url: http://localhost:80, status_code: [200, 301, 302] } ``` ```yaml restore.yml theme={null} - name: Reset nginx hosts: web become: true tasks: - name: Stop nginx systemd: { name: nginx, state: stopped, enabled: false } ignore_errors: true - name: Remove broken config file: { path: /etc/nginx/sites-available/default, state: absent } ignore_errors: true ``` Add an `inventory.ini` so Ansible knows how to reach the host: ```ini inventory.ini theme={null} [web] sandbox-web-01 ansible_host=10.0.1.10 ansible_user=ubuntu ansible_ssh_private_key_file=~/.ssh/sandbox.pem ``` ## Step 4 — run it From the repo root that contains `./scenarios/`: ```bash theme={null} 2501 runner start -s nginx/001-broken-config ``` The runner walks through Provision → Prepare → Execute → Validate → Restore, prints a summary table, and writes a ScenarioReport to the database. ## Step 5 — read the report Two places: * **Terminal output** — pass/fail per rule, plus the two aggregate scores. * **Command Center → Benchmarks** — the same run, charted over time alongside others. Pass an iteration count if you want more confidence than a single run gives you: ```bash theme={null} 2501 runner start -s nginx/001-broken-config -i 10 ``` 10 runs is usually enough to surface flakes and gauge consistency. For full confidence, run **100+ iterations** as part of a CI job. ## What to read next Every field of `scenario.json`, with examples. Every validator, every `where:` target, the scoring model. What prepare / validate / restore each do, and when they run. Skip the manual host setup — let the runner provision a fresh VM per run. # Overview Source: https://docs.2501.ai/0.12/benchmark/sandbox Isolated, ephemeral VMs for scenario runs and agent testing Most deployments use `--mode host` with pre-provisioned VMs. VM sandbox mode is for teams that want the runner to provision and destroy VMs automatically, with no manual VM management and no lingering state between runs. ## Why VM Mode In `--mode host`, the runner connects to existing hosts already registered in your 2501 deployment. The host persists between runs and must be manually reset. In `--mode incus` or `--mode lima`, the runner creates a fresh VM clone for each run and destroys it afterward. Every scenario starts from an identical, known-good state. This is useful for: * **Reproducible benchmarks**: no leftover state from previous runs that can affect results * **Parallel runs**: multiple VMs can run simultaneously without interfering (`--parallel`) * **Playbook development**: iterate on scenarios without having to manually reset a shared host *** ## Prerequisites ### Both modes **Ansible** must be installed and available in `PATH`: ```bash theme={null} # macOS brew install ansible # Debian/Ubuntu apt-get install ansible ``` ### `--mode lima` (macOS) Lima runs VMs using Apple's Virtualization framework. Networking requires `socket_vmnet`. ```bash theme={null} # 1. Install Lima and socket_vmnet brew install lima brew install socket_vmnet # 2. Copy socket_vmnet to the expected system path sudo mkdir -p /opt/socket_vmnet/bin sudo cp /opt/homebrew/opt/socket_vmnet/bin/socket_vmnet /opt/socket_vmnet/bin/socket_vmnet sudo chown root:wheel /opt/socket_vmnet/bin/socket_vmnet # 3. Install Lima sudoers config (allows Lima to manage networking without sudo prompts) limactl sudoers | sudo tee /private/etc/sudoers.d/lima # 4. Create the Lima runtime directory sudo mkdir -p /private/var/run/lima sudo chown root:daemon /private/var/run/lima sudo chmod 775 /private/var/run/lima ``` ### `--mode incus` (Linux) Incus manages system containers and VMs. Install it following the official guide for your distribution, then initialize: ```bash theme={null} # See: https://linuxcontainers.org/incus/docs/main/installing/ # After installation, initialize Incus (run once) incus admin init ``` *** ## How Templates Work VM modes use **templates**: stopped VM instances pre-provisioned with the required software (Docker, k3s, etc.). When a scenario runs, the runner clones the template, starts the clone, runs the scenario, and destroys the clone afterward. The template itself is never modified. Templates are created automatically the first time a scenario references them. See [Templates](/0.12/benchmark/sandbox-templates) for the full list and management commands. *** ## What's in This Section * [Templates](/0.12/benchmark/sandbox-templates): available templates, resources, and how to manage them * [VM from Scenario](/0.12/benchmark/sandbox-scenario): step into a scenario environment to inspect state and iterate on playbooks * [VM for Testing](/0.12/benchmark/sandbox-targets): provision standalone VMs for ad-hoc agent testing # From a Scenario Source: https://docs.2501.ai/0.12/benchmark/sandbox-scenario Step into a broken scenario environment to inspect state and iterate on playbooks `sandbox prepare` sets up the full scenario environment (provisions VMs and runs `prepare.yml` to introduce the failure condition, and registers hosts and agents in the DB) but stops before dispatching the task to any agent. This lets you SSH in, inspect the state, and experiment freely. When you're done, `sandbox restore` tears everything down. *** ## Typical Workflow ```bash theme={null} # 1. Prepare the environment 2501 runner sandbox prepare -s nginx/001-broken-config -m lima # 2. SSH in using the printed credentials ssh -i ~/.ssh/runner_key debian@10.0.0.5 -p 22 # 3. Inspect the broken state, test fixes, iterate nginx -t cat /etc/nginx/sites-available/default # 4. Tear down when done 2501 runner sandbox restore -s nginx/001-broken-config -m lima ``` After `prepare`, the runner prints everything you need: ``` [nginx/001-broken-config] Environment ready SSH: ssh -i ~/.ssh/runner_key debian@10.0.0.5 -p 22 Agents: agt_... Hosts: hst_... Restore: 2501 runner sandbox restore -s nginx/001-broken-config -m lima ``` *** ## `sandbox prepare` ```bash theme={null} 2501 runner sandbox prepare -s [options] ``` | Option | Default | Description | | ----------------------------- | ------------- | ------------------------------------------------------------ | | `-s, --scenarios ` | required | Scenarios to prepare: comma-separated tags or explicit keys. | | `-m, --mode ` | `host` | Execution mode: `host` \| `incus` \| `lima`. | | `-p, --scenarios-path ` | auto-detected | Path to scenarios directory. | | `--vm-templates-path ` | auto-detected | Path to VM templates directory. | | `--specialty ` | | Override specialty for all agents. | | `--env-file ` | auto-detected | Path to env file. | If `prepare.yml` exits non-zero, the runner prints a warning but still finishes setting up the environment and reports it as ready, so you can SSH in and investigate why the playbook failed. *** ## `sandbox restore` Tears down environments created with `sandbox prepare`: destroys VMs (incus/lima), runs `restore.yml` (host mode), and removes registered hosts and agents from the DB. ```bash theme={null} 2501 runner sandbox restore -s [-m ] ``` | Option | Default | Description | | ----------------------------- | ------------- | ---------------------------------------------------------------- | | `-s, --scenarios ` | required | Scenarios to restore: comma-separated tags or explicit keys. | | `-m, --mode ` | `host` | Execution mode used during prepare: `host` \| `incus` \| `lima`. | | `-p, --scenarios-path ` | auto-detected | Path to scenarios directory. | | `--env-file ` | auto-detected | Path to env file. | Always restore after preparing. If you interrupt a session without restoring, use `sandbox purge-vms` (see [Templates](/0.12/benchmark/sandbox-templates)) to clean up stale VMs, then manually remove orphaned host/agent records from the DB if needed. # Standalone VMs Source: https://docs.2501.ai/0.12/benchmark/sandbox-targets Provision standalone VMs for ad-hoc agent testing outside of scenario runs `sandbox create` provisions a standalone VM, not tied to any specific scenario, and registers it as a host and agent in the DB. Use it when you want a persistent environment for ad-hoc tasks, manual testing, or experimenting with agent behavior outside of a scenario run. *** ## `sandbox create` ```bash theme={null} 2501 runner sandbox create --template -m [options] ``` After creation, the runner prints SSH access details and the delete command: ``` Target ready: my-target SSH: ssh -i ~/.ssh/runner_key debian@10.0.0.7 -p 22 Agents: agt_... Hosts: hst_... Delete: 2501 runner sandbox delete -s my-target -m lima ``` | Option | Default | Description | | ---------------------------- | ------------------ | --------------------------------------------------------- | | `--template ` | required | VM template to boot (e.g. `debian-docker`, `debian-k3s`). | | `-m, --mode ` | required | VM provider: `incus` \| `lima`. | | `-n, --name ` | auto-generated | Name for the VM, host, and agent. | | `--specialty ` | `SERVICES_MANAGER` | Specialty key for the registered agent. | | `--vm-templates-path ` | auto-detected | Path to VM templates directory. | | `--env-file ` | auto-detected | Path to env file. | *** ## `sandbox delete` Deletes a standalone target created with `sandbox create`: destroys the VM and removes the host and agent from the DB. ```bash theme={null} 2501 runner sandbox delete -s -m ``` | Option | Description | | ------------------- | ------------------------------------------------------- | | `-s, --name ` | required. Name of the target (as given at create time). | | `-m, --mode ` | required. VM provider: `incus` \| `lima`. | | `--env-file ` | Path to env file (auto-detected if omitted). | Use `sandbox restore` (see [VM from Scenario](/0.12/benchmark/sandbox-scenario)) for environments created with `sandbox prepare`, not `sandbox delete`. The runner will error if you mix them up. # Templates Source: https://docs.2501.ai/0.12/benchmark/sandbox-templates VM base images available for incus and lima modes Templates are stopped VM instances that the runner clones from for each scenario run. Each template is a Debian 12 base image pre-provisioned with specific software. Cloning is fast (a few seconds); provisioning only happens once, when the template is first created. Template instances are named `template-{name}` in incus/lima (e.g. `template-debian-docker`). *** ## Available Templates | Template | vCPUs | RAM | Description | | ------------------- | ----- | ---- | ---------------------------------------------------------- | | `debian-base` | 1 | 1 GB | Debian 12 with basic tools (curl, wget, python3, vim, git) | | `debian-docker` | 1 | 2 GB | Debian 12 with Docker CE pre-installed | | `debian-podman` | 1 | 2 GB | Debian 12 with Podman pre-installed (rootful, cgroupfs) | | `debian-k3s` | 2 | 2 GB | Debian 12 with k3s (lightweight Kubernetes) pre-installed | | `debian-localstack` | 1 | 2 GB | Debian 12 with AWS CLI v2 and moto server on port 4566 | All templates use a 10 GB disk. *** ## Auto-Initialization Templates are created automatically the first time a scenario references them. When the runner starts and a referenced template doesn't exist as a stopped instance, it: 1. Boots a fresh VM from the Debian 12 cloud image 2. Runs the provisioning playbooks (installs Docker, k3s, etc.) 3. Stops and saves the VM as `template-{name}` This happens once per template per machine. Subsequent runs clone from the existing template and start in seconds. *** ## Listing Templates Check which templates have been initialized: ```bash theme={null} incus list template- ``` ```bash theme={null} limactl list | grep template- ``` *** ## Deleting a Template Delete a template to force re-initialization on the next run. Useful if a template becomes stale or you want to pick up a newer base image. ```bash theme={null} incus delete template-debian-docker ``` ```bash theme={null} limactl delete template-debian-docker ``` Deleting a template does not affect existing VM clones. The next run that references the deleted template will re-provision it from scratch, which takes a few minutes. *** ## Purging Stale Clones If a `run` or `sandbox prepare` session is interrupted, clone VMs may be left behind. Use `purge-vms` to clean them up. Template instances are never touched. ```bash theme={null} 2501 runner sandbox purge-vms [-m incus|lima] ``` If `-m` is omitted, both providers are tried. # Scenario Structure Source: https://docs.2501.ai/0.12/benchmark/scenario Scenario structure, keys, tags, and writing effective descriptions A scenario is a directory containing a `scenario.json` file and optional Ansible playbooks. All scenarios live under a shared root (auto-detected, or set with `-p`). ## Directory Layout ``` scenarios/ / / scenario.json # required prepare.yml # optional: Ansible setup before execution restore.yml # optional: Ansible teardown after execution validate.yml # optional: Ansible verification after execution inventory.ini # optional: Ansible inventory (inventory.yml also accepted) [support files] # config snippets, fixtures, test data, etc. ``` The directory names are conventions for organization: they have no special meaning to the runner. The runner walks the scenarios root recursively and indexes each scenario by the `key` field inside its `scenario.json` (the directory path is not used as the key). *** ## scenario.json The only required file. At minimum, five fields are needed: ```json theme={null} { "key": "nginx-broken-config", "title": "Nginx fails to start due to config syntax error", "description": "Nginx has a broken config and fails to start. Agent must find the syntax error, fix it, and ensure nginx is running on port 80.", "ticket_title": "Nginx Broken Configuration", "ticket_body": "The nginx web server is not running. Investigate why nginx is failing to start, fix the configuration issue, and ensure the service is running and serving traffic on port 80." } ``` ### Required Fields | Field | Description | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `key` | Unique identifier for the scenario. Used to select the scenario with `-s` and to key its report. Free-form; conventionally the leaf directory name (e.g. `nginx-broken-config`). | | `title` | Human-readable name shown in reports and the `validate` output. | | `description` | Internal note describing what the scenario tests. Not sent to the agent. | | `ticket_title` | Title of the ticket created for the run. | | `ticket_body` | The instruction sent to the agent. Write it as you would a real ticket: describe symptoms, not solutions. Supports the `{{schedule_time}}` placeholder, which the runner replaces with a timestamp five minutes in the future. | ### Optional Fields | Field | Description | | ------------ | ---------------------------------------------------------------------------------------------- | | `tags` | String array for grouping. Pass a tag to `-s` to run all matching scenarios (e.g. `-s nginx`). | | `hosts` | Target host definitions. See [Hosts](/0.12/benchmark/hosts). | | `agents` | Agent definitions. See [Agents](/0.12/benchmark/agents). | | `validation` | Validation rules. See [Validation](/0.12/benchmark/validation). | *** ## Writing Good Tickets The `ticket_body` is the exact instruction the agent receives. A few guidelines: * **Describe symptoms, not solutions.** "The nginx service is not running after a configuration change" is better than "Fix the nginx config file." * **Be concrete.** Reference the host name, service name, or observable behavior when you know it. * **Keep it realistic.** Write it as you would a real support ticket or runbook task. *** ## Scenario Keys and Tags The `key` is a free-form unique identifier defined inside `scenario.json`. The runner indexes every discovered scenario by this value, and you select a scenario by passing its key to `-s`. The key does not have to match the directory path; by convention it is the leaf directory name: ``` scenarios/nginx/nginx-broken-config/scenario.json └─ key: "nginx-broken-config" ``` Tags are free-form strings for grouping. A scenario can have multiple tags: ```json theme={null} { "key": "nginx-broken-config", "title": "Nginx fails to start due to config syntax error", "description": "...", "tags": ["nginx", "web", "config"] } ``` Run all scenarios with a given tag: ```bash theme={null} 2501 runner start -s nginx 2501 runner start -s web ``` When a `-s` token does not match a key exactly, it is treated as a tag. Tags and explicit keys can be mixed: ```bash theme={null} 2501 runner start -s nginx,disk-cleanup ``` # start Source: https://docs.2501.ai/0.12/benchmark/start CLI reference for 2501 runner start ```bash theme={null} 2501 runner start -s [options] ``` ## Execution Flow Every `2501 runner start` call goes through these phases: **Pre-flight**: validates the environment: configuration load, DB connectivity, org/tenant/user IDs, gateway connectivity, and Ansible availability. **Scenario Discovery**: scans the scenarios directory, loads all `scenario.json` files, resolves the `-s` argument against available keys and tags. **Per-Scenario Loop**: for each matched scenario (and each iteration when `-i` > 1): * **Provision**: resolve host and agent from the DB (or boot a VM clone in `incus`/`lima` mode) * **Prepare**: run `restore.yml` silently to clear stale state, then run `prepare.yml` (abort if it fails) * **Flush**: flush memory for the scenario's agents * **Execute**: create a ticket through the selected gateway, then poll until the job is found and completes * **Validate**: evaluate the validation rules declared in `scenario.json`, and run `validate.yml` if the scenario declares it * **Restore / Teardown**: in `host` mode run `restore.yml` to reset the host (non-fatal if it fails); in VM modes destroy the clone **Report**: prints a summary table and persists a `Benchmark` plus per-scenario `ScenarioReport` rows to the DB. The process exits `0` unless a pre-flight or infrastructure error occurs, or `--fail-on-error` is set and at least one scenario failed. *** ## Options | Option | Default | Description | | ------------------------------------- | ------------- | ------------------------------------------------------------------------------------------- | | `-s, --scenarios ` | required | Scenario keys or tags, comma-separated. Mix freely. | | `--source ` | `git` | Where scenarios are loaded from: `git` \| `fs` | | `-m, --mode ` | `host` | Execution mode: `host` \| `incus` \| `lima` | | `-g, --gateway ` | `runner` | Gateway used to submit tickets: `runner` \| `servicenow` | | `-i, --iter ` | `1` | Number of iterations per scenario | | `-p, --scenarios-path ` | auto-detected | Path to the scenarios root directory | | `--vm-templates-path ` | auto-detected | Path to the VM templates directory (`incus`/`lima` modes) | | `--env-file ` | auto-detected | Path to the env file (falls back to `process.env`) | | `--main-engine ` | | Override the main engine for all agents in this run | | `--secondary-engine ` | | Override the secondary engine for all agents in this run | | `--specialty ` | | Override the specialty for all agents in this run | | `--gateway-engine ` | | Override the tenant `llm_model` for all tickets submitted via the runner gateway | | `--gateway-multimodal-engine ` | | Override the tenant `multimodal_llm_model` for all tickets submitted via the runner gateway | | `--parallel` | | Run scenarios concurrently using available RAM. Requires `--mode incus` or `--mode lima`. | | `--parallel-ram-cap ` | | Cap the RAM budget for parallel execution in GB. Requires `--parallel`. | | `--fail-on-error` | | Exit with code 1 if any scenario fails | | `--log-file ` | | Mirror all output (ANSI-stripped) to a file | | `-v` / `-vv` | | `-v`: show failed checks. `-vv`: show all checks with full debug content. | *** ## Gateways The `-g, --gateway` flag controls how each scenario's ticket is submitted. Every scenario is dispatched by creating a ticket; the runner then waits for the resulting job to complete. **`--gateway runner`** (default): POSTs directly to the engine's runner gateway endpoint. Requires `ENGINE_API_URL` and `ENGINE_API_KEY`. ```bash theme={null} 2501 runner start -s nginx/001-broken-config --gateway runner ``` **`--gateway servicenow`**: creates an incident in ServiceNow, exercising the full integration (gateway to job to agent to ticket resolution). Requires the `SERVICENOW_*` env vars. ```bash theme={null} 2501 runner start -s nginx/001-broken-config --gateway servicenow ``` *** ## Scenario Source The `--source` flag controls where scenario files come from. **`--source git`** (default): clones the repository given by `GIT_RUNNER_REPOSITORY` (branch `GIT_REF`, default `main`) using `GIT_RUNNER_PAT`, then reads scenarios from the clone. With `--scenarios-path`, the path is resolved relative to the repo root. **`--source fs`**: reads scenarios from the local filesystem at `--scenarios-path` (or an auto-detected default). *** ## Environment Variables | Variable | Required | Description | | -------------------------------- | ---------------------- | -------------------------------------------------- | | `DATABASE_URL` | always | PostgreSQL connection string | | `ORG_ID` | always | Your organization ID | | `TENANT_ID` | always | Your tenant ID | | `USER_ID` | always | The user ID under which scenarios run | | `ENGINE_API_URL` | `--gateway runner` | Base URL of the 2501 engine API | | `ENGINE_API_KEY` | `--gateway runner` | API key for engine authentication | | `SERVICENOW_API_URL` | `--gateway servicenow` | ServiceNow instance URL | | `SERVICENOW_USERNAME` | `--gateway servicenow` | ServiceNow username | | `SERVICENOW_PASSWORD` | `--gateway servicenow` | ServiceNow password | | `SERVICENOW_ASSIGNMENT_GROUP_ID` | `--gateway servicenow` | Assignment group for created incidents | | `SERVICENOW_CALLER_ID` | `--gateway servicenow` | Caller ID for created incidents | | `GIT_RUNNER_REPOSITORY` | `--source git` | Git URL of the scenarios repository | | `GIT_RUNNER_PAT` | `--source git` | Personal access token used to clone the repository | | `GIT_REF` | `--source git` | Branch to clone (defaults to `main`) | *** ## Common Patterns ```bash theme={null} # Run a single scenario 2501 runner start -s nginx/001-broken-config # Run all scenarios with a tag 2501 runner start -s nginx # Mix tags and explicit keys 2501 runner start -s nginx,disk/001-cleanup # 5 iterations 2501 runner start -s nginx/001-broken-config -i 5 # Submit through ServiceNow instead of the runner gateway 2501 runner start -s nginx/001-broken-config --gateway servicenow # Run scenarios from the local filesystem with a log file 2501 runner start -s regression-suite --source fs -i 3 --log-file ./runner.log # Parallel VM runs 2501 runner start -s nginx --mode lima --parallel --parallel-ram-cap 16 ``` # validate Source: https://docs.2501.ai/0.12/benchmark/validate Lint scenario definitions or re-run validation rules against existing runs The `validate` command has two independent modes. One requires a flag: `--scenarios` or `--runs`. *** ## `validate --scenarios` Inspect and lint scenario definitions: checks JSON schema, field consistency, host/agent references, and validation rule structure, and verifies that referenced agent, host, and specialty IDs exist in the database (so it needs a database connection). Does not execute the scenario. Useful after editing `scenario.json` files to catch errors before a full run. ```bash theme={null} # Inspect all scenarios 2501 runner validate --scenarios # Inspect scenarios matching a tag or explicit key 2501 runner validate --scenarios nginx 2501 runner validate --scenarios nginx/001-broken-config ``` | Option | Description | | ----------------------------- | --------------------------------------------------------------------- | | `[filter]` | Optional. Scenario filter: tags or explicit key. Omit to inspect all. | | `-p, --scenarios-path ` | Path to scenarios directory (auto-detected if omitted). | *** ## `validate --runs` Re-runs the validation rules for a scenario against an existing job, task, or benchmark, without re-executing the scenario. Use this when iterating on validation rules and you don't want to wait for another full agent run. Scenario run results are also visible in the Benchmark page in Command Center. ```bash theme={null} 2501 runner validate --runs --job-id 2501 runner validate --runs --task-id 2501 runner validate --runs --benchmark-id ``` | Option | Description | | ---------------------- | --------------------------------------------------------------------------- | | `--job-id ` | Job to validate against. | | `--task-id ` | Task to validate against. | | `--benchmark-id ` | Validate all runs for a benchmark. | | `-s, --filter ` | Override the scenario key (auto-detected from the existing run if omitted). | | `--from ` | Validation source: `gateway` \| `job` \| `task` (default: `job`). | | `-g, --gateway ` | Gateway type (`servicenow`). Required when `--from gateway`. | | `--env-file ` | Path to env file (auto-detected if omitted). | | `-v, --verbose` | Show detailed output. | # Validation Source: https://docs.2501.ai/0.12/benchmark/validation Validation rules, validators, and scoring model Validation rules determine whether a scenario passed. They are declared in the `validation` object of `scenario.json` and evaluated after the agent finishes executing. *** ## Structure ```json theme={null} "validation": { "gateway": [...], "job": [...], "tasks": [...] } ``` Rules are organized into three scopes based on what they check: | Scope | When used | What it checks | | --------- | --------------------- | ------------------------------------------ | | `gateway` | `--from gateway` only | The ServiceNow ticket | | `job` | All entry points | The job record (status, plan, task count) | | `tasks` | All entry points | Per-task data (commands, summaries, plans) | For `tasks` rules, each rule is checked against every task. A rule passes if it matches on at least one task. *** ## Rule Structure Every rule shares a common set of fields: ```json theme={null} { "label": "Agent restarted nginx", "validator": "pattern_match", "pattern": "systemctl.*(restart|reload|start).*nginx", "where": "executed_commands", "required": true, "negate": false } ``` | Field | Default | Description | | ----------- | ------- | ---------------------------------------------------------------------------------------------------------- | | `label` | - | Required. Shown in the validation report. Make it descriptive. | | `validator` | - | Required. The type of check to run. See validators below. | | `required` | `true` | When `false`, the rule is informational: it contributes to the compliance score but does not block a pass. | | `negate` | `false` | Invert the result. The rule passes when the condition is NOT met. | *** ## Validators ### `pattern_match` Checks whether a regex pattern matches in a specific field of the job or task data. ```json theme={null} { "label": "Agent edited the nginx config", "validator": "pattern_match", "pattern": "/etc/nginx/", "where": "executed_commands" } ``` | Field | Description | | --------- | ------------------------------------------------- | | `pattern` | Regular expression. Matching is case-insensitive. | | `where` | The field to search. | **`where` targets:** | Target | Content | | ------------------- | ------------------------------------------------ | | `executed_commands` | All commands run by the agent, one per line | | `task_summary` | The agent's summary of what it did | | `task_description` | The task description as created | | `task_plan` | The agent's execution plan | | `agent_messages` | Full agent reasoning history | | `job_resolution` | The job's resolution summary | | `job_plan` | The job-level plan | | `gateway_messages` | Messages posted by the gateway bot on the ticket | | `gateway_summary` | The gateway's summary of ticket resolution | | `operational_rules` | Operational constraints from the agent's context | *** ### `job_resolution_status` Checks the job's final resolution status. ```json theme={null} { "label": "Job resolved successfully", "validator": "job_resolution_status", "pattern": "success" } ``` Allowed values: `success`, `agentic_failure`, `hard_failure`, `partial`, `no_action`. *** ### `ticket_status` Checks the status of the ServiceNow ticket. Only applicable with `--from gateway`. ```json theme={null} { "label": "Ticket was resolved", "validator": "ticket_status", "pattern": "resolved" } ``` *** ### `task_count` Verifies that the number of tasks created under the job falls within a range. ```json theme={null} { "label": "Resolved efficiently", "validator": "task_count", "min": 1, "max": 3 } ``` *** ### `ansible` Runs an Ansible playbook and treats its exit code as pass/fail. The most reliable way to assert actual machine state. ```json theme={null} { "label": "Nginx is running and serving traffic", "validator": "ansible", "ansiblePath": "validate.yml" } ``` A non-zero exit code fails the resolution gate and marks the scenario as failed. See [Playbooks](/0.12/benchmark/playbooks) for how to write `validate.yml`. *** ## Scoring Model **Compliance score**: percentage of all non-Ansible rules that passed (required + optional combined). Informational. Two gates determine the actual pass/fail result: **Compliance gate**: passes when every `required` non-Ansible rule passes. **Resolution gate**: if a `validate.yml` Ansible rule exists, passes when the playbook exits 0. If no `validate.yml` rule is defined, it passes when the compliance gate passes and the compliance score is at least 80%. **A scenario passes only when both gates pass.** # Dynamic Context Injection Source: https://docs.2501.ai/0.12/best-practices/dynamic-context How 2501 chooses what to put in an agent's context — and what stays out You shouldn't have to tell an agent "don't forget rule X, don't forget rule Y". 2501 figures out what's relevant when a ticket arrives and injects only that. This is what makes specialties stay polyvalent and rules stay focused: the **right** rules show up at the **right** time, and irrelevant ones stay out. ## What is dynamic vs static | Resource | Behavior | | ------------------------------- | -------------------------------------------------------------------------------------------- | | **Agent's host** | Static — set at agent creation; an agent can't change host mid-task | | **Specialty** | Static — set at agent creation; can't change mid-task | | **Which agent gets the ticket** | Dynamic — the gateway picks per-ticket based on host and intent | | **Operational Rules** | **Both** — tag-matched programmatically, plus dynamic top-ups from the gateway and the agent | | **Host Knowledge facts** | Always available when working on that host | | **Blacklist** | Runtime check, never injected. Each command is screened before execution. | ## How injection happens ``` Ticket arrives │ ▼ Gateway batch-reads hosts & agents → picks the relevant ones │ ▼ Tag-matched operational rules + host knowledge attached ◄── programmatic │ ▼ Gateway may inject additional rules it judges relevant ◄── dynamic │ ▼ Agent receives the task with all of the above │ ├──▶ Agent may pull additional rules / facts mid-task ◄── dynamic │ ▼ Tool calls go through the blacklist on each command ◄── programmatic ``` Three injection points; three layers of safety: 1. **Tag matching at dispatch** — fully deterministic. A rule with `os:linux` + `procedure:restart` lands on every Linux restart task. 2. **Gateway top-up** — the gateway can attach non-obvious rules it noticed in the ticket text. This is the LLM-judgment layer that catches what tags can't. 3. **In-task pull** — if the agent encounters something it isn't sure about, it can search for rules and facts mid-execution. ## Why this matters Without dynamic context injection, you'd be writing **giant specialties** that try to cover every edge case, or **massive operational rules** that fire on every task. Both bloat the agent's context and degrade decisions. With dynamic injection, you keep each rule **small and specific**, and trust the matching layer to surface it when it matters. ## How to think about it as an author * A specialty teaches the *style* — polyvalent. * An operational rule encodes *truth about your env* — small, specific, well-tagged. * The matching layer does the work of choosing which rules apply to a given ticket. * If a rule isn't reaching the tasks you expect, the **matching trace** on each task's detail page tells you which rules matched and which were skipped, and why. # Working with Knowledge Source: https://docs.2501.ai/0.12/best-practices/knowledge Feed your existing documentation into 2501 so agents inherit it automatically Most of what your team knows about your infrastructure already lives in a Confluence page, a runbook, a CSV inventory. Feeding that into 2501 lifts a lot of heavy work — operational rules, host mapping, the role of each machine — without writing them by hand. It is similar to telling a coding agent "respect the coding style already in place." Same idea, applied to infrastructure. ## What gets extracted Upload PDFs, DOCX, CSV, or Markdown to **Command Center → Knowledge**. The engine parses each document and creates: | Extracted into | What it looks like | | ------------------------ | --------------------------------------------------------------------------------------------------------------------------- | | **Operational Rules** | "When restarting a Docker container with MongoDB on Ubuntu, do X then Y." Tagged `os:linux`, `tech:docker`, `tech:mongodb`. | | **Host Knowledge facts** | "host db-01 runs Postgres 15 with replication to db-02." Attached to the host record. | | **Blacklist entries** | "Never run `vgremove` on storage nodes." | See [Knowledge configuration](/0.12/configure/knowledge) for file types, size limits, and the ingestion status pipeline. ## How knowledge reaches agents Two paths, both automatic: * **Programmatic** — host description and tags get matched against operational-rule tags. Relevant rules and host facts ship with the task at dispatch time. * **Dynamic** — the gateway and the agent can pull additional rules + facts on demand if the task's scope evolves. You don't tell the agent "remember the runbook". The runbook is already in its context when relevant. ## Managing extracted knowledge Auto-generated rules can be **edited or marked outdated** from the Knowledge UI. A few things to know: * **Re-upload replaces extraction.** Uploading a newer version of the same document replaces the old extracted rules and facts. Your manually-curated rules are untouched. * **Conflicts: prefer your own rule.** If extracted knowledge contradicts what you want, create your own operational rule and disable the extracted one. Manual rules outrank extracted ones in conflicts. * **Don't expect 100% extraction quality.** Review what was extracted after each upload, especially for the first few documents. ## What to upload first In order of payoff: 1. **Runbooks** — they translate directly into operational rules (procedures). 2. **Infrastructure inventory spreadsheets** — they translate into host knowledge facts. 3. **Internal "don't do this" guides** — they translate into constraints + blacklist entries. 4. **Architecture docs and network diagrams** — useful only if your tenant has a multimodal model configured (diagrams need vision). 5. **Tribal-knowledge pages** ("the old way we restart auth-service") — high payoff but review extraction carefully. # MCP & Plugins Source: https://docs.2501.ai/0.12/best-practices/mcp-plugins When to give agents extra tools — and when not to A **Plugin** (MCP) is an abstraction over a complex CLI or API. Instead of the agent shelling out to a command line, it calls tools described in natural language. The MCP server handles the underlying logic. See [Plugins](/0.12/configure/plugins) for configuration. This page covers **when** to add one. ## Pros and cons * Simplifies hard-to-use CLIs * Better tool definitions = fewer wrong arguments * Abstracts complex multi-step logic * Can expose capabilities the agent wouldn't normally have * Each plugin adds a list of tools to the agent's context * That cost compounds quickly — many plugins = bloated context * Better to scope plugins per-agent than enable everywhere ## When to add an MCP If you observe an agent **consistently struggling** with a specific tool or CLI, that's the signal. Symptoms: * Repeated errors on the same command * Wrong arguments, hallucinated flags * Trouble parsing the tool's output * Calling commands that don't exist A real example from the field: agents handled VMware `govc` poorly — wrong paths, wrong arguments, invented commands. After adding a **VMware MCP**, the agent could manage machines, snapshots, and datastores with no friction. Night-and-day difference. ## When NOT to add one * **Standard CLIs the agent already handles well** (`kubectl`, `docker`, `aws`, `psql`, basic shell) — adding an MCP just bloats context for no gain. * **One-off use cases.** If you only need a tool once a quarter, the context cost outweighs the benefit. * **Wrapping something already in a [specialty](/0.12/configure/specialties).** A specialty teaching `aws ec2` patterns is often enough. ## Scoping plugins A plugin can be **tenant-level** (available to every org's agents) or **org-level** (only that org's agents). Per-agent assignment lets you be even more selective — give VMware MCP only to the vSphere-handling agents, not the database ones. Rule of thumb: a plugin's cost is **paid by every agent it's attached to, on every task**. Be selective. # Scoping Agents Source: https://docs.2501.ai/0.12/best-practices/scoping-agents Why focused agents beat general-purpose ones — and how to scope them A general-purpose "do-anything infrastructure agent" is too broad. Such an agent will solve a disk-full ticket, but it might solve it by **increasing the disk size by 500%** — quick, technically correct, no problem for a while. That's not the answer you want. A specialized agent — properly scoped — knows the right way: logrotate first, compress, identify what's actually writing logs, propose extension only when appropriate. ## Scoping is security too For AIOps on critical infrastructure, it is **always worth** reviewing specialties, operational rules, and task histories regularly. Two hours spent tightening a specialty can save 50 hours of post-hoc cleanup the next time that ticket type recurs. ## Three knobs to scope with | Knob | What it controls | When to tune | | --------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------ | | [**Specialty**](/0.12/prompting/specialty) | How the agent thinks about a domain | Whenever the agent enters a new tech vertical | | [**Operational Rules**](/0.12/prompting/operational-rule) | Hard constraints + procedures specific to your env | When you discover a "we always do it this way" pattern | | [**Gateway prompt**](/0.12/prompting/gateway) | Which agent gets which ticket, and how tasks sequence | When tickets cross multiple hosts or need a fixed flow | ## Worked patterns ### Many disk-full tickets * **Specialty: Disk Manager (Linux)** — knows about logrotate, compress, mount points, when extension is safe vs not. * **Specialty: Disk Manager (Windows)** — separate from Linux to keep command syntax sharp. * **Operational Rule** — log rotation policy + maximum disk extension allowed in your env. Why split Linux and Windows: each gets the right commands, fewer "this command doesn't exist on Windows" moments. ### Reverse proxy changes (HAProxy, nginx, Traefik) * **Specialty: Reverse Proxy** — covers redirections, SSL, ACLs for technologies with similar config syntax. * As scope grows complex, **split**: * Specialty: SSL Certificate Manager * Specialty: ACL & Routing Manager Splitting at the first sign of bloat keeps each specialty tight and easier to maintain. ## Anti-patterns * **One specialty per ticket type.** Too narrow. Specialties should cover a *technology* or *domain*, not a single task. * **One specialty for everything in a cloud.** Too broad. Split by service family (storage, compute, networking, identity). * **Putting environment specifics in a specialty.** That's what Operational Rules are for. # chat Source: https://docs.2501.ai/0.12/cli/chat Talk to the AI Assistant from your terminal `2501 chat` is the terminal interface to the [AI Assistant](/0.12/understand/command-center#ai-assistant). Same tools, same approval flow as in Command Center — just in your shell. ```bash theme={null} 2501 chat ``` The CLI opens an interactive session. Type freely. The assistant looks up resources automatically and pauses for approval when it wants to **create, update, or delete** anything. ## When to use it * Quick spot-checks: "Show me failed tasks from the last hour on host db-01." * Drafting rules and blacklist entries from the terminal while doing other work. * Triage during an incident without leaving your shell. For the full tool surface and approval flow, see [Command Center → AI Assistant](/0.12/understand/command-center#ai-assistant). # infra Source: https://docs.2501.ai/0.12/cli/infra Deploy and operate 2501 on Docker Swarm or Kubernetes `2501 infra` manages the on-premise deployment: the stack itself, plus tenants, organizations, users, webhooks, and feature toggles. Most subcommands have short aliases (`t`, `o`, `u`, `ft`). For deployment fundamentals (env files, configuration, troubleshooting), see [Deploy](/0.12/deployment/overview). ## Lifecycle ```bash theme={null} # Workspace 2501 infra init # create /etc/2501 with env templates (run by the installer) 2501 infra config # show the effective configuration (defaults + overlay) 2501 infra config --defaults # show only the embedded defaults # Deploy / upgrade / inspect 2501 infra deploy # interactive: pick a version and deploy 2501 infra deploy --yes # CI mode: fail on missing values, no prompts 2501 infra deploy --target kubernetes --image-tag 2501 infra deploy --remove # tear the stack down 2501 infra deploy --restart [engine|command-center] 2501 infra deploy --history # last few deploys 2501 infra logs # tail engine + command-center 2501 infra status # alias: ps — service health ``` ## Tenant / org / user ```bash theme={null} 2501 infra tenant create --name "My Company" --llm-model "..." --multimodal-model "..." 2501 infra tenant list 2501 infra t create ... # `t` alias 2501 infra org create --name "Engineering" --tenant-id ten_xxx 2501 infra org list 2501 infra o ... 2501 infra user create --email you@company.com --role ADMIN --tenant-id ten_xxx --org-id org_xxx 2501 infra user list 2501 infra u ... ``` ## Webhooks (ServiceNow) ```bash theme={null} 2501 infra webhook create \ --name prod-incidents \ --source-type servicenow \ --event-type incident \ --gateway-id gtw_xxx 2501 infra webhook delete --id whk_xxx 2501 infra webhook ls ``` After creation, the CLI prints both the URL/secret and the **ready-to-paste ServiceNow Business Rule script**. See [Webhooks](/0.12/configure/webhooks). ## Feature toggles ```bash theme={null} 2501 infra feature list # show all toggles + state 2501 infra feature enable # alias: e 2501 infra feature disable # alias: d 2501 infra feature enable # interactive multi-select picker ``` Full list of toggle keys: [Feature Toggles Reference](/0.12/deployment/troubleshooting#feature-toggles-reference). ## Where the rest lives * **First-time install** — your account team's installer chains `init` automatically. See [Deploy → Overview](/0.12/deployment/overview). * **Docker Swarm setup** — [Docker Swarm](/0.12/deployment/docker-swarm). * **Kubernetes** — [Kubernetes](/0.12/deployment/kubernetes). * **LLM provider seeding** — [LLM Providers](/0.12/deployment/llm-providers). * **Troubleshooting** — [Troubleshooting](/0.12/deployment/troubleshooting). # login & status Source: https://docs.2501.ai/0.12/cli/login-status Sign in to your Command Center, confirm who you are ## `2501 login` Sign in to your Command Center instance. ```bash theme={null} 2501 login ``` The command prompts interactively for **email** and **password** — the same credentials you use to sign in to the Command Center web UI. The session is persisted to your local config so subsequent commands (`resources`, `chat`) re-use it without prompting. ## `2501 status` Confirm which instance the CLI is pointing at and who is signed in. ```bash theme={null} 2501 status ``` Prints the configured base URL, current user, organization, and CLI version. Use this any time you're about to run a `sync`, a `deploy`, or any other potentially-impactful command — confirm you're aimed at the right environment first. ## What login is **not** for | You want | Use | | -------------------------- | ------------------------------------ | | Sign in as a user | `2501 login` | | Create or manage users | [`2501 infra user`](/0.12/cli/infra) | | Reset a forgotten password | Command Center → Users page (admin) | # 2501 CLI Source: https://docs.2501.ai/0.12/cli/overview One binary, six command groups — manage the platform from your terminal or CI The `2501` CLI is how you manage 2501 from a terminal or a CI/CD pipeline. It is **one binary** built from `apps/cli-2501` and ships as `2501-linux`, `2501-macos`, etc. ## Command groups Sign in with email + password. `status` shows who you are and which instance you're pointed at. Configuration as code. `pull` to snapshot, `sync` to apply. On-premise deployment: `deploy`, `init`, `config`, `logs`, plus tenant/org/user/webhook/feature management. Run and validate [Benchmark](/0.12/benchmark/overview) scenarios. `start`, `validate`, `chaos`, `flush`, `sandbox`. Talk to the AI Assistant from the terminal. Current instance, signed-in user, version. ## Quick reference ```bash theme={null} # Auth 2501 login 2501 status # Configuration as code 2501 resources pull -d ./config 2501 resources sync -d ./config # On-prem lifecycle (uses aliases — t/o/u/ft are also accepted) 2501 infra deploy 2501 infra tenant create --name "My Company" 2501 infra org create --name "Engineering" --tenant-id ten_xxx 2501 infra user create --email you@company.com --role ADMIN ... 2501 infra webhook create --gateway-id gtw_xxx --source-type servicenow --event-type incident --name prod 2501 infra feature enable enableElasticSearch # Benchmarks (sandbox only) 2501 runner start -s nginx/001-broken-config 2501 runner validate --scenarios nginx ``` ## Two CLIs, one binary Before the merge: `2501-infra` for deployment, `2501-runner` for benchmarks, `2501` for resources. All three are now subcommands of one `2501` binary. If you have muscle memory for `2501-infra deploy`, that becomes `2501 infra deploy`. A hidden stub for `infra update` prints a redirect because that command was folded into `deploy`. ## Installing Your 2501 account team ships the installer. The installer places a checksum-verified static binary, sets up a sudo wrapper for privileged commands, and runs `2501 infra init` non-interactively so the workspace is ready when it finishes. To update the binary later, re-run the installer (idempotent — it never touches an existing workspace) or run an interactive `2501 infra deploy`: picking a version self-updates the binary first. # resources Source: https://docs.2501.ai/0.12/cli/resources Manage platform resources as version-controlled MDX files `2501 resources` syncs a directory of MDX files with your platform. Each kind has its own subdirectory; each file declares one resource with YAML frontmatter (and a body where the kind has long-form text). For the full reference, examples, and the resource file format, see [Configuration as Code](/0.12/configure/configuration-as-code). ## `2501 resources pull` Export the platform's current state into a local directory. ```bash theme={null} 2501 resources pull -d ./config ``` `pull` is a true mirror: existing `.mdx` files in each kind folder are removed before writing, so deletes on the platform also disappear from your repo. Commit the result for an exact snapshot. Credential **secret values are never exported** — only the metadata. ## `2501 resources sync` Apply the directory back to the platform. ```bash theme={null} 2501 resources sync -d ./config ``` `sync` prints a plan (create / update / delete / unchanged) and applies in dependency-safe order. Stops on the first failure; resources already applied stay applied — re-run to continue. ### Deletes are opt-in By default, resources on the platform that aren't in your directory are **reported, not deleted**. To prune them, pass `--prune`: ```bash theme={null} 2501 resources sync -d ./config --prune ``` A prune that would orphan dependents (a host with active tasks, a specialty referenced by agents) is flagged **BLOCKED** and skipped. To override, add `--force`: ```bash theme={null} 2501 resources sync -d ./config --prune --force ``` `--prune --force` removes resources even when they have dependents. Review the printed plan carefully before approving. ### CI usage ```bash theme={null} 2501 resources sync -d ./config --dry-run # print plan, don't apply 2501 resources sync -d ./config --auto-approve # skip the y/N prompt ``` ## Scope to one organization ```bash theme={null} 2501 resources pull -d ./config --org platform-team 2501 resources sync -d ./config --org platform-team ``` Always pair `pull --org` with `sync --org` so the two operations cover the same scope. ## Supported kinds | Kind | Subdirectory | Body | | ----------------- | -------------------- | ------------------------ | | Agents | `agents/` | — | | Hosts | `hosts/` | Host knowledge | | Specialties | `specialties/` | Specialty prompt | | Operational Rules | `operational_rules/` | Rule text | | Credentials | `credentials/` | — (secrets via `${ENV}`) | | Gateways | `gateways/` | Inbound prompt | | Blacklist | `blacklist/` | — | # runner Source: https://docs.2501.ai/0.12/cli/runner Drive Benchmark scenarios from the CLI `2501 runner` is the [Benchmark](/0.12/benchmark/overview) driver. It dispatches scenarios as if they were real tickets, scores the result against your validation rules, and writes a report to the database. Typically only used in **sandbox environments**. ## Subcommands ```bash theme={null} 2501 runner start # run scenarios 2501 runner validate # lint scenarios, or re-score an existing run 2501 runner flush # delete scenario run data 2501 runner chaos # resilience testing — kills the engine mid-task 2501 runner sandbox # VM lifecycle (lima / incus): prepare, restore, create, delete, purge-vms ``` ## `start` (was `run`) ```bash theme={null} 2501 runner start -s nginx/001-broken-config 2501 runner start -s nginx # all scenarios tagged nginx 2501 runner start -s nginx,disk -i 5 # 5 iterations across two tags 2501 runner start -s nginx --gateway servicenow # use a real ServiceNow instead of the runner gateway 2501 runner start -s nginx --mode lima --parallel --parallel-ram-cap 16 ``` | Common flag | Meaning | | ---------------------------------------------------- | --------------------------------------- | | `-s, --scenarios ` | Scenario keys or tags | | `-m, --mode ` | Pre-provisioned hosts vs ephemeral VMs | | `-g, --gateway ` | Where to submit the ticket | | `-i, --iter ` | Number of iterations per scenario | | `--main-engine`, `--secondary-engine`, `--specialty` | Per-run overrides | | `--parallel` | Concurrent runs (VM modes only) | | `--fail-on-error` | Exit non-zero if any scenario fails | | `--log-file ` | Mirror output (ANSI-stripped) to a file | For the full flag and env-var reference, see [Benchmark → start](/0.12/benchmark/start). ## `validate` ```bash theme={null} # Lint scenarios without running them 2501 runner validate --scenarios 2501 runner validate --scenarios nginx # Re-score an existing run without re-executing the scenario 2501 runner validate --runs --job-id 2501 runner validate --runs --benchmark-id ``` Use `validate --runs` while iterating on validation rules — much faster than re-running the agent. ## `flush` ```bash theme={null} 2501 runner flush --older-than 7d --preview 2501 runner flush --scenario nginx/001-broken-config 2501 runner flush --deprecated # delete records for scenario keys no longer on disk 2501 runner flush --all # nuclear; requires typing yes ``` Removes ScenarioReport rows plus their associated Benchmark / Job / Task / Ticket records. ## `chaos` Drives resilience testing: runs a scenario but kills the engine at random points during execution and verifies the system recovers. Used in CI to catch regressions in restart / resume behavior. ## `sandbox` VM management for `--mode lima` or `--mode incus`: ```bash theme={null} 2501 runner sandbox prepare -s nginx/001-broken-config -m lima # … SSH in, inspect, iterate … 2501 runner sandbox restore -s nginx/001-broken-config -m lima 2501 runner sandbox create --template debian-docker -m lima -n my-target 2501 runner sandbox delete -s my-target -m lima 2501 runner sandbox purge-vms -m lima # clean up stale clones ``` See [Benchmark → VM Sandbox](/0.12/benchmark/sandbox) for the full sandbox surface. # API Keys Source: https://docs.2501.ai/0.12/configure/api-keys Generate scoped keys to call the 2501 API programmatically An **API key** authenticates programmatic requests to the 2501 API without a user session. Generate one from **Settings** to script against your inventory - hosts and agents, over the [versioned API](/0.12/api/overview) - from outside Command Center. API Keys ## Creating a key Only **administrators** can create API keys. Open **Settings** → **API Keys** and click **New API Key**. * **Name** - a label to recognize the key later, e.g. `cmdb-sync`. * **Scope** - **This organization** limits every call made with the key to your current organization. Tenant-level administrators can instead choose **Entire tenant**, which lets the key act on any organization in the tenant. Org-level administrators can only mint org-scoped keys. * **Expiration** - 30 days, 90 days, or 1 year, or check **Never expires** for a key that has no expiry at all. A date you set by hand has to be in the future and within 10 years. The raw key is shown **exactly once**, immediately after creation. Copy it before closing the dialog - 2501 stores only its hash and cannot show it to you again. If you lose it, revoke it and create a new one. ## Using a key Send it as a bearer token on the versioned API: ```bash theme={null} # Any authenticated call returning 200 confirms the key works. curl "https:///api/v1/hosts?org_id=" \ -H "Authorization: Bearer " ``` [API Overview](/0.12/api/overview) documents the endpoints a key can reach, and the conventions they share. A request with a valid key is treated as an administrator within the key's scope - full read/write on that organization, or the whole tenant for a tenant-scoped key. This reach is fixed at creation time: it does not change if the user who created the key is later demoted, removed, or loses org access. Only revoking or expiring the key changes what it can do. An API key cannot be used to create, list, or revoke other API keys - key management stays session-only, so a leaked key can't mint itself a successor. ## Revoking a key Any administrator can revoke any key in the tenant from the **API Keys** list, regardless of who created it. Revoking is immediate and permanent - there is no un-revoke, only creating a new key. An expired key stops working on its own once its expiration date passes; you don't need to revoke it separately. # Blacklist Source: https://docs.2501.ai/0.12/configure/blacklist Prevent agents from executing specific commands The Blacklist feature blocks agents from executing certain commands during task execution. This is useful for preventing problematic operations like calling unstable endpoints, using uninstalled tools, or running destructive commands. When an agent tries to execute a blacklisted command, 2501 checks it against your defined patterns and rejects it if there's a match, prompting the agent to find another approach. ## Managing Blacklists Go to **Command Center** → **Blacklist** and click **Create Entry** to add a new entry. Blacklist ### Pattern The string or glob pattern that defines which commands to block. Patterns match commands that **contain** the text anywhere, not just exact matches. Glob wildcards are also supported: * `*` matches any sequence of characters * `?` matches a single character Example: `rm -rf *` blocks any `rm -rf` command regardless of the target path. `vim` blocks any command containing the word `vim`. The Command Center shows a **live preview** of matching and non-matching commands as you type, so you can verify a pattern before saving it. A collapsible **syntax reference** is available inline in the create and edit dialogs. ### Description Explains why the command is blocked. Example: `Vim is interactive and can't be operated by LLMs` ### Organization By default, a blacklist is scoped to your current organization and applies only to that org's agents. Toggle the scope checkbox to **Scoped to Tenant** to make the blacklist available to all organizations in the tenant. ## Common Use Cases **Destructive Operations** Block high-risk commands for additional safety: * `rm -rf /` * `awscli terminate-instances` * `sudo shutdown` **Interactive Tools** Agents can't interact with prompts or shells: * `redis-cli` * `vim`, `nano`, `vi` * `mysql`, `psql` * `python` **Missing or Unstable Tools** Block commands for tools that aren't installed, have known issues, or are deprecated. ## Best Practices Document why each command is blocked. Review your blacklist regularly as your infrastructure changes. For enforcing preferred alternatives instead of just blocking, see [Operational Rules](/0.12/configure/operational-rules). # Configuration as Code Source: https://docs.2501.ai/0.12/configure/configuration-as-code Manage platform resources as version-controlled MDX with the 2501 CLI The `2501` CLI syncs a directory of MDX files with your platform. Reviewable diffs, pull requests, reproducible source of truth. For the per-resource frontmatter schema, see the **Resource Reference** pages below. ## Authentication ```bash theme={null} 2501 login # email + password, interactive 2501 status # confirm which instance + user ``` ## How it works Each resource is one MDX file. YAML frontmatter holds the fields; the body (when the kind has one) holds long-form text — the specialty prompt, the rule text, the host knowledge, the gateway's inbound prompt. | Kind | Subdirectory | Body | Reference | | ---------------- | -------------------- | ------------------------ | --------------------------------------------------------------- | | Agent | `agents/` | — | [Agent](/0.12/configure/resources/agents) | | Host | `hosts/` | Host knowledge | [Host](/0.12/configure/resources/hosts) | | Specialty | `specialties/` | Specialty prompt | [Specialty](/0.12/configure/resources/specialties) | | Operational Rule | `operational_rules/` | Rule text | [Operational Rule](/0.12/configure/resources/operational-rules) | | Credential | `credentials/` | — (secrets via `${ENV}`) | [Credential](/0.12/configure/resources/credentials) | | Gateway | `gateways/` | Inbound prompt | [Gateway](/0.12/configure/resources/gateways) | | Blacklist | `blacklist/` | — | [Blacklist](/0.12/configure/resources/blacklist) | ## Workflow ```bash theme={null} # Snapshot the live platform into your repo 2501 resources pull -d ./config # Apply the repo back to the platform 2501 resources sync -d ./config ``` `pull` is a true mirror — existing `.mdx` files in each kind folder are removed before writing, so deletes on the platform also disappear from your repo. Credential secret values are **never** exported. `sync` diffs each resource, prints a plan (create / update / delete / unchanged), and applies in dependency-safe order. An agent can reference a host declared in the same directory, even before the host exists on the platform. Sync stops on the first failure; already-applied resources stay applied — re-run to continue. For CI, `--dry-run` prints the plan without applying and `--auto-approve` skips the y/N prompt. ## Deletes are opt-in By default, `sync` never deletes. Resources on the platform that aren't in your directory are **reported, not removed**. To prune them, pass `--prune`: ```bash theme={null} 2501 resources sync -d ./config --prune ``` A prune that would orphan dependents is flagged **BLOCKED** and skipped: * A **host** is blocked while it has active tasks or attached agents. * A **specialty** is blocked while agents or rules still reference it. * A **gateway** is blocked while webhooks or tickets still reference it (jobs never block). To override, add `--force`: ```bash theme={null} 2501 resources sync -d ./config --prune --force ``` `--prune --force` removes resources even when they have dependents. Review the printed plan carefully before approving. ## Scoping to one organization ```bash theme={null} 2501 resources pull -d ./config --org platform-team 2501 resources sync -d ./config --org platform-team ``` On `pull`, only that org's resources are exported. On `sync`, files declaring a different org are rejected. Always pair `pull --org` with `sync --org` so the two operations cover the same scope. ## Keeping secrets out of git Credential `value` fields reference environment variables with `${ENV_VAR}`: ```mdx theme={null} value: ${PROD_DB_PASSWORD} ``` At sync time, the placeholder is resolved from the shell or CI secret store. See [Credential](/0.12/configure/resources/credentials) for the full credential file format. ## Tag validation Host and operational-rule tags are validated against the platform's [tag vocabulary](/0.12/configure/operational-rules#tag-axes) at plan time. **Only tags introduced by a create or update are checked** — resources already carrying an out-of-vocab tag on the platform round-trip untouched, so adopting Configuration as Code never forces a tag cleanup first. * `app:` open-namespace tags are accepted on both hosts and rules. * `procedure:*` tags are valid on operational rules only — never on hosts. # Credentials Source: https://docs.2501.ai/0.12/configure/credentials Enable agents to access your services Credentials store secret keys, tokens, and authentication data for your services. All credentials are encrypted at rest, and agents can only access them programmatically during task execution (they never see the actual values). Credentials list ## Managing Credentials Go to **Command Center** → **Credentials** and click **New Credentials** to create an entry. Credentials ### Name A descriptive identifier for the credential. Use naming conventions that show its purpose and target system. Example: `prod_db_ssh_root` or `aws_prod_api_key` ### Description Additional context about what this credential is for. Example: `Root SSH credentials for production Ubuntu database server` ### Scope By default, credentials are scoped to the current organization, so only that org's agents can use them. Switch the scope to **Tenant** to make a credential available to all organizations. ### Type * **Value**: Store the credential directly in 2501's encrypted storage * **Vault Path**: Reference a secret stored in an external vault like HashiCorp Vault When using vault paths, make sure the vault is accessible from where your agents run. ### Value The actual credential data or vault path reference. All values are encrypted and only decrypted when needed. ⚠️ **Important:** Escape special characters properly to prevent authentication errors. ### Agent Accessible Controls whether agents can use this credential during tasks. **Enable for:** * SSH configurations for remote execution * API keys for CLI tools * Database credentials for queries * Any credential the agent needs to pass to commands **Disable for:** * Service credentials used only by 2501 infrastructure * MCP server authentication tokens * Backend service bearer tokens ## Assigning Credentials to Agents When creating or editing an [agent](/0.12/core-concepts/agents), find the credentials section to assign what's needed. Assign Creds For each credential, configure: * **Role**: How the credential will be used (see below) * **Priority**: Order of precedence when multiple credentials of the same role exist * **Required**: Whether the agent can work without it Only credentials marked "Agent Accessible" will show up here. ## Credential Roles Roles define how agents use credentials during execution. They cover remote access over SSH: * **SSH Username**: Remote system login name * **SSH Password**: Password-based SSH authentication * **SSH Private Key**: Private key for key-based authentication * **SSH Public Key**: Public key (rarely needed) ## Common Use Cases **SSH Remote Execution (key-based)**\ Roles: SSH Username, SSH Private Key\ Agent Accessible: Yes\ Required: Yes **SSH Remote Execution (password-based)**\ Roles: SSH Username, SSH Password\ Agent Accessible: Yes\ Required: Yes ## Credential Priority When multiple credentials with the same role are assigned to an agent, priority determines which gets tried first. Lower numbers = higher priority. Useful for: * Failover scenarios (try primary, fall back to secondary) * Multi-environment access (different credentials for different systems) * Credential rotation (keep old credentials briefly while transitioning) ## Windows Authentication with gMSA Windows (WinRM) hosts can authenticate using a gMSA (group Managed Service Account) over Kerberos instead of a static Windows username and password. This avoids storing a long-lived Windows admin password. Manage these under **Command Center** → **Credentials** → **gMSA Configurations**. Administrators can create, edit, and delete configurations. Before adding a gMSA configuration, create a credential holding the password of the Active Directory service account that 2501 binds as. The configuration references that credential and captures the AD connection details: * The **gMSA account** (its `sAMAccountName`, ending in `$`) and the AD **realm** * The **Domain Controller** host and **LDAPS port** (default `636`) * The **LDAP search base** and the **bind DN** of the service account * Optional **KDC endpoints**, and for domain controllers that use a private certificate authority, a **PEM CA certificate** for the LDAPS connection One configuration can serve multiple Windows hosts that share the same gMSA, and a configuration that is still attached to hosts cannot be deleted. When a host uses gMSA authentication, 2501 reads the managed password from Active Directory over LDAPS and obtains a Kerberos ticket to authenticate to WinRM, bypassing the static password path. ## Security Best Practices Only mark credentials as "Agent Accessible" when necessary. Use organization-level scoping to limit exposure. Rotate credentials regularly and after any suspected compromise. For production systems, prefer vault paths over direct values. Review which agents have access to sensitive credentials, and always document the credential's purpose clearly. # Network Discovery Source: https://docs.2501.ai/0.12/configure/discovery Scan your network to build host inventory automatically Network Discovery lets 2501 scan one or more subnets and populate your host inventory without manual entry. Instead of adding hosts one at a time, you point a scan at a range of addresses and 2501 finds what's live, identifies it, and — with your approval — promotes it to a managed host. Start a scan from the **Discovery** page in Command Center. Every scan runs through an **entry host** — the host the engine connects through to reach the target subnets (see [Reaching non-routable subnets](#reaching-non-routable-subnets)). To let a deep scan identify what it finds, store credentials for the hosts you expect to discover first — see [Credentials](/0.12/configure/credentials). ## Starting a scan Go to **Command Center** → **Discovery** and start a new scan. A scan takes these inputs: * **Entry host** *(required)* — every scan starts from an entry host; the engine reaches the target subnets through it. To tunnel through it as a bastion, give its SSH user and key/secret credentials; leave both empty to run the scanner directly from the engine host. If the host has both a public and private IP, you also choose which one to dial. * **Scope** *(required)* — one or more CIDRs to cover (e.g. `10.0.1.0/24`, or a comma-separated list). A CIDR must be `/16` or narrower; broader ranges are rejected to keep a single scan bounded. * **Credential allowlist** *(optional)* — restricts which stored credentials a deep scan may try against discovered hosts. Leave it empty and all of the org's credentials are eligible; narrow it to specific credentials when you want tight control over which logins are ever attempted, such as on sensitive or fragile network segments. ### Scan depth A reachability sweep only — 2501 checks which addresses in the range are live and which ports they have open. Nothing is logged into. Everything in **Scan**, plus authenticated recon: 2501 attempts to log into each responding host with the eligible stored credentials and characterizes it (OS, services, routes, and technology stack). Deep scan is what populates a host's tags. ### Review mode Choose how much of the scan runs without you in the loop: * **Auto** — recon starts automatically and every host that responds is promoted to a managed host. No manual step. * **Semi-auto** — recon starts automatically, but you confirm each host before it's created or updated. * **Manual** — nothing fires on its own. You trigger recon and host creation yourself, node by node. ## How it works The scanner runs inside a short-lived container, one per subnet. It probes each address in the range, then — on a deep scan — attempts to authenticate to responding hosts using the eligible stored credentials. Hosts that accept a connection are characterized automatically — OS, services, routes, and technology stack are derived and attached as [tags](/0.12/core-concepts/hosts#host-tags). Discovered nodes appear in the infrastructure map as a review panel. For each node you can: * **Confirm** — promote it to a managed host with all derived tags applied, making it immediately available for agent assignment. * **Ignore** — dismiss the node without adding it to inventory. ## Credentials and characterization On a deep scan, 2501 tries the eligible stored credentials against each responding host to identify it. Credential handling is deliberately conservative: * 2501 only ever tries credentials already stored in the org — it never generates or guesses logins. The credential allowlist narrows this further; an empty allowlist makes every org credential eligible. * A node is marked characterized only when a command actually proves access — not on a hopeful match. * To protect accounts, 2501 stops using a credential after a configurable number of failed authentication attempts (**max failures per credential**). * A host that responds but that no credential can authenticate to is still recorded as discovered, so you can supply a credential and re-run recon on it later. ## Coverage A scan reports honest coverage per CIDR. Every subnet in scope shows what happened — swept directly, reached through the entry host, or unreachable. An unreachable subnet is recorded as such rather than silently dropped, so you always know exactly what a scan did and did not cover. ## Idempotency Scans are idempotent. Re-scanning a subnet links newly found IPs to any hosts already in inventory rather than creating duplicates. Running a scan regularly is safe: you see new nodes without losing existing host records. A node you previously ignored can resurface on a later scan if it's still live — ignoring is a decision about one scan, not a permanent suppression. ## Reaching non-routable subnets When the target subnets aren't reachable from the engine directly, pick an entry host that sits inside or adjacent to them and give it SSH credentials. The scanner then tunnels through that host to reach the target CIDRs — the entry host acts as the bastion. See [Hosts — Jump host](/0.12/core-concepts/hosts#jump-host-ssh-bastion) for how bastion hosts are registered. When a node reached through an entry host is promoted, the path used to reach it is carried onto the new host's configuration, so the agent can connect to it the same way the scan did. ## After confirmation Confirmed hosts are immediately available for agent assignment. Tags applied during discovery (OS, shell, type, technologies) are editable from the host's detail page if the characterization needs adjustment. For bulk onboarding from a file, see [Import and Export](/0.12/configure/import-export). To manage hosts declaratively, see [Configuration as Code](/0.12/configure/configuration-as-code). # Import and Export Source: https://docs.2501.ai/0.12/configure/import-export Bulk import and export hosts and agents as CSV or JSON Move a fleet in and out of 2501 as a file. Export what you have, edit it in a spreadsheet, import it back. Useful for onboarding an inventory you already keep somewhere else, for scripted automation, and for copying a setup between environments. Hosts and agents each have their own file. Import order for a new fleet is **hosts, then agents** - an agent file points at hosts and cannot create them, so export your hosts after importing them to get the ids the agent file needs. ## The files One format per entity, in CSV or JSON. The columns are the same either way, and they are the fields of the API - so a file doubles as documentation of what you can set. **A file belongs to one organization.** The organization comes from the request, and everything a row points at is an id, which only means something inside that organization. ### hosts | column | notes | | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | `id` | present on every export. On import, filled means "update this host", empty means "create one" | | `name` | required to create a host; identifies it within the organization, and is how a later row in the same file can point at it as a jump host | | `public_ip`, `private_ip`, `ip_connect_mode` | `auto` (default), `public`, or `private` | | `target_type`, `target_port`, `skip_tls_verify` | `ssh` (default) or `winrm`; the TLS flag applies to WinRM only | | `additional_names` | list | | `tags` | list, from the closed tag vocabulary | | `knowledge` | free text | | `jump_hosts` | ordered list of hosts to route through, engine side first. Host ids, or the name of a row above this one | | `is_jump_host` | `false` by default | | `jump_host_username_credential` | credential id holding the relay login user; required when `is_jump_host` is true | | `jump_host_secret_credential` | credential id holding the relay password or key; required when `is_jump_host` is true | | `eligible_subnets` | list of IPv4 CIDRs; jump hosts only | ### agents | column | notes | | ------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `id` | present on every export. On import, filled means "update this agent", empty means "create one" | | `name` | required on every import row - part of what identifies the agent | | `specialty` | specialty id, required to create an agent | | `host` | host id, required to create an agent, and cannot be changed afterwards | | `main_engine`, `secondary_engine` | model keys; both or neither | | `remote_execution` | `true` by default | | `ssh_winrm_username`, `ssh_winrm_password`, `ssh_private_key` | CSV only - one column per login credential, each holding a credential id. The username and password columns serve an SSH or a WinRM login alike; keys are SSH-only, since Windows has no key auth | | `accessible_credentials` | list of credential ids the agent may reference in placeholders | In JSON, an agent's credentials are one `credential_config` array instead of the three columns. Use JSON when you need something the columns cannot express: an optional credential, two credentials on one role, a hand-tuned order, or a public key (which has no column, because it is stored but never used to build a connection). Ids come from an export - a host's is also in the URL of its page in Command Center. Start a hand-written agent file from a hosts export, so the `host` ids are already right. ### Host tags are a closed list `tags` is the one column whose values are not free text. Operational rules and prompts target these tags, so a tag nobody defined **matches nothing** — you get silent no-ops rather than an error when you type it. Four closed namespaces, and one open one: | namespace | values | | --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `os:` | `linux`, `windows`, `aix`, `solaris`, `fortios`, `cisco-ios`, `junos`, `nx-os`, `esxi`, `ibm-i` — give the family and the distro where both apply | | `shell:` | `posix`, `non-posix`, `dsl-cli` (a vendor CLI with no shell: Cisco IOS, FortiOS, JunOS, NX-OS). One per host | | `type:` | `database`, `web`, `jump-host`, `backup`, `monitoring`, `identity`, `compute`, `object-storage`, `network-appliance`, `hypervisor`, `control-plane`, `generic-storage`, `iot`, `vault` | | `tech:` | `docker`, `kubernetes`, `postgres`, `mysql`, `redis`, `mongodb`, `elasticsearch`, `kafka`, `nginx`, `active-directory`, `ansible`, `terraform`, `tomcat`, `iis`, `weblogic`, `springboot`, `fastapi`, `kong`, `oracle`, `exadata`, `mssql`, `db2`, `ibmmq`, `cics`, `saa`, `hsm-luna`, `splunk`, `cyberark`, `f5`, `palo-alto`, `cisco-firepower`, `ecs-fargate`, `powermax`, `netapp`, `vsphere`, `veritas`, `jenkins`, `artifactory`, `aws`, `gcp`, `azure`, `proxmox`, `podman`, `rabbitmq`, `hashicorp-vault` | | `app:` | **open** — name your own business applications: `app:` followed by lowercase letters, digits, `.`, `_` or `-` | `tech:` values are deliberately unversioned: the specific version a host runs belongs in `knowledge`, so a rule tagged `tech:tomcat` matches every Tomcat host. Two behaviours worth knowing: * Only tags a row **introduces** are checked. A tag already stored on a host stays editable even if the vocabulary changes later, so one legacy tag cannot block every future edit of that host. * `type:jump-host` is **managed for you**, in lockstep with `is_jump_host`. Setting it by hand does nothing; removing it does not unflag the host. ### References Anywhere a file points at another entity - a credential, a specialty, an agent's host - write its **id**. Exports write ids, and an id survives a rename. There is one exception, so that a whole fleet fits in one file: in a hosts file, `jump_hosts` may hold the **name of a row above it**, which is how you create a bastion and the hosts routed through it in a single import. That name is matched only against earlier rows of the same file, never against your existing hosts. So **file order matters**: put a bastion above the hosts routed through it, and the innermost bastion first in a chain. A reference pointing further down the file is an error that tells you which line to move, and the preview reports all of them at once. A two-hop fleet in one file, ordered innermost bastion first. `edge-gw` is written before `bastion-dmz` refers to it, and both before `sensor-01` routes through the pair: ```csv theme={null} name,private_ip,is_jump_host,jump_hosts,jump_host_username_credential,jump_host_secret_credential edge-gw,10.0.0.1,true,,cred_1f3c8a90-...,cred_2a7d5e10-... bastion-dmz,10.30.0.5,true,edge-gw,cred_3b8e6f21-...,cred_4c9a7b32-... sensor-01,10.30.4.12,,"edge-gw,bastion-dmz",, ``` Swap the first and third lines and `sensor-01` fails with "edge-gw is defined on row 3, below this one". Everything else a file references - credentials, specialties, an agent's host - must already exist. ## What an import does to existing data Two rules cover everything. **A row with an `id` updates that entity. A row without one creates it.** There is no third case. An `id` that does not exist, or belongs to another organization, is an error - an `id` never creates one. **An update states the whole record.** A file that updates anything has to carry **every column**: a value sets a field, an **empty cell clears it**, and a missing column is rejected with the column named. There is no partial update by file. What that means in practice: * **Bulk-edit by editing an export.** An export already contains every column, so export → change what you want → import does exactly what it looks like: the fleet matches the file. * **A two-column update file is an error, not a shortcut.** `id,knowledge` will not import. That is deliberate: the alternative is a file that quietly blanks every field it forgot to mention. * **To change a single host or agent**, use its own API endpoint (`POST /api/v1/hosts/{id}`) rather than a file. * **A file that only creates may carry any subset of columns**, because there is nothing to lose - `name,private_ip,jump_hosts` is a perfectly good first-import file. Anything the file leaves out takes its default. In JSON the same rule applies per key: an update row carries every key, and `null` clears. Validation runs on the result, not on the row: a create must produce a complete, valid entity, and an update may not clear something mandatory or leave an invalid combination. Nothing can produce a half-valid host or agent - a row that would fails instead. Re-importing a file whose `id` cells are empty tries to create everything again. Host names must be unique in an organization, so you get errors rather than duplicates - keep the ids if you want a file you can re-apply. ## Importing In Command Center, open **Hosts** or **Agents** and use **Import**. Pick a `.csv` or `.json` file and press **Preview** first: the preview is the real server verdict for every row - names resolved against your live inventory, licences checked - not a guess made in the browser. Nothing is written until you confirm. Options: * **Stop at the first failure.** Off by default, so every row is attempted and you get a full report in one pass. On, the run stops at the first failing line and the rows below it are left untouched. Rows are applied in file order. Every one comes back with a result - created, updated, skipped or failed - against its line number, so a partial import tells you exactly what happened where. There is no rollback: rows that succeeded stay. ### Via the API The same endpoints back the UI, so anything you can do in Command Center you can script. ```bash theme={null} # CSV: the file is the body curl -X POST "https:///api/v1/hosts/batch?org_id=&dry_run=true" \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: text/csv" \ --data-binary @hosts.csv # JSON: rows in an envelope curl -X POST "https:///api/v1/agents/batch" \ -H "Authorization: Bearer $API_KEY" \ -H "Content-Type: application/json" \ -d '{"org_id":"","dry_run":true,"rows":[ ... ]}' ``` `dry_run: true` validates everything and writes nothing. The other control is `on_error` (`continue`, the default, or `stop`). A batch is capped at 10000 rows. The response is HTTP 200 with one result per input row even when every row failed - the per-row verdicts are the answer. A 4xx means the request itself was wrong: bad authentication, no access to the organization, a malformed envelope, or a file-level problem such as an unknown column, which fails the whole file because no row can be trusted after it. ## Exporting **Export** on the Hosts or Agents page downloads the current organization as CSV. Or call the endpoint: ```bash theme={null} # CSV curl "https:///api/v1/hosts/export?org_id=" \ -H "Authorization: Bearer $API_KEY" -H "Accept: text/csv" # JSON - a plain array of rows, ready to post straight back to /batch curl "https:///api/v1/agents/export?org_id=" \ -H "Authorization: Bearer $API_KEY" ``` Add `template=true` for the header row alone, as a starting point for a hand-written file. Reading requires read access, importing requires write access - so an auditor can export a fleet and can never import one. **Secrets never leave.** A credential is exported as a reference, never its value. A fleet rebuilt from an export needs its secrets supplied again. Export is capped at 10000 rows, the same as import, so an export is always a file that can be imported. ### Round trip Exporting and re-importing without editing reports every row as `updated` and changes nothing. This holds within one organization: every reference in the file is an id, and ids do not exist in another organization. Two more edges: * A **CSV** export of an agent loses credential detail the role columns cannot hold - an optional credential, two credentials on one role, a custom order, a public key. JSON keeps all of it, and a lossy CSV export says so in an `x-export-lossy` response header. * An agent with no host is not a valid row in either format, so it is left out of the file and counted in a response header. ## Versioning The columns are part of the `v1` API contract. New optional columns can be added, and files written today keep importing. Renaming or removing a column, or changing what one means, would be a new API version with new templates - your `v1` files would keep importing against `v1`. # Knowledge Base Source: https://docs.2501.ai/0.12/configure/knowledge Upload documents and let the engine extract reusable knowledge for your agents The Knowledge Base turns your existing documentation into knowledge your agents can use. Upload runbooks, network diagrams, inventory spreadsheets, and procedures, and the engine automatically parses each document and extracts reusable operational knowledge. You manage the Knowledge Base in **Command Center** → **Knowledge**. Knowledge page ## Uploading Documents Drag and drop files onto the Knowledge page, or use the file picker to select one or more documents at once. Each file is processed as its own ingest job. **Supported file types:** PDF, DOCX, CSV, and Markdown. **Maximum size:** 200 MB per file. When you upload several files together, each one becomes a separate ingest job with its own status. Documents are processed two at a time per organization, in the order you uploaded them, and the rest queue and start as slots free up. A step that runs too long is automatically timed out so the queue keeps moving. ## Processing Status Each document shows its live status as it moves through the pipeline: | Status | Meaning | | ---------- | ------------------------------------------------------------------- | | Uploading | The file is being transferred to the engine. | | Parsed | The document has been read and its text extracted. | | Extracting | The engine is pulling out rules, blacklist entries, and host facts. | | Done | Processing is complete and the extracted knowledge is available. | | Failed | Processing could not complete. | | Canceled | You canceled the ingest before it finished. | You can cancel an in-progress ingest at any time, or delete a document once it is done. ## What Gets Extracted From each document the engine extracts three kinds of knowledge: * **Operational rules**, directives that guide agent behavior. These appear alongside any rules you create by hand. See [Operational Rules](/0.12/configure/operational-rules). * **Blacklisted commands**, forbidden commands that agents must not run. These appear alongside your manually created entries. See [Blacklist](/0.12/configure/blacklist). * **Per-host facts**, details about specific machines, such as roles, IPs, VLANs, and procedures tied to a host. ### How Host Facts Are Matched Host facts are matched to machines in your inventory by name, IP, or alias. When a fact maps to a known [host](/0.12/core-concepts/hosts), it is attached to that host. Facts that cannot be matched to a known host are still kept so you can review them and reconcile them with your inventory later. ## PDF Vision Captioning Network documentation often lives in diagrams rather than text. When a multimodal (vision) [model](/0.12/configure/models) is configured for your tenant, the engine describes images embedded in PDFs, along with diagram and scanned pages, and folds those descriptions into the extracted content. This means topology, IP addresses, and VLANs drawn in diagrams are captured rather than lost. Without a vision model configured, PDFs are processed as text only. Diagrams and scanned pages will not contribute their visual detail to the extracted knowledge. ## Re-Uploading and Deleting ### Re-Uploading Re-uploading a document with the same name supersedes the previous version. The old document's extracted rules, blacklist entries, and host facts are removed and replaced by the new version's results. Rows you created or edited by hand in Command Center are never touched by re-ingest. Re-uploading only replaces knowledge that was extracted from that document. ### Deleting Deleting a knowledge document also deletes every operational rule, blacklist entry, and host fact that was extracted from it. Manually curated rows are left intact. ## How Agents Use Knowledge Extracted host facts and the per-host **Knowledge** field (shown on each [host](/0.12/core-concepts/hosts)) are automatically provided to agents at task time. Agents can also pull additional relevant rules and host facts on demand while working through a task, so the most pertinent knowledge is available without overloading every task with the entire Knowledge Base. # Licensing & Usage Source: https://docs.2501.ai/0.12/configure/licensing Apply your 2501 license, understand entitlement caps, and track usage from Command Center A **license** is a signed token (JWT) issued by 2501 that sets your tenant's caps on **hosts**, **gateways**, and **tasks**. Without one, the tenant runs on a limited free tier so you can evaluate the product. The Usage page lets administrators track consumption against the caps and explore LLM usage over time. The Usage page, plan data, analytics, and the license banner are visible to **administrators only**. Licenses can be contract or trial, and have a validity window. A license with no end date is shown as **Perpetual**. ## Free tier (no license) If no license has been applied, the tenant runs on a built-in free tier with capped resources: | Dimension | Free-tier cap | | --------------------------------------------- | ------------- | | Tasks (lifetime, across ad-hoc and recurring) | 50 | | Hosts | 10 | | Gateways | unlimited | The free tier applies only when there is **no active license**. Once you reach the cap on a dimension, creation in that dimension is rejected until a license is applied. An **expired** license is a hard block, not a free-tier fallback — apply a fresh license to keep going. ## Applying a license In **Command Center**, go to **Settings** > **License**. Paste the signed token (a JWT) issued by 2501 into the license field. Click **Apply License**. The new license becomes active immediately, and the previously active license is archived. Only **one license is active at a time**. The License screen also shows a **License history** of every license you have applied; applying a new one archives the previous one. License settings ## How the task cap is enforced The **tasks** cap is a tenant-wide **lifetime count** of every task ever created (ad-hoc and recurring combined). It is **not** reset per month, and it is enforced everywhere tasks are created: in Command Center (UI and API) and when the engine turns incoming [gateway](/0.12/core-concepts/gateways) tickets into work. Once the lifetime task cap is reached (on a license, or on the free tier), **task creation is rejected** until a license is applied or its cap is raised. An expired license also blocks task creation. See [Tasks](/0.12/core-concepts/tasks#task-limits) for how this surfaces during normal use. ## The Usage page The **Usage** page (administrators only) is where you watch consumption and dig into LLM activity. Usage page ### Plan usage The **Plan-usage** card shows a used-vs-cap bar for each entitlement (hosts, gateways, and tasks), plus the contract start and end dates and days remaining. ### Usage banner A tenant-wide banner appears across Command Center when **hosts** or **tasks** usage reaches **90% or more** of its cap. The banner links straight to the Usage page. ### Activity and LLM usage The Usage page **Activity** section explores LLM usage over a date range, with a toggle between **Cost** and **Tokens**. * **Tokens mode** counts all token usage in the range: input, output, and cached tokens. * **Cost mode** stays anchored to tickets and [pricing plans](/0.12/configure/models#models). Define pricing plans on your models so Cost mode can attribute spend. # Models Source: https://docs.2501.ai/0.12/configure/models Catalog the LLMs your agents can pick as engines A **Model** is a specific LLM your agents can pick — main engine, secondary engine, or as the tenant's text / multimodal default. Each model belongs to a [Provider](/0.12/configure/providers). Models are managed in **Command Center → Settings → Models**. Models catalog ## Fields | Field | Required | Description | | ----------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Provider** | Yes | The provider that serves this model | | **Model ID** | Yes | Must match the model's **deployment name** on the provider exactly — this is the identifier sent to the provider's API, not a label you choose | | **Key** | No | Display name shown in Command Center and referenced by agents when selecting an engine. Auto-derived from the Model ID if blank. Edit this field to rename | | **Context length** | Yes | Maximum tokens the model accepts. Drives context-management guidance in [Engine & Agents](/0.12/understand/engine-agents) | | **Accepts image input** | No | Enable for vision-capable models. Image-capable models become selectable as the tenant's **Multimodal model** | | **Enabled** | No | Only enabled models appear as selectable engines. Disable to retire without losing configuration or history | Advanced sampling parameters (temperature, top P, top K, presence/frequency penalty, max output tokens, seed, stop sequences) can be set per model. You can also record pricing plans per model and run an on-demand performance test that reports reachability, time to first token, and throughput. ## Thinking effort A **thinking-effort** level trades depth of reasoning against latency and cost: | Level | Description | | ----------- | ------------------------------------------ | | **Off** | No reasoning tokens — fastest, lowest cost | | **Minimal** | Very light reasoning | | **Low** | Basic reasoning pass | | **Medium** | Balanced reasoning depth | | **High** | Deep reasoning for complex tasks | | **xHigh** | Maximum reasoning; highest cost | Some custom or compatible providers do not accept a reasoning argument and reject any request that includes one. If your provider does not support reasoning, leave thinking effort unset — do not select any level. ## Tenant defaults The tenant's default **Text LLM model** and **Multimodal model** are chosen in **Settings → Tenant** from the catalog's enabled models. Only image-capable models appear as Multimodal options. These defaults power gateway routing, the AI Assistant, knowledge ingestion, and any agent that doesn't pick its own engines. See [Engine & Agents](/0.12/understand/engine-agents) for how agents combine main and secondary models, and where the tenant defaults are used. ## Deletion rule A model assigned to any agent — main or secondary — cannot be deleted. Reassign those agents or disable the model instead. ## Deploy-time seeding On deploy, the installer seeds models from the providers detected in the engine environment. After that, management moves to Command Center — no engine restart needed. # Operational Rules Source: https://docs.2501.ai/0.12/configure/operational-rules Tag-matched guardrails that apply to the right tasks, not every agent Operational Rules are directives that guide how agents execute tasks. In 0.7, rules are a tag-matched knowledge store: instead of injecting every rule into every agent, 2501 matches rules to each task based on what the task does and which host it targets. This keeps rules relevant, avoids noise, and lets you scope a guardrail precisely. **Example:** A rule like "Always restart a service after editing its configuration" can be scoped so it only surfaces on tasks that touch a Linux web server and involve a restart action, rather than appearing for every agent in the organization. Rules are matched using tags. The same tag vocabulary that classifies your machines drives this matching, so it helps to tag hosts well. See [Hosts](/0.12/core-concepts/hosts) for how host tags work. ## How matching works Each rule carries tags that decide which tasks it applies to. When a task runs, 2501 compares the rule's tags against two things: the target host's tags, and tags inferred from the task description. Tags fall into two roles: * **Scope tags** (Technologies, OS, Type, Shell, and application tags) describe *which hosts* a rule covers. * **Procedure tags** describe *which action* a rule covers (for example restart, deploy, backup). ### Scope: which hosts a rule covers A rule's scope tags must **all** be satisfied by the target host's tags together with tags inferred from the task. Scope tags are restrictive: adding more of them narrows a rule to fewer hosts, and never widens its reach. * A rule with **no scope tags** applies broadly across the organization. * A rule tagged `os:linux` applies only to Linux hosts. * A rule tagged `os:linux` **and** `tech:nginx` applies only to Linux hosts that run nginx. ### Procedure: which action a rule covers A rule's Procedure tags are matched against the action 2501 infers from the task description. * A rule with **no procedure tag** is treated as an always-on guardrail. It applies whenever its scope matches, regardless of the action. * A rule with **one or more procedure tags** applies when the task's action matches **any** of them. A rule tagged with both `restart` and `deploy` now matches restart tasks *and* deploy tasks (earlier releases only matched one). ### Rules surface up front For remediation tasks, the organization's relevant rules and procedures are surfaced at the **start** of the task, ranked by relevance to the task description. The agent has the applicable guardrails in hand from the beginning rather than only when its intent happens to match exactly. ## Tag axes Tags come from a controlled vocabulary organized by axis. You pick values from the in-product picker when creating or editing a rule; free-text values are rejected. The picker always shows the complete, current list for each axis. | Axis | What it describes | Examples | | ---------------- | ------------------------------------------- | ---------------------------------- | | **OS** | The operating system of the host | `os:linux`, `os:windows` | | **Shell** | The shell family the agent uses on the host | `shell:posix`, `shell:non-posix` | | **Type** | What a host is *for* | `type:database`, `type:hypervisor` | | **Technologies** | What a host *runs* | `tech:nginx`, `tech:aws` | | **Procedures** | The action or verb the rule governs | `restart`, `deploy`, `backup` | The **OS**, **Shell**, **Type**, and **Technologies** axes are scope tags. **Procedures** is the action axis. ### Application tags Application tags are the one exception to the controlled vocabulary. They take the form `app:` (for example `app:billing-api`), are per-organization, and scope a rule to a specific business application so it only applies to the hosts that run a particular internal app. ### New tags in 0.7 This release adds new values to the Type and Technologies axes: * **New Type tags:** `type:hypervisor`, `type:control-plane`, `type:generic-storage`, `type:iot`, `type:vault` * **New Technology tags:** `tech:aws`, `tech:gcp`, `tech:azure`, `tech:proxmox`, `tech:podman`, `tech:rabbitmq`, `tech:hashicorp-vault` The Vault technology tag was renamed from `tech:vault` to `tech:hashicorp-vault`. Existing hosts and rules that used the old value are migrated automatically and keep matching, so you do not need to retag anything. ## Managing Operational Rules Go to **Command Center** → **Operational Rules** and click **Create Rule** to add a new entry. Open a rule from the list to edit it, or use the delete action on its row to remove it. Operational Rules ### Name A clear identifier for the rule. Example: `Restart services after config changes` ### Description The directive itself, the guidance an agent should follow when the rule applies. Write rules that are concise, specific, and focused on a single operational concern. Overly strict or conflicting rules can prevent agents from completing tasks. Test new rules carefully before rolling them out broadly. Example: `Always restart processes after updating their configuration files` ### Tags Pick the rule's scope tags (OS, Shell, Type, Technologies) and its Procedure tags from the picker. Leave scope tags empty for an organization-wide guardrail; add them to narrow the rule to a subset of hosts. Leave the Procedure empty for an always-on rule, or set one or more procedures to limit the rule to matching actions. ### Organization By default, a rule is scoped to the current organization. Switch it to **Scoped to Tenant** to make the rule available to all organizations. An untagged, tenant-wide rule behaves like the always-applied rules from earlier releases. ## Inspecting matches Each task's detail page shows a **matching trace**: which operational rules were applied to the task and which were skipped, with the reason for each skip (for example, an intent mismatch where the task's action did not match the rule's procedure). Use this to confirm a rule is reaching the tasks you expect and to debug rules that are too broad or too narrow. ## Best practices Be specific with tags. Scope each rule to the technologies, type, OS, shell, and procedures it actually applies to, so it surfaces only for the relevant hosts and actions. A rule left broad (few or no scope tags) applies everywhere it can match, which adds noise and risks an irrelevant rule getting in the agent's way. Tag your hosts consistently. Matching depends on host tags plus tags inferred from the task description, so good host tagging directly improves which rules surface. See [Hosts](/0.12/core-concepts/hosts). Keep rules outcome-focused rather than command-specific where you can, and review rule effectiveness regularly using the matching trace. Use [Blacklist](/0.12/configure/blacklist) to hard-block prohibited commands, and Operational Rules to guide behavior. Combine with [Specialties](/0.12/configure/specialties) for domain workflows that do not need tag-scoped enforcement. # Plugins Source: https://docs.2501.ai/0.12/configure/plugins Connect AI Agents to MCPs and plugins Plugins and MCP are ways to allow your agents to access various tools in a LLM-friendly way via function calling rather than some API endpoints or complex CLI commands. Plugins ## Adding a new Plugin In order to add a new plugin to 2501, please first contact us. We provide custom implementation for each customer, in order to have it working seamlessly with our agents. Plugins can be publicly available MCPs that are queried on a distant endpoint with or without authentication, and also in-house made MCPs. ## Managing Plugins Go to **Command Center** → **Plugins** and click the cog icon of the plugin you want to manage. ### Tenant-level When the **Tenant-level** toggle is on, the plugin is available to all organizations in the tenant. Turn it off to associate the plugin with the currently selected organization only, restricting it to that org's agents. ### Enabled Status Allow to quickly enable or disable a plugin. Useful if you realize some MCP functions do not behave in the way you expected. If a plugin is disabled, the agent can't call any of its functions, even if the plugins are associated to an agent. ### Assign to All Agents When you add a new plugin, you may want all agents to have access to this plugin/MCP's functions For example, a plugin could allow any agent to execute commands to push to a remote repository and create a pull request. ## Plugins per Agent You can have granular control over which agent has access to a certain range of plugins. If order to do so, either select an agent on the graph or navigate to **Agents**, and select the agent you want to manage. From here, you can enable or disable one or multiple plugins for this agent. ### Considerations Plugins and MCP functions are loaded in agent's context, which can impact an agent performance overtime (especially smaller models), so don't forget to clear your agent memory every once in a while to avoid hallucination, especially with plugins exceeding a few hundreds commands. # Prompting Source: https://docs.2501.ai/0.12/configure/prompting Learn the basics of prompting AI Agents ## General Guidelines When prompting LLMs for IT shell tasks, you need to keep in mind these general concepts: * **Context:** clearly explain the context agent will face. Environment, machine details, expectations, caveats... * **Hallucination and drift:** too much context will often end up with contradictory elements, drifting, wrong decisions, and more. 500 to 1000 words is usually decent enough for a good agent. * **Examples:** when providing examples, make sure they apply to multiple use-cases and provide at least 5-6 different situations examples. Too specific examples will often make the agent lose autonomy when facing unusual tasks, or too stubborn trying to fix issues with the same commands over and over. * **Autonomy and feedback:** be clear on when the agent should stop a task and return to user. While you usually want agents to solve issues end-to-end, explicitly indicating cases where user input is necessary will prevent it from making costly mistakes because of a lack of context or too much experimental commands. * **Phrasing:** LLMs are trained on a reward system most of the time, replicating it leads to better performance. An example of a better phrasing to "encourage" LLM to not do an action, rather than penalizing it if this action is done: "Do not shut down any service, you will break the app and fail." can become "We must keep all services running to avoid being locked in the task." ## Writing a prompt ### Anatomy of a prompt It is important to structure your prompt in a logical way for the LLMs to better understand its purpose. Usually, when you de-construct a good working prompt, you will find 4 big parts: * **Identity:** who the AI Agent is * **Goal:** what the agent is meant to do * **Process:** how the agent will manage its task * **Warnings:** indications where the agent may bump into caveats Think that from here, the agent will adopt the "personality" and "critical thinking" based on its identity, in order to achieve a goal, following a certain process and dealing correct when meeting caveats. ### Identity Define a proper identity to your AI Agent: it will help it gather the correct internal knowledge in its "mind" when working on a task. If an agent is about to work on DevOps tasks, you want to enforce it to think like a senior DevOps, not a general engineer: they don't think the same way, and probably tackle problems with different tools and point of view. For example, telling an agent "You are a software developer" to work on services running on your cloud: it will think primarily on the coding aspect of the service. While "You are a DevOps/Cloud Engineer", it will think CLI-first. ### Goal Explaining the goals of an agent is crucial to encourage autonomy and end-to-end resolution. Without proper goals, agents can drift towards resolving things you did not ask for. It also helps the agent to see the broader picture, better understand why some context is needed, and to use it in order to achieve the said goals. If you ask an agent to work on a broken CI/CD pipeline for example, explaining this pipeline needs to have zero downtime, log specific errors and should be exclusively using embedded shell scripts in yaml files will likely lead to intended results. Simply saying "This pipeline is broken, fix it" is a goal, but the agent will likely fix it in ways that are not following your business rules. ### Process Process explanation is usually the most important: you will explain how to fix an issue using a defined set of tools, how to get proper context and how to act when facing some unintended cases. What we like to do here is to explain: * What is available to the agent: MCPs, CLI tools and scripts * How to deal with responses and act based on it * Known edge-cases and business rules that are crucial to the task completion * Examples for basic tasks and how to solve it the right way Where difficulty lies in explaining the process is finding the right balance of specializing the agent into solving a task, while keeping the topic broad enough to be able to manage a wide variety of similar tasks. You can explain the process to an agent in order to correctly manage machine via Terraform, but you should avoid being too specific for upgrading some aspects of the machine unless absolutely necessary. You want the agent to see the bigger picture to be able to manage all aspects of a machine via Terraform, not only a couple lines like disk and CPU. ### Warnings This last part of the prompt is the place to indicate various counter-intuitive problems the agent can face when troubleshooting, as well as evoking dangerous actions that should not be done. For example, if one of your tools are at a newer version and there's no way for the agent to know what was updated, you can mention "The method 'list' of this tool does not return all fields. You need to query ID individually". It can also be to prevent dangerous or unnecessary commands, like "Always delete logs that are at least a month old, and keep the recent logs for traceability". ### Format A prompt can be written in various formats, but we encourage structured content rather than plain text. The reason is, more structured the content is, more understandable it becomes, and it applies to both humans and LLMs. We encourage you to write prompts using the industry standard, **markdown**. Some engineers prefer **XML** as it can appear more structured for complex tasks. ## 2501 Notes ### Specialities, Operational Rules and Blacklists We have deconstructed various aspects of the prompts given to agents in order to obtain better performances and let you have more control/organization over how your agents behave. For a general rule of thumb: * [Specialities](/0.12/configure/specialties) are used to specialize the agent into solving a certain set of tasks. It is meant to be about a specific topic, but broad within that topic to encourage end-to-end resolution and autonomy. It is the main influential factor for an agent's behavior. * [Operational Rules](/0.12/configure/operational-rules) are instructions shared across all agents, as well as ways to process for some tasks. It has higher priority than the speciality and can be used to enforce some resolutions in a specific way rather than letting the agent the freedom of solving a problem in various ways. * [Blacklists](/0.12/configure/blacklist) are a programmatic way to prevent some commands from being executed. You can still mention prohibited commands to your agents in the specialities, and provide an additional layer of security via Blacklists. ### Traceability For traceability and clarity of agent's task execution over time, we like to tell agents to indicate full path of current location before executing a function via `{{workspace_path}}`. Without `{{workspace_path}}`: ```bash theme={null} # We don't know where file.log is, nor where it is located file.log | grep "ERROR" ``` With `{{workspace_path}}`: ```bash theme={null} # "Precede all commands with {{workspace_path}}" cd /myapp/api/logs && file.log | grep "ERROR" ``` ### OS, tools and Remote Execution Context It is important to specify the machine's type to agent, especially when operating on software-specific distribution systems. Omitting some crucial context about the agent's environment may lead to unreachable tools, installation of unwanted tools, and attempting to reach non-existent locations. In general, you want to indicate OS, Distribution, and possibly version if major changes happened from a version to another that matters to you. Then, indicate the commonly-available tools: some distribution systems (like FortiOS) may not have UNIX or BSD distribution despite being a Linux machine. Finally, make sure you write prompts in the context of how the agent will be used: the tools location, availability and permission may change based on the execution method, workspace or user. For example, an agent executing commands without remote-execution, will most of the time have access to non-UNIX installed CLI tools. However, if remotely executing commands, the user may not be root and don't have access to the tools unless the full path is mentioned. For an agent with full permissions by default, as the root user: ```bash theme={null} sqlplus << EOF set pagesize 0 set feedback off select username, con_id from cdb_users where username = 'TABLE_NAME'; exit; EOF ``` For an agent remotely executing commands, with a non-root user: ```bash theme={null} # .bashrc may not be loaded. Full path necessary, identify as administrator $ORACLE_HOME/sqlplus -S / as sysdba << EOF set pagesize 0 set feedback off select username, con_id from cdb_users where username = 'TABLE_NAME'; exit; EOF ``` # Providers Source: https://docs.2501.ai/0.12/configure/providers Register the LLM endpoints the engine can call A **Provider** is an LLM endpoint the engine talks to. One provider can expose many [Models](/0.12/configure/models). Providers are managed in **Command Center → Settings → Providers**, scoped to your tenant. Viewing the catalog is open to any tenant user; only **tenant-level** users can add, edit, or delete providers. See [Users & Organizations](/0.12/deployment/users-organizations). Providers catalog ## Fields | Field | Required | Description | | -------------------- | --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Key** | Yes | Short identifier, unique within your tenant. Example: `openai`, `cloudtemple` | | **Provider type** | Yes | Integration used to reach the provider — see [Provider types](#provider-types) | | **Base URL** | Compatible & Azure types | Endpoint root. Required for compatible/Azure types; leave empty for native types | | **API Key Var Name** | When auth is required | Name 2501 resolves to find the secret — see [API key resolution](#api-key-resolution) | | **Auth Header Name** | Compatible types (optional) | Override the HTTP header used to send the API key. Defaults to each type's native header — `Authorization` (Bearer) for `openai-compatible`, `x-api-key` for `anthropic-compatible`. Set to e.g. `api-key` for Azure API Management or other proxy gateways that require a non-standard header. | | **Organization** | No | Scope this provider to a specific organization. When set, only agents in that organization use this provider and its credentials. Leave unset to make the provider available tenant-wide. | | **Requires auth** | — | On by default. Turn off for endpoints with no key (e.g. self-hosted on a private network) | | **Enabled** | — | Disabling makes all of its models unavailable without deleting anything | ## Provider types | Family | Types | Notes | | -------------- | ---------------------------------------------------------------------------------- | ------------------------------------------------------- | | **Native** | `openai`, `anthropic`, `mistral`, `deepseek`, `cohere`, `openrouter`, `togetherai` | Built-in integrations. No endpoint URL needed. | | **Compatible** | `openai-compatible`, `anthropic-compatible`, `azure` | Point at any API-compatible endpoint via a **Base URL** | Use a **compatible** type to register a self-hosted model server (vLLM, Ollama, LM Studio) or any vendor exposing an OpenAI- or Anthropic-style API. Adding a new endpoint of an existing type is pure configuration. Adding a brand-new *type* (a vendor with its own API shape) requires a 2501 release — contact your Account Executive. ## API key resolution The API key is resolved at runtime in two steps: 1. 2501 looks for a [Credential](/0.12/configure/credentials) matching the **API Key Var Name** — for an **org-scoped** provider, one scoped to that organization first, then a tenant-wide one; for a **tenant-wide** provider, only a tenant-wide credential. 2. If none exists, it falls back to an environment variable of the same name on the engine. This lets you rotate keys in the UI without touching the engine environment. When you save the provider, the dialog checks that the key resolves and warns if it cannot be found. For a **tenant-wide** provider (no Organization set), the matching Credential must be **tenant-scoped** — a credential restricted to a specific organization will not be found. For an **org-scoped** provider, the credential may be scoped to that organization or tenant-wide. Providers that do not need a key (e.g. a self-hosted model on a private network) can turn off **Requires auth**. ## Azure settings For the `azure` type, two extra fields appear: | Field | Default | Description | | --------------- | -------------------- | --------------------------------------------------------------- | | **API Version** | `2025-04-01-preview` | The Azure OpenAI API version to target | | **API Mode** | `chat` | Which API surface to call: `chat`, `responses`, or `completion` | Azure API Management (APIM) gateways are supported as the endpoint: set the **Base URL** to your APIM gateway URL to route model traffic through it for centralized policy, rate limiting, and logging. ## Deletion rule A provider that still has models cannot be deleted. Remove or reassign its models first, or simply disable the provider instead. ## Deploy-time seeding On deploy, the installer seeds providers from API keys detected in the engine environment. After that, management moves to Command Center — no engine restart needed for additions or edits. # Agent Source: https://docs.2501.ai/0.12/configure/resources/agents MDX frontmatter reference for agent resources **Subdirectory:** `agents/` · **Body:** none Agents are scoped to the **organization of their host** — you do not set `org` on the agent itself; it is resolved from the host reference at sync time. ## Frontmatter | Field | Type | Required | Default | Description | | ------------------- | ------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | No | — | Display name. Omitting it lets the platform auto-generate one. | | `specialty` | string | **Yes** | — | Reference an existing [Specialty](/0.12/configure/resources/specialties) by its `name`. | | `host` | string | **Yes** | — | Reference an existing [Host](/0.12/configure/resources/hosts) by its `name`. The agent's organization is inherited from this host. | | `main_engine` | string | **Yes** | — | Model `key` for the main engine (must exist in the tenant model catalog). | | `secondary_engine` | string | **Yes** | — | Model `key` for the secondary engine. | | `remote_execution` | boolean | No | `true` | Almost always `true`. Set `false` only for the rare local-execution case. | | `credential_config` | array | No | `[]` | Credential bindings. **Array position carries priority** — index 0 is tried first. | ### `credential_config[]` shape | Field | Type | Required | Description | | ----------------- | ------- | ------------------- | ------------------------------------------------------------------------------------------------- | | `credential` | string | **Yes** | Reference a [Credential](/0.12/configure/resources/credentials) by its `name`. | | `credential_role` | string | **Yes** | Role this credential fills (`ssh_username`, `ssh_password`, `ssh_private_key`, `ssh_public_key`). | | `is_required` | boolean | No (default `true`) | Whether the agent fails to start without this credential. | ## Fields NOT supported These are managed elsewhere and cannot be set in MDX: * `id`, `tenant_id`, `org_id`, `created_at`, `updated_at` — system-set * `archived_at` — managed from the UI * `host_id`, `specialty_id`, `credential_config[].credential_id` — resolved from the **name** references above Any extra key in frontmatter causes a strict-schema validation error. ## Example ```mdx agents/web-01-agent.mdx theme={null} --- name: web-01-agent specialty: linux-administration host: web-01 main_engine: anthropic/claude-sonnet-4-6 secondary_engine: openai/gpt-4o remote_execution: true credential_config: - credential: web-admin-user credential_role: ssh_username is_required: true - credential: web-admin-password credential_role: ssh_password is_required: true --- ``` ## Gotchas * **Agents have no `org` field.** They inherit it from `host`. Renaming a host you reference from an agent is a coordinated change — pull, edit both, then sync. * **`main_engine` and `secondary_engine` are model *keys*, not Model IDs.** The key is the display identifier in the catalog; the Model ID is what's sent to the provider's API. * **Pull skips agents with no host.** An agent file is only meaningful with a host reference, so the exporter drops orphaned ones. # Blacklist Source: https://docs.2501.ai/0.12/configure/resources/blacklist MDX frontmatter reference for blacklist entries **Subdirectory:** `blacklist/` · **Body:** must be empty Blacklist entries carry no body — the pattern is in frontmatter. ## Frontmatter | Field | Type | Required | Default | Description | | ------------- | -------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------ | | `org` | string \| null | No | tenant | Organization name. Omit (or `null`) for tenant-wide blocks applied to every org. | | `pattern` | string | **Yes** | — | The pattern that blocks matching commands. Substring match by default; `*` and `?` glob wildcards supported. | | `description` | string \| null | No | `null` | Why this command is blocked. Surfaces in the matching error. | ## Fields NOT supported * `id`, `tenant_id`, `created_at`, `updated_at` — system-set Body must be empty — putting anything in it fails validation. ## Example ```mdx blacklist/rm-rf-root.mdx theme={null} --- org: platform-team pattern: rm -rf / description: Catastrophic. Always block — agents must scope deletes to a specific directory. --- ``` ```mdx blacklist/interactive-editors.mdx theme={null} --- pattern: vim description: Interactive editor — LLMs cannot drive its TUI. Use `sed` or write the file directly with `cat <` pin exists — rule retrieval is pure tag intersection. To target a specific host, give it (and the rule) a matching `app:` tag. * **Auto-extracted rules from Knowledge can be replaced by re-uploads.** A rule you declare in MDX is independent of the extractor and won't be touched by re-ingest. # Specialty Source: https://docs.2501.ai/0.12/configure/resources/specialties MDX frontmatter reference for specialty resources **Subdirectory:** `specialties/` · **Body:** the specialty prompt The MDX body is the specialty prompt itself — the text the agent reads to shape how it thinks about a domain. See [Prompting for a Specialty](/0.12/prompting/specialty) for what to put in the body. ## Frontmatter | Field | Type | Required | Default | Description | | ---------------- | ---------------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------- | | `org` | string \| null | No | tenant | Organization name. Omit (or `null`) to make the specialty available to every org in the tenant. | | `name` | string | **Yes** | — | Display name. Used by agents to reference this specialty. | | `description` | string | No | `""` | Optional short summary shown in lists. | | `execution_mode` | `investigate_only` \| `null` | No | `null` | Pin every agent using this specialty to read-only mode (overrides ticket tags). | ## Fields NOT supported * `id`, `tenant_id`, `created_at`, `updated_at`, `key` — system-set (the `key` is derived from `name`) * The prompt itself — lives in the **body**, not frontmatter ## Example ```mdx specialties/linux-administration.mdx theme={null} --- org: platform-team name: linux-administration description: Generalist Linux operator — system services, file system, package management. --- # Identity You are a senior Linux operator. You prefer reversible, incremental changes and you always verify state before and after acting. # Process 1. Inspect first — confirm service status, recent logs, and configuration before reaching for a fix. 2. Validate the proposed change in isolation when possible (`nginx -t`, `systemctl --dry-run`, `--dry-run` on Ansible, etc.). 3. Apply the smallest change that resolves the issue. 4. Re-verify after acting and report what changed. # Warnings - Service restarts can drop in-flight requests. Prefer `reload` where the service supports it. - Do not `mkfs` on any block device without explicit confirmation from a human. ``` ## Gotchas * **`execution_mode: investigate_only` is a ceiling.** Any agent using this specialty runs read-only even if a ticket asks for `@2501:remediate`. The resolution comes back as `partial`. Use this on critical-system specialties; lift it explicitly when you want remediation. * **Changes apply immediately.** Editing the body and syncing affects every agent using this specialty on its very next task — test on a non-critical specialty first. * **Renaming a specialty is a coordinated change** — every agent referencing it by `name` must update too. Pull first, edit both sides, then sync. # Specialties Source: https://docs.2501.ai/0.12/configure/specialties Configure your agent for specific task domains Specialties let you specialize agents for specific tasks. Every agent needs a specialty to run. If you don't select one during initialization, it defaults to `SYSOPS`. ## Purpose of Specialties While 2501 agents are built for autonomous system operations, specialization improves execution accuracy and reliability. Instead of operating generically, specialized agents follow domain-specific guidelines tailored to your infrastructure. **Example:** An agent with an AWS CLI specialty can manage EC2 instances with specific guidance on CLI usage patterns, error handling, and decision-making logic (like determining appropriate CPU upgrades when asked to "upgrade my sandbox machine"). Specialties work well for: * Providing context for proprietary or lesser-known tools * Establishing workflows without full MCP integrations * Documenting internal conventions and procedures Specialties list ## Separation of Concerns To maximize accuracy, use different tools for different purposes: * **Specialties**: Define workflows, provide context, establish best practices * **[Operational Rules](/0.12/configure/operational-rules)**: Enforce specific tool usage or behavioral requirements * **[Blacklist](/0.12/configure/blacklist)**: Prevent execution of prohibited commands ## Managing Specialties Go to **Command Center** → **Specialties** and click **Create Specialty** to add a new one. ### Name The display name for your specialty. Use naming conventions that reflect the service domain or agent role. Example: `TERRAFORM_SPECIALIST` ### Key A read-only identifier automatically generated from the name. Use this key in CLI commands to assign specialties during agent initialization. ### Description Context about the specialty's purpose and scope. Agents read the full description — not just the name — when deciding how to plan and assign tasks, so a clear description improves routing accuracy for specialties where the name alone is ambiguous. Example: `Handles Terraform infrastructure files for sandbox environments` ### Scope A specialty is scoped to either a single organization or the whole tenant. By default it is **scoped to the current organization**, so only that org's agents can use it. Toggle the scope to **tenant** to make it available to all organizations. ### Investigate-Only Mode A specialty can be pinned to **investigate-only**, meaning any agent using it will always run in read-only mode regardless of what the ticket or gateway requests. This acts as a ceiling: even if a ticket is tagged `@2501:remediate`, agents with an investigate-only specialty will only diagnose and report. This is useful for safely deploying large fleets of agents (e.g., 1000 Oracle DB agents) where you want read-only analysis by default. You can then selectively enable remediation for specific groups by changing this single setting on the specialty. When a ticket requested remediation but investigate-only specialties prevented it, the job resolution is flagged as **partial** rather than a failure. See [Agents: Investigate vs Remediate](/0.12/core-concepts/agents#execution-modes-investigate-vs-remediate) for the full priority chain. ### Prompt The specialty definition itself. Use structured formats like Markdown or XML for clarity. **Important:** Changes to a specialty immediately affect all agents using it. To test modifications safely, create a new specialty for testing, assign it to a test agent, validate the behavior, then update the production specialty. For guidance on effective agent prompting, refer to our [Prompting Guide](/0.12/configure/prompting). ## Best Practices Keep specialty prompts focused and domain-specific. Document expected behaviors and decision criteria with clear examples for complex workflows. Test specialty changes on isolated agents before rolling them out to production. For critical systems, consider version-controlling your specialty definitions. # Users Source: https://docs.2501.ai/0.12/configure/users Manage users and permissions from the Command Center ## Managing users In your Command Center, Administrator users will be able to see a "Users" tab on the **Settings** page. From this tab, you are able to create, update, or modify users able to access your 2501 interface. Users ## Roles 2501 supports three roles: Administrator, User, and Auditor. Each role has different levels of access to resources and operations. ### Administrator The administrator user has access to all pages of the Command Center and can modify every resource 2501 manages. Administrators can create new users with the button **"Create User"** and assign them a role. To revoke the access of a user to Command Center, you can deactivate the user from the table. Deactivation is a soft delete: the account is disabled (it can no longer sign in) but its related records are preserved, and the user can be reactivated later. You may also want to reset a user's password (including administrators), using the reset-password action. Note on passwords: a password must contain at least 8 characters, including at least one uppercase letter, one lowercase letter, one number, and one special character. **Permissions:** * Full read and write access to all resources across all organizations * Can create, modify, and delete users * Can manage organizations and tenant settings ### User Regular users can read and write resources within their assigned organizations, but have **read-only access** to shared (tenant-wide) resources such as specialties, operational rules, credentials, and blacklists that are not scoped to a specific organization. Regular users do not have access to the "Users" page and cannot manage other users or organizations. **Permissions:** * Read and write access to org-scoped resources (agents, hosts, tasks, jobs, etc.) in assigned organizations * Read-only access to shared resources (specialties, operational rules, credentials, blacklists with no organization scope) * Read-only access to organization and tenant information * No access to user management ### Auditor Auditors have read-only access across all resources they can see. They share the same visibility as the User role but cannot create, modify, or delete any resource. **Permissions:** * Read-only access to org-scoped resources in assigned organizations * Read-only access to shared resources * Read-only access to organization and tenant information * No write permissions anywhere * No access to user management ## Organization Access Regardless of their role, any user can be granted access at two levels: * **Organization-level access**: The user can only see and interact with resources in their explicitly assigned organizations. * **Tenant-level access**: The user can see and interact with resources across all organizations in the tenant, including any organizations created in the future. This applies to all roles. An Administrator with organization-level access will only manage resources within their assigned organizations, while an Auditor with tenant-level access can audit all organizations. Tenant-level access is configured when creating or updating a user via the CLI. See [Users & Organizations](/0.12/deployment/users-organizations) for details. # Webhooks Source: https://docs.2501.ai/0.12/configure/webhooks Set up real-time webhook notifications from ServiceNow Webhooks allow ServiceNow to notify 2501 immediately when an incident or change request is created or updated, instead of waiting for the next polling cycle. Each webhook registration generates a unique URL and shared secret that you configure on the ServiceNow side. ## Creating a Webhook Use the `2501 infra` CLI to register a new webhook: ```bash theme={null} 2501 infra webhook create \ --name \ --source-type \ --event-type \ --gateway-id ``` ### Parameters | Parameter | Required | Description | | --------------- | -------- | ------------------------------------------------------------------------ | | `--name` | Yes | Human-readable name for this webhook (e.g., `prod-incidents`, `staging`) | | `--source-type` | Yes | Ticketing system type (e.g., `servicenow`) | | `--event-type` | Yes | Resource type this webhook receives (e.g., `incident`, `change_request`) | | `--gateway-id` | No | Target gateway ID. Auto-selected if only one active gateway exists | | `--description` | No | Optional description for this webhook | ### Example ```bash theme={null} 2501 infra webhook create \ --name prod-incidents \ --source-type servicenow \ --event-type incident \ --gateway-id gtw_a1b2c3d4-... ``` Output: ``` Webhook created! ID: whk_e5f6a7b8-... URL: https://cmd.example.com/api/webhooks/ingest/4f8a2c1e9b3d7a0f6e5c8d2b Secret: a1b2c3d4... Event type: incident Gateway: gtw_a1b2c3d4-... --- ServiceNow Business Rule configuration --- Table: incident [incident] Active: ✓ Advanced: ✓ When: after Insert: ✓ Update: ✓ Filter Conditions: Assignment group is Active is true --- Script (paste into Advanced tab) --- (function executeRule(current, previous) { try { var request = new sn_ws.RESTMessageV2(); request.setEndpoint('https://cmd.example.com/api/webhooks/ingest/4f8a2c1e9b3d7a0f6e5c8d2b'); request.setHttpMethod('POST'); request.setRequestHeader('Content-Type', 'application/json'); request.setRequestHeader('X-Webhook-Secret', 'a1b2c3d4...'); request.setRequestBody(JSON.stringify({ sys_id: current.getUniqueValue() })); request.setHttpTimeout(5000); request.executeAsync(); } catch (e) { gs.error('2501 webhook failed: ' + e.message); } })(current, previous); ``` ## Configuring ServiceNow After creating a webhook, configure ServiceNow to send events to it using a **Business Rule**. ### Business Rule 1. In ServiceNow, navigate to **System Definition** > **Business Rules** 2. Click **New** 3. Configure: * **Name**: a descriptive name (e.g., `2501 Webhook - Incidents`) * **Table**: `incident` (or the table matching your `--event-type`) * **When**: `after` * **Insert**: checked * **Update**: checked 4. Check **Advanced** 5. Paste the generated script into the **Script** field 6. Optionally add a **Filter Condition** to scope which incidents trigger the webhook (e.g., `Assignment group is `) 7. Click **Submit** Both the `incident` and `change_request` tables are supported. A gateway that handles incidents and changes uses two webhooks, one per table, each with its own Business Rule. For how change requests are picked up and executed, see [Change Requests](/0.12/core-concepts/gateways#change-requests). ## Deleting a Webhook ```bash theme={null} 2501 infra webhook delete --id whk_e5f6a7b8-... ``` This removes the webhook registration from 2501. You should also deactivate or delete the corresponding Business Rule in ServiceNow. ## Network Requirements The ServiceNow instance must be able to reach your Command Center's URL over HTTPS. The webhook endpoint is: ``` POST https:///api/webhooks/ingest/ ``` ## How It Works 1. ServiceNow fires the Business Rule when a record (incident or change request) is created or updated 2. The script sends a POST request with the record's `sys_id` to the webhook URL 3. Command Center proxies the request to the engine 4. The engine validates the shared secret, then enqueues the ticket for processing 5. The engine fetches the full record details from ServiceNow and processes it through the [gateway](/0.12/core-concepts/gateways) pipeline Webhooks are complemented by a polling reconciliation that runs every \~2 minutes, ensuring no records are missed even if a webhook delivery fails. # Agents Source: https://docs.2501.ai/0.12/core-concepts/agents The LLMs autonomously browsing your systems Agents are the core execution units of 2501: autonomous AI systems that perform operational tasks on your infrastructure. Each agent combines LLM-powered intelligence with programmatic access to your systems, enabling complex workflows, issue diagnosis, and infrastructure management through natural language instructions. ## What is an Agent? A 2501 agent is an AI-powered operator capable of understanding context by analyzing tasks and interpreting system states, planning execution by breaking down complex operations into logical steps, taking action through commands and file modifications, adapting dynamically based on outputs and errors, and operating autonomously to complete multi-step tasks without constant intervention. Unlike traditional automation scripts, agents reason about their environment and make informed decisions rather than following rigid procedural logic. Agents list ## Agent Architecture ### Engine Pair Every agent uses two LLMs in tandem: * **Main Engine**: Handles direct task execution, file manipulation, and command execution * **Secondary Engine**: Manages orchestration, planning, validation, and oversight This dual-engine architecture separates execution from planning, improving accuracy and safety. Learn more in [Engines](/0.12/understand/engine-agents). ### Specialty Configuration Agents are assigned a [Specialty](/0.12/configure/specialties) that provides domain-specific guidance and workflows, ranging from general-purpose (`SYSOPS`) to highly specialized configurations like `TERRAFORM_SPECIALIST` or `AWS_CLI_EXPERT`. ### Operational Constraints Agents operate within boundaries defined by [Operational Rules](/0.12/configure/operational-rules) (organization-wide mandatory procedures), [Blacklists](/0.12/configure/blacklist) (prohibited commands), and [Credentials](/0.12/configure/credentials) (secure access to systems). ### Credential Access Control Each agent has an explicit **credential allowlist** — a list of credentials it may reference by name during task execution. Configure the list from the agent create or edit form. Only credentials on the allowlist are advertised to the agent; secret values are never exposed. Named credential placeholders in commands (e.g. `{{secret:my-api-key}}`) resolve only when the referenced credential appears on the agent's allowlist. An empty allowlist means no named credentials are available to the agent. ### Memory and Context Agents maintain task history within their context window, allowing them to reference previous operations, build on prior work, and maintain continuity across related tasks. When context limits are approached, tasks can be archived to clear memory while preserving agent configuration. ## Execution Modes: Investigate vs Remediate Agents support two execution modes that control what actions they can take: * **Remediate** (default): The agent diagnoses issues **and** applies fixes: commands, file changes, service restarts, etc. * **Investigate**: Read-only analysis. The agent diagnoses and reports findings without making any changes to the target system. The mode is determined at two levels: | Level | How | Scope | | ------------- | -------------------------------------------------------------------------------- | --------------------------------------------------------------------------------- | | **Ticket** | Tag with `@2501:investigate` or `@2501:remediate` in the ticket body or comments | Single ticket/job: defaults to remediate if no tag is present | | **Specialty** | Pin to investigate-only in the [specialty](/0.12/configure/specialties) settings | All agents using that specialty: acts as a ceiling that overrides ticket requests | The ticket tag sets the job mode, which propagates to individual tasks. If multiple tags appear (in the description and comments), the last one wins. Aliases like `@2501:investigation` and `@2501:remediation` also work. The specialty constraint is a **ceiling**: if a specialty is pinned to `investigate_only`, any agent using it will always run in read-only mode, even if the ticket is tagged `@2501:remediate`. When this happens, the resolution is flagged as **partial** rather than a failure. This lets you safely deploy large fleets of agents and selectively enable remediation by changing a single setting on the specialty. Tickets and jobs running in Investigate mode show a visible **Investigate** badge in the Command Center. ## Local vs Remote Execution **Remote execution** is the default and what virtually every deployment uses. The agent runs in 2501's infrastructure and connects to the target machine via the configured protocol (SSH, WinRM, gMSA over Kerberos). Tasks run without installing the CLI on the target, agent management stays centralized, and the same agent can operate across a fleet. **Local execution** is also supported — the agent runs directly on the machine where the 2501 CLI is installed, with direct filesystem and process access — but it's reserved for niche developer workflows and is rarely chosen in production. The execution mode is transparent to the agent itself. It uses the same capabilities regardless of where it runs. Execution Modes ## Agent Lifecycle ### Creation Agents are created through the Command Center UI (full-featured) or CLI (streamlined for quick deployment). During creation, assign a [host](/0.12/core-concepts/hosts), select main and secondary engines, assign a specialty, enable remote execution if needed, assign plugins, and configure credentials (for remote execution). Agents Create ### Configuration After creation, agents can be modified to change engine assignments, update specialty configurations, add or remove credentials, and adjust operational constraints. ### Task Execution Agents receive tasks through natural language instructions. The secondary engine analyzes the request and gathers context. It creates an execution plan, then the main engine executes actions and validates results. The agent adapts as needed and reports completion or escalates issues. ### Memory Management As agents work, their context window fills with task history. Manage memory by archiving completed tasks individually, clearing all memory for a fresh start, or selectively archiving unrelated tasks while preserving relevant context. ### Modification and Deletion Agents can be edited or removed through the Command Center UI (full management) or CLI (limited management for active agents). Agent Dialog ## Agent Organization ### Organization Scoping Agents belong to specific [organizations](/0.12/core-concepts/organizations), with available specialties, operational rules, blacklisted commands, and accessible credentials. This scoping enables different teams or environments to maintain separate operational standards while sharing infrastructure. ### Agent Naming Choose agent names that indicate purpose or responsibility (e.g., `aws-prod-manager`, `db-backup-agent`), target environment (e.g., `staging-deployer`, `prod-monitor`), or specialty domain (e.g., `terraform-provisioner`, `k8s-operator`). ## Testing Connectivity For agents configured for remote execution, you can verify connectivity to the assigned host directly from the Command Center. On the Agents page, use the **Test connection** action on the agent's row: this checks that the agent can reach its host using the configured protocol (SSH or WinRM) and credentials. ## Troubleshooting **Agent Not Completing Tasks:** Check context window usage and archive tasks if near limits. Verify credentials are correctly assigned and accessible. Review operational rules for conflicts. Ensure specialty provides adequate guidance. Consider upgrading engines for complex tasks. **Execution Errors:** Validate remote-access credentials — use the **Test connection** action to quickly verify connectivity. Check the blacklist for inadvertently blocked commands. Review task history for failure patterns. Verify target system accessibility and permissions. **Unexpected Behavior:** Review the agent's specialty for conflicting guidance. Check for overly strict operational rules. Examine task history to understand decision-making. Test with simplified tasks to isolate the issue. Consider adjusting engine assignments. For additional support, refer to the Prompting Guide for techniques to improve agent task understanding and execution. # Gateways Source: https://docs.2501.ai/0.12/core-concepts/gateways Connect agents to external services Gateways route tickets from your ticketing system to the right AI agents. They are the primary way to create jobs automatically from the tools your teams already use. **ServiceNow is the main, default gateway** and what most deployments run. 2501 can also be integrated with **custom-built gateways** for ticketing systems that are not in the catalog — contact your Account Executive if you need one. A separate `runner` gateway exists for [benchmarking in a sandbox environment](#runner-gateway) so scenario runs do not pollute ServiceNow or require a development account; it is not used in normal production. A gateway is itself agentic. It uses an LLM (your tenant's text LLM model) to read each incoming ticket, understand the problem, and decide which agents and hosts should handle it, and it uses your tenant's multimodal model to read any attachments on the ticket. Both come from your tenant defaults in the [model catalog](/0.12/configure/models), and you shape how the gateway behaves with two optional prompts — the [inbound prompt](#inbound-prompt) (routing and scope) and the [outbound prompt](#outbound-prompt) (what gets written back when a job finishes). Because routing is lighter work than executing a task, a smaller model is usually enough here. Gateways list When a ticket is created in your ticketing system, 2501 parses and handles it through the jobs system. Gateways can parse tickets in two ways: * **Automatic**: tickets are parsed as soon as they are created in the ticketing system * **Semi-Automatic**: tickets are parsed only when @2501 is mentioned After the gateway parses your ticket and any attachments, it determines how many tasks are required to resolve the issue. For each task, it identifies the best agents based on: * Host information in the ticket (e.g., "target\_machine: UNIX\_PROD\_442") * Nature of the incident (e.g., "A service has timed out") * Available agents with explicit specialties (e.g., "AWS Manager") * Other ticket details that indicate where and how the incident should be resolved Once all tasks are mapped to agents, a job is created containing these tasks. The gateway may schedule tasks for execution at a specific date and time, or set up a [recurring schedule](/0.12/core-concepts/jobs#recurring-jobs) when the ticket describes a repeating cadence. After a job is created, you can interact with it by commenting @2501 in the ticket. See [Working with Active Jobs](#working-with-active-jobs) to learn how to add follow-up work or update jobs mid-execution. Gateways handle two kinds of ServiceNow records: **incidents** and **change requests**. Incidents are the default reactive flow described above. Change requests follow a stricter, plan-driven flow described in [Change Requests](#change-requests). ## Organization Scoping Gateways are scoped to a specific [organization](/0.12/core-concepts/organizations). A gateway only routes tickets to agents within its organization and creates jobs within that organization's context. ## Managing Gateways Go to **Command Center** → **Gateways** to create, view, edit, activate/deactivate, and delete gateways. To **create** a gateway, click **New Gateway**, choose the type (ServiceNow or Runner), configure the [inbound](#inbound-prompt) and [outbound](#outbound-prompt) prompts and their override toggles, the deduplication window, and any metadata fields, then save. To **pause** a gateway without losing its configuration, open its detail page and toggle **Active** off — an inactive gateway stops processing tickets but keeps its settings. **Delete** removes a gateway entirely. Existing gateways also expose an editable **Metadata** field for non-standard configuration keys that are not surfaced as dedicated form inputs. ### Type Indicates what type of gateway it is, usually a service where IT tickets are created. Example: `servicenow` ### Active Status Turns the gateway on or off. Useful if you want to temporarily prevent agents from automatically picking up tickets or creating jobs when 2501 is mentioned in a ticket. ### Webhooks Tickets are processed in near real-time via webhooks. Each gateway gets a unique webhook URL and secret that you configure once in your ticketing system (for example, a ServiceNow Business Rule). No standing access to your ticketing instance is required after setup. A polling reconciliation loop is retained as a safety net, running every \~2 minutes to catch any missed webhooks. The gateway detail page shows active webhooks (name, event type, and masked secret) and lets you delete individual webhooks. You can re-create them via the CLI. For setup instructions, see [Webhooks](/0.12/configure/webhooks). ### Models Gateways do not have their own model settings. Routing uses your tenant's **Text LLM model** (the LLM in charge of understanding the ticket and routing tasks to the appropriate agent(s)), and attachments are parsed with your tenant's **Multimodal model**. Both are tenant defaults set in **Settings** > **Tenant** from the [model catalog](/0.12/configure/models). You can allow a smaller weight for these defaults, as gateway routing requires less compute than the agents that actually perform tasks. A model between 70b and 300b performs well enough for most routing tasks. ### Gateway Prompts A gateway is shaped by two independent prompts, each an optional **override** you toggle on. When a side's override is off, the gateway uses its built-in default handling for that side (today's behavior). You can enable one side without the other. #### Inbound Prompt The inbound prompt specifies how to route tickets to the appropriate agent. It extends the gateway's system prompt to allow routing of specific ticket types that require special handling or particular agents. Best practice: include something like `If there is no exact agent for the specified task on the current host - take the closest. But ensure exact matching for hosts`. This emphasizes using the ticket's information to identify the correct agent while providing a fallback when an exact match is not available. The inbound prompt also acts as a **scope gate**. If it defines which kinds of requests the gateway should or should not handle, tickets whose request type falls outside that scope are skipped (see [Routing and Scope](#routing-and-scope)). #### Outbound Prompt The outbound prompt governs what 2501 writes back to the ticket once a job finishes. With its override on, the agent maps the internal outcome to the external ticket: it sets the **status** (using your ticketing system's real states and close codes — for ServiceNow, an incident resolves to "Resolved" with a close code, a change request moves to "Review", and so on), optionally posts a **comment** (a public comment or an internal work note), and optionally **escalates** by reassigning the ticket to another assignment group. The gateway validates each choice against the legal states for that ticket type. With the override off, the existing default mapping applies unchanged — for example, a successful incident is resolved. See [Completion Behavior](#completion-behavior). ## Routing and Scope Before any task is generated, the gateway checks whether the ticket is in scope for this gateway, based on the request-type rules in your inbound prompt. * **Out-of-scope tickets are skipped.** A skipped ticket is marked with a distinct **"Skipped"** status, visible and filterable in Command Center (like a "Duplicate"), rather than being silently processed. Scope is about the *type* of request, not which machines exist, so a ticket that names a host you have not registered is not skipped for that reason alone. * In **default** mode (inbound override off), a skip is dropped silently: no comment is posted, and the ticket's status and assignment are left untouched. * In **override** mode, a skip may **escalate** — that is, reassign the ticket to another assignment group — if the inbound prompt instructs it. A skip's only possible side effect on the ticket is that reassignment: it still never posts a comment and never changes the ticket's status. * **Take-over happens only when work starts.** A ticket's status is flipped to in-progress only once a task is actually created. Tickets that are filtered out or that produce no match keep their original status and owner. * **Strict host matching.** The @2501 mention is treated as the bot being addressed, never as a host name, and a host whose name merely contains "2501" is not matched because of it. If a ticket names a target machine and no registered host matches that name, the gateway returns no match rather than substituting a plausible host. * **Genuine no-match.** When a ticket is in scope but no task can be created, the ticket receives a public comment explaining why and is returned to the queue (for ServiceNow incidents, status is reset to open) so a human can pick it up. ### Completion Behavior When a job finishes, the gateway writes the outcome back to the ticket. By default it applies a built-in mapping — for example, a successful incident is resolved with a close code, and a change request moves to Review (see [Change Requests](#change-requests)). Enable the [outbound prompt](#outbound-prompt) override to let the agent decide the write-back from your instructions instead: which status to set, whether to post a public comment or an internal work note, and whether to reassign the ticket to another assignment group. Because the gateway is the only writer to the ticket, the status, comment, and any reassignment are applied together as one update. ### Comment visibility Only outcomes that a requester should see are posted as public comments: the final success summary, a partial-result summary, and any "we are investigating" notice. Intermediate execution plans, per-task progress, and failure details are kept as private internal notes, so the requester sees the result rather than the play-by-play. ## Change Requests In addition to incidents, a ServiceNow gateway can execute **change requests**. Register a separate webhook with event type `change_request` for the change\_request table; a gateway that handles both incidents and changes uses two webhooks, one per table. **Eligibility.** 2501 acts on a change request only once it reaches the **Implement** state and is **Approved**. Changes in any other state, or not yet approved, are ignored. There is no restriction on change type: standard, normal, emergency, and custom change models are all handled the same way once approved and in Implement. The same assignment-group and environment filters that apply to incidents also apply to changes. **Execution follows the plan.** When 2501 picks up a change it adds a comment but does not move the change state (it is already in Implement). It treats the change's **Implementation Plan** as the authoritative, pre-approved action list and executes it faithfully, without re-diagnosing, substituting alternative approaches, or expanding scope. The **Backout Plan** is used only to roll back if a step fails, and the **Test Plan** defines the verification steps run after implementation. **Completion.** * On full success, 2501 moves the change from Implement to **Review** with close code `successful`, recording the technical actions performed in the close notes. * On partial success, it moves the change to **Review** with close code `successful_issues`, with the actions in the close notes. * On failure (or if no action could be taken), it leaves the change in **Implement** and posts a work note describing the outcome. It does not assign a close code or move the change, so a human can review and retry within the change window. Humans retain ownership of the final Review to Closed transition. Commenting @2501 on a change request posts a comment but does not reopen or change the change's state. Change requests are exempt from [deduplication](#ticket-deduplication): because change records are created intentionally, each is always processed and is never linked as a duplicate. ## Escalation Groups When a ServiceNow ticket finishes in a state that needs human follow-up, 2501 can hand it off to a designated **escalation group** so it doesn't sit orphaned. Escalation is opt-in per gateway. Set `servicenow_escalation_group_id` in the gateway metadata to the `sys_user_group` sys\_id you want unresolved tickets reassigned to. Without it, the gateway never escalates. With an escalation group configured, the resolution determines the handoff: | Resolution | Incidents | Change Requests | | ------------------------ | ------------- | -------------------------------------------------------------------- | | Success | No escalation | No escalation | | Partial | **Escalate** | No escalation (handed to Review with close code `successful_issues`) | | Agentic failure | **Escalate** | **Escalate** | | Hard failure | **Escalate** | **Escalate** | | No action | **Escalate** | **Escalate** | | Unknown / stuck workflow | **Escalate** | **Escalate** | A change request that ends partial is a deliberate Review hand-off, not a failure, so it is not escalated. Incidents that end partial still need a human and are escalated. ## Runner Gateway A separate **runner** gateway exists for benchmarking and scenario runs. It accepts tickets directly from the `2501 runner` CLI and dispatches them as jobs through the same pipeline, but it does not talk to any ticketing system. Use it so scenario runs never touch your ServiceNow instance and so you don't need a ServiceNow developer account just to exercise agents end-to-end. The runner gateway is not used in normal production. ## Ticket Deduplication It is common for the same issue to generate multiple tickets. Several people may report the same problem independently, or a monitoring system may fire multiple alerts for the same incident. Without deduplication, each ticket would spawn its own job, wasting agent time on work that is already in progress. When a new **incident** arrives, 2501 uses an LLM (not an embedding or keyword search) to compare it against recent tickets and decide whether it describes the same problem on the same target. If a duplicate is found, no job is created: the duplicate ticket is linked to the original and automatically resolved when the original's job completes. Deduplication applies to incidents only; change requests are always processed. Two tickets are considered duplicates when they concern the **same issue on the same target** (e.g., two disk-space alerts for `/var/log` on the same host) or share a **provable root cause** (e.g., two services failing due to the same dependency). Same host but different resources (e.g., `/var/log` vs `/tmp`) are not duplicates. If a duplicate ticket contains new details, those details are not merged into the running job and no follow-up job is created: the original job runs on the original ticket's content. To act on the new details, comment `@2501 unlink-ticket` on the duplicate. This detaches it from the original and allows a new job to be created for it. ### Configuration Set `dedup_window_minutes` in the gateway metadata to control how far back the engine looks when comparing tickets. It defaults to `30` minutes and applies to both still-open tickets and recently completed ones considered for comparison. Set it to `-1` to disable deduplication. ## Working with Active Jobs After a gateway creates a job from a ticket, you can interact with it by commenting @2501 in the ticket. The system responds differently based on the job's current state: | Job Status | @2501 Comment Action | What Happens | | --------------- | -------------------------------------- | --------------------------------------------------------------------------- | | **IN PROGRESS** | Restarts the job with new instructions | Cancels incomplete tasks, keeps completed tasks, generates a new plan | | **COMPLETED** | Reopens the job | Resets the job to pending and re-runs it; completed tasks remain as context | | **FAILED** | Reopens the job | Resets the job to pending and re-runs it; earlier tasks remain as context | **Key concept:** The comment always acts on the same job. While the job is running it restarts with the new requirements; once the job is finished (completed or failed), the comment reopens it and re-runs it, with the earlier tasks kept as context. ### Reopening Finished Jobs **When to use:** After a job completes or fails, you need additional work that builds on what was done. **What happens:** 1. **Same job reopened** - Your @2501 comment reopens the existing job and resets it to pending so it runs again 2. **Plan and resolution cleared** - The previous plan and resolution are cleared, and a fresh plan is generated from the ticket plus your new comment 3. **Earlier tasks kept as context** - Tasks from earlier runs remain in the database and are visible to the new plan, so completed work is not redone 4. **Re-runs with context** - Agents start the new work but can reference the earlier actions and outcomes ### Updating Active Jobs **When to use:** A job is currently running and you need to add requirements or change direction mid-execution. **What happens:** 1. **Incomplete tasks cancelled** - Any tasks that have not finished are stopped 2. **Completed tasks remain** - Already-finished work stays as context and is not redone 3. **New plan generated** - The system creates a fresh execution plan incorporating your @2501 comment 4. **Job resumes** - Agents continue with the updated requirements, building on completed work **Important note:** Use this when you genuinely need to change direction or add requirements. For simple clarifications or questions, waiting for the agent to ask may be more efficient than restarting the entire job. **Common scenarios:** * Investigation reveals additional areas to check * Requirements expanded during execution * Different approach needed mid-task * Priority shifted to a different aspect of the problem **Quick rule:** Is the job still running? Your comment restarts the job with the new requirements. Is the job done (completed or failed)? Your comment reopens the same job and re-runs it. # Hosts Source: https://docs.2501.ai/0.12/core-concepts/hosts Your machines where agents act autonomously Hosts represent the target systems where agents execute tasks. Each host defines the network location and connection details for a machine in your infrastructure. ## What is a Host? A host is a registered target system that agents can operate on. Hosts provide network addressing through IP addresses, connection methods via SSH or WinRM protocols, organizational context for grouping and management, and discovery metadata for filtering and targeting. Hosts list ## Host Configuration Host configuration form ### Network Addressing Each host requires at least one IP address: **Public IP:** The externally accessible IP address. Used when the agent reaches cloud instances, accesses systems across different networks, or operates on internet-facing infrastructure. **Private IP:** The internal network IP address. Used when operating within the same network or VPC, accessing systems behind firewalls or NAT, or connecting through VPN or private network tunnels. At least one IP address is mandatory. Providing both creates a fallback strategy if one isn't valid when an agent attempts to execute a task. ### Connection Protocols **SSH (Linux/Unix):** Standard protocol for Linux, Unix, and macOS systems. Requires SSH credentials (username plus password or key-based authentication), uses default port 22, and is used for most infrastructure operations. **WinRM (Windows):** Windows Remote Management protocol for Windows systems. Requires Windows credentials, supports both HTTP and HTTPS connections, and enables PowerShell remote execution. Windows hosts can authenticate with static Windows credentials or with a gMSA over Kerberos (see [Credentials](/0.12/configure/credentials#windows-authentication-with-gmsa)). ### Connection Options The create and edit forms expose a few transport settings: **Port:** Each protocol has a sensible default (22 for SSH, 5985 for WinRM). Override it only when your target listens on a non-standard port, for example set 5986 (or 443/8443) when WinRM is served over HTTPS. **Skip TLS verification (WinRM over HTTPS):** Accepts self-signed certificates on the target. Leave this off for CA-signed targets so the certificate chain is validated. Turn it on only when you knowingly connect to a host presenting a self-signed certificate. ### Jump host (SSH bastion) For targets reachable only through a bastion, register the **bastion as its own host** with **Is jump host** enabled, then point each target host at it via **Jump host**. Single-hop only — a jump host itself cannot route through another jump host. | Field on the target host | What it does | | ------------------------ | ------------------------------------------------------------------ | | **Jump host** | Which registered host to relay through. Empty = direct connection. | | Field on the bastion host | What it does | | ------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Is jump host** | Marks the host as a relay. Requires `target_type=ssh`. | | **Jump host user** | Credential holding the SSH **username** used to log in to the bastion. | | **Jump host credential** | Credential holding the **password or PEM private key** for that login. | | **Eligible subnets** (optional) | IPv4 CIDRs whose hosts are reasonable to route through this bastion. UX-only — used to suggest default routing in Command Center; the executor never reads it. | The engine refuses to dial a target whose jump host is missing or no longer flagged `is_jump_host`, so the constraint is enforced end-to-end. ### Central agents (orchestration hosts) Some agents don't operate on a single machine — they operate on **many machines from one machine**. The typical pattern: a single host with `kubectl` configured against the cluster, the AWS CLI authenticated to your account, the `govc` VMware client, or similar fleet-wide tooling. The agent logs into that one machine and drives the others through those CLIs. Register the orchestration machine as a normal host (SSH usually), attach an agent to it with the relevant specialty (`Kubernetes Operator`, `AWS Central`, `VMware Operator`), and route tickets affecting the fleet to it via the gateway. The agent reaches the actual targets through the tools installed there — you don't need to register every cluster node, every EC2 instance, or every ESXi VM as a 2501 host. Common shapes: * A **management host** with `kubectl` + a kubeconfig — handles every Kubernetes ticket for the cluster * A **CLI bastion** with AWS / GCP / Azure CLIs authenticated to the account — handles cloud-resource tickets * A **vSphere control node** with `govc` or the [VMware MCP](/0.12/configure/plugins) — handles hypervisor and snapshot tickets The host count stays small (one per orchestration surface), and the agent's specialty + operational rules carry all the context about *which* downstream resources are in scope. ### Naming and Identification Hosts have two separate labeling concepts. **Additional Names** are free-form aliases. **Tags** are a structured, closed vocabulary that scopes which operational rules an agent receives. They are described in detail below. **Host Name:** The primary identifier for the host, typically matching the machine's hostname or a descriptive name. Examples: `prod-web-01`, `staging-db-primary`, `dev-workstation` **Additional Names:** Optional free-form aliases that help you identify or match a host (for example `web-01`, `app-server-01`). They are purely for recognition and search. They do not influence which operational rules an agent receives. ## Host Tags Tags are a structured, closed vocabulary you assign to a host. Unlike Additional Names, they are not free text: you pick from a fixed set of axes, and each tag carries a recognizable icon. Tags scope which [Operational Rules](/0.12/configure/operational-rules) an agent receives when it works on the host. The host tag picker is organized into axes: | Axis | Selection | Examples | | ------------------------------- | ----------- | ------------------------- | | **OS** | Pick one | Linux, Windows, ESXi | | **Shell** | Pick one | POSIX, non-POSIX | | **Type** (what the host is for) | One or more | database, web, hypervisor | | **Technologies** (what it runs) | One or more | postgres, nginx, tomcat | OS and Shell are single-select. Type and Technologies accept multiple values. Technologies are unversioned. Record specific versions (for example the exact PostgreSQL or Tomcat version) in the host's **Description** field instead. ### Setting Tags Tags are assigned from a dedicated axis-based picker in the **TAGGING** section of a host's detail page. The quick create and edit dialogs do not include the tag picker, so open the host's detail page to set or change tags. In **Command Center** select the host to open its detail page. Find the **TAGGING** section and use the picker to choose values along each axis. Select one OS and one Shell, then add any Type and Technologies values that apply. ### How Tags Scope Operational Rules Host tags drive operational-rule matching. A rule reaches a host only when the host's tags satisfy **every** scope tag the rule carries. Adding more scope tags to a rule narrows where it applies; a rule with no scope tags applies broadly. For example, a rule scoped to `postgres` reaches only hosts tagged with the `postgres` technology. A rule scoped to both `linux` and `database` reaches only hosts that carry both tags. Procedure and verb intent (what the agent should do) is matched from the ticket, not from host tags. Procedure tags cannot be set on a host. See [Operational Rules](/0.12/configure/operational-rules) for how scoping works end to end. ## Host Knowledge Two parts of a host feed agents context about that machine before they start working. **Knowledge:** The host detail page has a read-only **KNOWLEDGE** section. It shows facts automatically extracted from documents you upload, resolved to this host and grouped by source document. These facts are surfaced to agents as context when they work on the host. See [Knowledge](/0.12/configure/knowledge) for how documents are ingested and turned into facts. **Description:** The free-text Description field is your place for operator notes (installed tools, quirks, where credentials live, exact software versions). Its content is also surfaced to agents as context, so keep it accurate and current. ## Host Management ### Creating and Editing Hosts 1. Go to **Command Center** → **Hosts** (sidebar) 2. Click **Create Host** to open the create form (click an existing host to open its detail page and edit it) 3. Configure host name, public and/or private IP, connection protocol, port, and additional names Set structured tags from the **TAGGING** section of the host's detail page after the host exists. Hosts are always scoped to your current organization. For bulk onboarding, see [Import and Export](/0.12/configure/import-export) or manage hosts declaratively with [Configuration as Code](/0.12/configure/configuration-as-code). Create Host ### Subnet Grouping The Command Center interface organizes hosts by subnet for better visibility and management. Hosts are automatically grouped using the first three octets of their private IP address (e.g., `192.168.1.x`). This enables quick identification of hosts within the same network segment, visual topology understanding, easier management of network-scoped operations, and identification of network configuration issues. ### Deleting Hosts Deleting a host permanently removes the host and archives all of its agents and their tasks. A confirmation dialog in the Command Center shows how many agents will be affected before you proceed. Use this to clean up decommissioned machines without leaving orphaned agents behind. ### Task Queuing Tasks targeting the same host run one at a time. Additional tasks queue automatically and start when the current one finishes, preventing conflicts from concurrent changes on the same system. ## Connection Priority When both public and private IPs are configured, the agent picks the right one based on network accessibility, performance (private IPs often have lower latency), and security (private networks are preferred for sensitive operations). The agent automatically selects the appropriate IP address without manual intervention. ## Credential Assignment [Agents](/0.12/core-concepts/agents) must have appropriate [credentials](/0.12/configure/credentials) assigned to reach their host: SSH username + password or key for Linux/Unix, WinRM credentials for Windows, plus the host addresses. ## Troubleshooting **Connection Failures:** Verify IP address accessibility from where the agent runs. Check firewall rules allow SSH (port 22) or WinRM (ports 5985/5986). Validate credentials are correctly assigned. Use the **Test Connection** button on the agent's row to diagnose connectivity (see [Agents: Testing Connectivity](/0.12/core-concepts/agents#testing-connectivity)). **Host Not Appearing in Command Center:** Confirm the import or creation succeeded, and that you're viewing the right [organization](/0.12/core-concepts/organizations) in the picker. **Incorrect Subnet Grouping:** Verify IP address format and accuracy. Update host details if IP addresses have changed. Check for VPN or proxy interference in IP detection. **Agent Can't Reach Host:** Confirm both public and private IPs if operating across networks. Verify routing between agent execution environment and target host. Test connectivity using ping or direct SSH/WinRM attempts. Review network security groups or firewall rules. For detailed credential configuration, see [Credentials](/0.12/configure/credentials). For agent execution modes, see [Agents](/0.12/core-concepts/agents). # Job Schedules Source: https://docs.2501.ai/0.12/core-concepts/job-schedules Recurring jobs that fan out into a fresh Job each tick A **Job Schedule** registers a recurring cadence. When a ticket asks for work on a repeating schedule — *"check disk space every Monday at 09:00"*, *"restart the worker every weekday at 07:30"*, *"every 15 minutes"* — 2501 doesn't run the job once; it registers a schedule. Each tick spawns a fresh [Job](/0.12/core-concepts/jobs) that follows the normal lifecycle. Schedules are managed under **Job Schedules** in Command Center. Job Schedules ## How a schedule is created Schedules are **registered automatically** when an incoming ticket describes a recurring action. The gateway detects the cadence in the ticket text and creates the schedule instead of a one-off job. You don't create schedules manually in Command Center. The action of "make this recurring" comes from the ticket itself — usually a comment or a description like *"run this every Monday morning"*. A single cadence can also name a **set of days**, such as *"every weekday at 07:30"*, *"every Monday and Wednesday at 10:00"*, or *"every weekend at 11:00"*. It registers as one schedule that runs on each of those days, not one schedule per day. The phrase still needs a recurring word like *"every"*: naming days alone (*"on Monday and Wednesday at 10:00"*) reads as one-off work, not a schedule. ## What you see for each schedule | Field | What it shows | | ----------------- | ---------------------------------------------------------------------------- | | **Cadence** | The interpreted schedule (e.g. *"weekly, Mondays at 09:00 UTC"*) | | **Source ticket** | The original ITSM record that introduced the schedule | | **Next run** | When the next Job will spawn | | **Run history** | The Jobs that have fired from this schedule so far, each with its own status | | **Status** | `active` or `cancelled` | ## Managing a schedule * **Pause / stop** a schedule from its detail page. Stopping it cancels future ticks but does not affect Jobs already in flight. * **Re-activate** is not currently surfaced — to "restart" a schedule, comment on the source ticket and let the gateway re-register. * **Edit cadence** is also driven from the source ticket; comment on the ticket to change the schedule and the gateway will reinterpret. ## Interval flooring Very fast intervals are **floored to 5 minutes in production**. Asking for "every minute" registers as "every 5 minutes". This protects the agent fleet from runaway schedules and matches what realistic remediation cadences look like. ## License interaction Scheduled jobs count against the **same task cap** as ad-hoc jobs. The cap is a lifetime tenant total — see [Licensing](/0.12/configure/licensing). If the tenant has reached its cap when a tick fires, that tick **skips** rather than queues. The schedule itself stays active; the next tick will try again once headroom exists. # Jobs Source: https://docs.2501.ai/0.12/core-concepts/jobs Coordinate multiple tasks across agents A job coordinates multiple related tasks across one or more agents. Jobs enable complex operations that require multiple agents (different specialists handling different aspects), sequential execution (tasks that depend on prior completion), parallel operations (simultaneous execution across multiple hosts), or complex workflows (gateway-generated work split into coordinated subtasks). Jobs list ## Organization Scoping Jobs are scoped to a specific [organization](/0.12/core-concepts/organizations). All tasks within a job belong to the same organization, and only agents within that organization can be assigned to the job's tasks. ## Job Creation Jobs are created by [Gateways](/0.12/core-concepts/gateways) when integrated systems generate work requiring multi-agent coordination (e.g., one agent provisions infrastructure, another deploys applications), host fleet operations (e.g., rolling updates across multiple servers), or dependency chains (e.g., database migration must complete before application restart). Recurring schedules also create jobs: each scheduled run spawns a fresh job that follows the normal lifecycle. See [Job Schedules](/0.12/core-concepts/job-schedules) for how schedules are registered, paused, and managed. ## Job Orchestration When a gateway creates a job, it decomposes the task by splitting complex work into individual tasks. Each task is assigned to the appropriate agent based on specialty alignment, host accessibility and credentials, and agent availability and workload. Tasks execute according to dependencies, where sequential tasks wait for predecessors to complete while parallel tasks execute simultaneously where possible. Job status reflects aggregate task states. ## Investigate vs Remediate Jobs inherit their execution mode from the ticket tags (`@2501:investigate` / `@2501:remediate`), defaulting to remediate. Individual tasks within the job may be further constrained by their assigned agents' [specialties](/0.12/configure/specialties). See [Agents, Investigate vs Remediate](/0.12/core-concepts/agents#execution-modes-investigate-vs-remediate) for the full priority chain. When a ticket requested remediation but investigate-only specialties prevented it, the job resolution is flagged as **partial** rather than a failure. ## Job Monitoring Jobs are visible only through the Command Center UI, showing job overview (status), the list of tasks within the job with their individual states, and links to the originating ticket and any recurring schedule. ## Task-Job Relationship When viewing a task in Command Center, tasks linked to a job display the job identifier. Navigate to the parent job to see all related tasks and understand the broader context within the workflow. This visibility helps troubleshoot multi-task operations by understanding which tasks succeeded, failed, or are pending. ## Troubleshooting **Job Tasks Not Coordinating:** Review job configuration and task dependencies. Check agent availability and workload. Verify each agent has appropriate credentials and access. Examine individual task failures that may be blocking downstream tasks. For advanced task orchestration through integrations, see [Gateways](/0.12/core-concepts/gateways). For agent configuration affecting task execution, see [Agents](/0.12/core-concepts/agents), [Specialties](/0.12/configure/specialties), and [Operational Rules](/0.12/configure/operational-rules). # Organizations Source: https://docs.2501.ai/0.12/core-concepts/organizations Organize your tenant by separating between organizations Organizations provide logical separation of infrastructure within your 2501 account. They enable multi-tenancy, allowing you to partition resources, agents, and operations based on your operational structure. ## What is an Organization? An organization is a scoping boundary that groups related infrastructure and operations. All operational resources are associated with an organization, providing isolation and access control within your account. ## Organization Structure Your tenant is the isolation boundary: nothing is ever visible across tenants, whatever a user's role or scope. Inside the tenant, every resource is scoped in exactly one of three ways. ### Organization-scoped resources These resources always belong to one organization and cannot be made tenant-wide: * **[Hosts](/0.12/core-concepts/hosts)**: target systems, including their gMSA configuration * **[Agents](/0.12/core-concepts/agents)**: each agent belongs to its host's organization * **[Tasks](/0.12/core-concepts/tasks)** and **[Jobs](/0.12/core-concepts/jobs)**: execution history * **[Job Schedules](/0.12/core-concepts/job-schedules)**: recurring and deferred work * **[Tickets](/0.12/core-concepts/tickets)** and their comments * **Chats** and their messages * **[Gateways](/0.12/core-concepts/gateways)**: ticket routing integrations * **[Webhooks](/0.12/configure/webhooks)**: outbound notifications * **[Network Discovery](/0.12/configure/discovery)**: scans and the nodes they find * **[Knowledge](/0.12/configure/knowledge)**: uploaded documents and the host facts extracted from them ### Shared resources These resources can be scoped to a specific organization or left unscoped, which makes them available across all organizations in your tenant: * **[Credentials](/0.12/configure/credentials)**: authentication secrets * **[Specialties](/0.12/configure/specialties)**: agent domain configurations * **[Operational Rules](/0.12/configure/operational-rules)**: mandatory procedures * **[Blacklists](/0.12/configure/blacklist)**: prohibited commands * **[Plugins](/0.12/configure/plugins)**: MCP integrations and tools * **[Providers](/0.12/configure/providers)**: LLM endpoints, tenant-wide unless **Scoped to current organization** is ticked when creating one. [Models](/0.12/configure/models) belong to a provider and take its scope: an organization-scoped provider's models are available only to that organization * **API keys**: scope is chosen when the key is created and cannot be changed afterwards * **Verifier exceptions**: one tenant default plus one override per organization, managed by administrators When a shared resource is organization-scoped, only agents and operations within that organization can use it. When unscoped, it is available to every organization. An organization can also hold a credential with the same name as a tenant-wide one: the organization's own is used first and the tenant-wide one is the fallback, so a shared default can be overridden per organization. Creating or editing a tenant-wide shared resource requires [tenant-level access](/0.12/configure/users). Users restricted to specific organizations can read tenant-wide resources but only write within their own organizations. Tenant-wide providers and the model catalog are further limited to tenant-level administrators. Verifier exceptions are the one exception to the read rule: an organization-restricted administrator sees and edits only their organization's override, not the tenant default it replaces. ### Tenant-level resources These have no organization at all - they describe the tenant itself: * **Organizations** and **[Users](/0.12/configure/users)**, including which organizations each user belongs to * **[Licensing](/0.12/configure/licensing)**: plan and usage limits ## Creating Organizations Organizations can be created from **Command Center → Settings → Organizations**. Click **New Organization**, enter a name, and save. Previously this required the CLI or direct database access. The CLI path is still available via `2501 infra`. See [Users & Organizations](/0.12/deployment/users-organizations) for the full setup workflow. # Tasks Source: https://docs.2501.ai/0.12/core-concepts/tasks Assign complex tasks to your agents in various ways Tasks are the fundamental work units in 2501: the instructions you assign to agents for execution. [Jobs](/0.12/core-concepts/jobs) coordinate multiple related tasks, enabling complex multi-agent or multi-step operations. ## What is a Task? A task is a discrete assignment given to an agent, ranging from simple operations ("check disk space on prod-web-01") to complex multi-step workflows ("deploy the latest application version and verify health checks"). Tasks represent the primary interface for directing agent behavior. ## Creating Tasks **Command Center UI:** Open the agent, create a new task in the task box, and watch execution stream in real time. **[Gateways](/0.12/core-concepts/gateways):** If gateways are enabled, tasks and jobs are created automatically from ServiceNow (the default gateway) or any custom-built gateway integrated for your tenant. Gateway-created tasks may be assigned directly to a specific agent or routed as [jobs](/0.12/core-concepts/jobs) for multi-agent coordination. Interact with gateway jobs by commenting @2501 in the ticket — see [Working with Active Jobs](/0.12/core-concepts/gateways#working-with-active-jobs). Creating Task ## Task Lifecycle **User Assigned:** The task has been created and is awaiting agent pickup. With manual trigger, the task waits for explicit CLI execution. With automatic pickup, the agent continuously listens and processes tasks as they arrive. **Planning Phase:** The agent's secondary engine analyzes the task, breaks down requirements into actionable steps, gathers necessary context from the target system, identifies required credentials and tools, and exposes the execution plan for review. This phase provides visibility into the agent's intended approach before commands are executed. **In Progress:** The agent actively executes the task by running commands on target hosts, reading and modifying files, interacting with services and APIs, adapting based on outputs and errors, and logging all actions and decisions. **Completion States:** *Completed:* Task finished successfully. Objective achieved, all planned steps executed, results validated, and context retained for follow-up tasks. *Failed:* Task could not be completed due to errors during execution, operational constraints violations, unavailable resources, or agent escalation for human intervention. Tasks list ## Task Queuing Tasks targeting the same host run one at a time. If multiple tasks are submitted to agents on the same host, additional tasks queue automatically and start when the current one finishes. This prevents conflicts from concurrent changes on the same system. ## Task Limits Your tenant's [license](/0.12/configure/licensing) sets a tenant-wide cap on the total number of tasks (ad-hoc and recurring, lifetime). Without a license, a built-in free-tier cap applies instead. Task creation is blocked when the cap is reached or when the license has expired. See [Licensing & Usage](/0.12/configure/licensing) to view your caps and current usage. ## Task Summaries When a task reaches a terminal state (completed or failed), 2501 generates a summary using the secondary engine. Summaries are concise (under 300 words) and include the task description, actions taken, results, and any resources manipulated or verified. Summaries include verbatim command output when the output is the meaningful result (e.g., `ps aux`, `df -h`, `kubectl get pods`), preserving exact formatting rather than paraphrasing. For gateway-created tasks, the summary is posted as a comment on the originating ticket, visible to all users. ## Command Timeouts Commands that take too long are handled gracefully. The agent receives a clean timeout message and retries with an adjusted approach rather than failing the entire task immediately. ## Task Monitoring Tasks are monitored from Command Center. The task detail page streams agent messages, commands, and output in real time as the agent writes them — no need to refresh, and no need to wait for a status change to see what's happening. The same page shows the execution timeline, command outputs, errors, and the agent's reasoning trail. Task Finished ## Task Management **Stopping Tasks:** Tasks can be stopped at any time during execution. This immediately halts agent operations, preserves execution history up to the stop point, marks the task as canceled, and retains context for potential resumption. **Task Context and Follow-up:** Completed or failed tasks remain in the agent's context window, enabling follow-up assignments that reference previous work, iteration on failures with adjusted approaches, and multi-step workflows that build on completed tasks. Example: If a deployment task fails during health checks, assign a follow-up task: "investigate why the health check failed" without needing to re-explain the deployment context. **Task Archiving:** When an agent's context window approaches capacity, archive completed tasks. This removes task history from agent memory, frees context for new operations, keeps archived tasks viewable in Command Center for auditing, but makes context unavailable for follow-up. See [Agents - Memory Management](/0.12/core-concepts/agents#memory-management) for archiving strategies. ## Troubleshooting **Task Not Starting:** Verify agent is running and listening for tasks. Check agent has necessary credentials for the target host. Ensure no [operational rules](/0.12/configure/operational-rules) or [blacklists](/0.12/configure/blacklist) blocking execution. Confirm task is assigned to the correct [organization](/0.12/core-concepts/organizations). **Task Failing Repeatedly:** Review execution logs to identify failure points. Verify host accessibility and credentials. Check if operational rules or blacklists are too restrictive. Consider adjusting agent [specialty](/0.12/configure/specialties) or providing more context. Try stopping and following up with refined instructions. For advanced task orchestration through integrations, see [Gateways](/0.12/core-concepts/gateways) and [Jobs](/0.12/core-concepts/jobs). For agent configuration affecting task execution, see [Agents](/0.12/core-concepts/agents), [Specialties](/0.12/configure/specialties), and [Operational Rules](/0.12/configure/operational-rules). # Tickets Source: https://docs.2501.ai/0.12/core-concepts/tickets Key data extracted from your ITSM, translated for 2501's pipeline A **Ticket** is 2501's mirror of an incident or change request in your ITSM. It is not a copy of every field — only what the gateway needs to route, dispatch, and report back: title, description, comments, attachments, assignment group, requester, environment, priority. Once a ticket exists in 2501, the [gateway](/0.12/core-concepts/gateways) decides whether it is in scope. In-scope tickets become a [Job](/0.12/core-concepts/jobs) made of one or more [Tasks](/0.12/core-concepts/tasks). Out-of-scope tickets are dropped silently. ## What's in a ticket record | Field | Source | | ----------------------------- | ---------------------------------------------------- | | Title, body, priority | ITSM record | | Comments + attachments | ITSM record, parsed by the tenant's multimodal model | | Assignment group, environment | ITSM record — used as routing filters | | Requester / caller | ITSM record | | Status mirror | Updated as the job progresses | The Tickets page in Command Center shows every ticket that reached 2501, including those that were deduplicated, dropped, or escalated. ## Ticket → job → task ``` Ticket (ITSM record mirror) │ ▼ gateway parses and routes Job (the unit of work for this ticket) │ ├─▶ Task 1 (agent A on host X) ├─▶ Task 2 (agent B on host Y) └─▶ Task N ... ``` A job is at most 5 tasks. Tasks within a job may run in sequence or parallel depending on host targets. See [Agentic Flow](/0.12/understand/agentic-flow) for the lifecycle and [Jobs](/0.12/core-concepts/jobs) for orchestration details. ## Visibility back to the ITSM The gateway posts back at meaningful boundaries — never the full play-by-play: * A **public comment** with the final outcome (success / partial / no-action explanation) * **Internal notes** for intermediate plans, per-task progress, and failure details — visible to operators in the ITSM, not to the requester This keeps the requester's view focused on the result while preserving the audit trail for your team. See [Gateways → Comment visibility](/0.12/core-concepts/gateways#comment-visibility). ## Tags that change behavior A few keywords in the ticket body or a comment change how the agent runs: | Tag | Effect | | --------------------- | ------------------------------------------------------------------------------------------------ | | `@2501:investigate` | Force read-only mode for this ticket. Aliases: `@2501:investigation` | | `@2501:remediate` | Force remediate mode. Aliases: `@2501:remediation` | | `@2501 ` | Once a job exists, restarts (in progress) or reopens (completed/failed) it with new instructions | | `@2501 unlink-ticket` | Detach a dedup-linked ticket so it gets its own job | See [Agents → Execution Modes](/0.12/core-concepts/agents#execution-modes-investigate-vs-remediate) and [Gateways → Working with Active Jobs](/0.12/core-concepts/gateways#working-with-active-jobs). ## Reopening a finished ticket Once a ticket's job has finished, two things bring the agent back: * **An `@2501` comment** — an explicit re-request with new instructions. * **Reopening the ticket in the ITSM** — moving a ServiceNow incident back to **New** or **In Progress** (or updating it while it sits in one of those states) after the job finished. No mention needed: the platform notices that a person — not its own integration account — touched a ticket it considered done, and re-engages. On a state-driven reopen with no new instructions, the agent first **re-verifies the original symptom** before deciding whether more work is needed. Work completed before the reopen is treated as belonging to the previous run — it can't satisfy the reopened request on its own — while the agent still uses it as context. If verification shows the issue is genuinely resolved, the agent closes the ticket out again with an explanatory comment. Updates made by the platform's own ServiceNow account never trigger a reopen, so the agent's closing comments and status changes don't re-trigger itself. # Configuration Source: https://docs.2501.ai/0.12/deployment/configuration How 2501 deployment configuration works 2501 is configured defaults-first. Every default (registry, ports, resources, Swarm and health settings) is embedded in the CLI binary, so you only specify what differs from those defaults. ## Configuration files | File | Purpose | Required? | | -------------------- | --------------------------------------------------------- | --------- | | `env.engine` | Environment variables for the 2501 Engine | Yes | | `env.command-center` | Environment variables for the Command Center UI | Yes | | `2501-infra.yml` | Optional overlay for values that differ from the defaults | No | A plain Docker Swarm deployment with an external database needs no `2501-infra.yml` at all, just the two environment files. The CLI fills in everything else from its embedded defaults. ## The 2501-infra.yml overlay `2501-infra.yml` is optional and deep-merged on top of the embedded defaults, so it only needs to hold the values you actually want to change. The CLI writes one for you when your deployment is non-default, such as a Kubernetes target, managed PostgreSQL, a stored registry key, or a pinned image tag. To see exactly what the CLI will apply, run `2501 infra config` for the effective merged configuration, or `2501 infra config --defaults` for just the embedded defaults. This is the quickest way to confirm a value before deploying. ## Environment files `env.engine` and `env.command-center` hold the environment variables for the Engine and the Command Center. The CLI generates, defaults, and copies everything it can, prompting only for human-required values such as LLM provider API keys and the external `DATABASE_URL`. Contact your Account Executive at 2501 for the list of supported LLM providers and their configuration. ## Applying configuration `2501 infra deploy` syncs your environment files to the current version's spec on every run, generating and defaulting what it can and prompting only for human-required values. There is no separate apply step. Workspace layout and resolution order Deploy to production with Swarm Deploy to a Kubernetes cluster Configure provider API keys # Docker Swarm Source: https://docs.2501.ai/0.12/deployment/docker-swarm Production deployment with Docker Swarm Docker Swarm is the default deployment target. For a Kubernetes cluster, see [Kubernetes](/0.12/deployment/kubernetes). ## Prerequisites ### 1. Initialize Docker Swarm On the manager node: ```bash theme={null} docker swarm init ``` ### 2. Join worker nodes (optional) On each worker node, run the join command provided by `docker swarm init`: ```bash theme={null} docker swarm join --token :2377 ``` ## Registry authentication Registry authentication is zero-touch. The CLI fetches the registry token itself on every deploy and hands fresh credentials to Swarm. You do not install a credential helper, run `docker login`, or manage expiring tokens by hand on any node. ## Deployment Deploy the stack: ```bash theme={null} 2501 infra deploy ``` For CI/CD pipelines, add `--yes`. In `--yes` mode the CLI auto-fixes everything it can, then aborts with an actionable list if a human-required value (such as an LLM API key or the external `DATABASE_URL`) is still missing. Run an interactive `2501 infra deploy` to upgrade: the CLI lists published releases newest-first and self-updates its own binary to match the version you pick before deploying. Back up PostgreSQL first, and see [Upgrading](/0.12/deployment/overview#upgrading) for the rollback path. For all deployment options, run `2501 infra deploy -h`. ## Post-deployment Verify the services are healthy: ```bash theme={null} curl http://localhost:1337/health # Engine curl http://localhost:3000/health # Command Center docker stack services 2501 ``` If you run `2501 infra deploy` without `--yes`, the CLI interactively prompts you to create your first tenant, organization, and admin user. In `--yes` mode, create them afterward with `2501 infra tenant create`, `2501 infra org create`, and `2501 infra user create` (use `-h` on each for details). # Feature Toggles Source: https://docs.2501.ai/0.12/deployment/feature-toggles Toggle experimental and optional behaviors on or off Feature toggles flip optional or experimental behaviors. Manage them with `2501 infra feature`. ```bash theme={null} 2501 infra feature list # show all toggles + state 2501 infra feature enable # alias: e 2501 infra feature disable # alias: d 2501 infra feature enable # interactive multi-select picker ``` ## Reference ### Integration | Toggle Key | Description | Default | | --------------------- | -------------------------------- | ---------- | | `enableElasticSearch` | Enable Elasticsearch integration | ❌ Disabled | | `enableDockerSwarm` | Enable Docker Swarm integration | ❌ Disabled | ### Security & Logging | Toggle Key | Description | Default | | ---------------------------- | -------------------------------- | ---------- | | `enableSecureLogs` | Enable secure logging | ❌ Disabled | | `enableCommandVerification` | Verify commands before execution | ❌ Disabled | | `enableLogLlmInputAndOutput` | Log LLM input and output | ❌ Disabled | ### Jobs | Toggle Key | Description | Default | | -------------------- | ---------------------------------------------------- | ---------- | | `enableFollowUpJobs` | Detect and create follow-up jobs after terminal jobs | ❌ Disabled | | `enableJobRestart` | Restart a running job when the user adds a comment | ❌ Disabled | ### Experimental | Toggle Key | Description | Default | | ------------------------------------ | --------------------------------------------- | ---------- | | `enableSlowInference` | Reduce parallel tasks/jobs for slow inference | ❌ Disabled | | `enableOperationalRulesVerification` | Verify operational rules at task time | ❌ Disabled | ## When to flip what * **`enableElasticSearch`** if you have an Elasticsearch instance and want the engine + Command Center to ship logs to it. * **`enableSecureLogs`** for tenants with strict log-encryption requirements. * **`enableSlowInference`** if your tenant runs on a self-hosted model with low throughput — reduces task concurrency to avoid queue starvation. * Anything labeled **experimental**: open a support ticket before enabling in production. # Init Command Source: https://docs.2501.ai/0.12/deployment/init-command Initialize your 2501 deployment workspace The `init` command creates your deployment workspace and auto-generates the required secrets. You normally do not run `init` by hand. The installer chains a non-interactive `init` automatically, so a fresh install already leaves you with a ready workspace. Run `init` manually only when you want to recreate or relocate a workspace. ## What gets created `init` lays out a workspace containing the two environment files you edit before deploying: * `env.engine` for the 2501 Engine * `env.command-center` for the Command Center UI It also generates and preserves the required secrets (credential keys, API key, auth secret). These are generated once and must never be regenerated after the initial deployment, as that would break existing credentials. An optional `2501-infra.yml` overlay is written only when your deployment differs from the embedded defaults (for example a Kubernetes target, managed PostgreSQL, a stored registry key, or a pinned image tag). See [Configuration](/0.12/deployment/configuration). ## Database mode The default is an external database: you supply `DATABASE_URL` in `env.engine` and the CLI does not manage a database for you. Pass `--postgres` to enable a managed PostgreSQL instance instead. Managed PostgreSQL is intended for experiments only, not production. For all available options, run `2501 infra init -h`. # Kubernetes Source: https://docs.2501.ai/0.12/deployment/kubernetes Run the 2501 engine on any Kubernetes cluster with the Kubernetes pod executor backend 2501 runs on any Kubernetes distribution — vanilla k8s, OpenShift, k3s, EKS, GKE, AKS, Minikube. The engine runs each tool executor in its own isolated sandbox. On Docker and Docker Swarm that sandbox is a container; on Kubernetes the engine creates a short-lived Kubernetes Pod instead — a pod is created when work starts, the engine attaches to it over stdin and stdout, and the pod is deleted when the task ends. This page covers how to generate the deployment manifests with the `2501` CLI, how the Kubernetes backend selects itself, and the RBAC the engine needs. Docker Swarm remains the default deployment target, with a fully managed `deploy` lifecycle. The Kubernetes target is generate-only: the CLI renders a complete manifest tree and you apply it with `kubectl` (or your distribution's equivalent CLI). This keeps you in control of how changes reach the cluster, including through GitOps tooling. ## Generating the manifests Run the deploy command with the Kubernetes target: ```bash theme={null} 2501 infra deploy --target kubernetes ``` The CLI prompts for a stack name, namespace, image tag, Command Center public URL, and your Elasticsearch and AWS (registry pull) values, then renders a ready-to-apply manifest tree under `/kubernetes/`: ``` kubernetes/ ├── namespace.yml ├── configmap/ # engine, command-center ├── secret/ # engine, command-center, postgres, ecr-pull-secret, license-public-key ├── deployment/ # engine, command-center, postgres ├── service/ # engine, command-center, postgres ├── pvc/ # postgres └── rbac/ # executor ServiceAccount, Role, RoleBinding ``` For non-interactive use, pass the values as flags: ```bash theme={null} 2501 infra deploy --target kubernetes --image-tag \ --stack-name ai-2501 --namespace ai-2501 --yes ``` `--image-tag` (alias `--tag`) is required with `--yes`. `--stack-name` and `--namespace` both default to `ai-2501`, and both are validated as DNS-1123 labels. ### Flags ```text 2501 infra deploy -h theme={null} Usage: 2501 infra deploy [options] Deploy 2501 stack Options: --remove Remove the deployed stack --stop Stop the deployed stack -t, --tag Image tag to deploy (overrides config, e.g. for testing a build) --target Deployment target: swarm or kubernetes (overrides config) --stack-name Kubernetes: stack name for Service names/labels (default: ai-2501) --namespace Kubernetes: target namespace (default: ai-2501) --image-tag Kubernetes: image tag to deploy (required for --target kubernetes; alias of --tag) -r, --restart [service] Restart with env reload (optional: engine, command-center) --history Show deployment history -d, --dir Config directory (default: /etc/2501) -y, --yes Non-interactive mode - skip prompts and seeding, fail on errors --cleanup-resources Clean up unused images and resources before deployment --cleanup-docker-resources Alias for --cleanup-resources (deprecated) --disable-services [services] Disable services (comma-separated: engine, command-center, or omit to disable all) -h, --help display help for command ``` `--remove`, `--stop`, `--restart`, `--history`, `--cleanup-resources`, and `--disable-services` drive the Swarm lifecycle; on the Kubernetes target the relevant options are `--target`, `--stack-name`, `--namespace`, `--image-tag`, `-d, --dir`, and `-y, --yes`. ## Filling in secrets Secret manifests use `stringData`, so every value is plain text and the cluster encodes it server-side on apply. Before applying, fill in any blank values directly in the YAML: * `AWS_*` and `ELASTICSEARCH_*` in `secret/engine.yml` and `secret/command-center.yml` * `BASE_URL` in `configmap/command-center.yml` (your real ingress hostname) * The license public key in `secret/license-public-key.yml`, unless a PEM file already exists at `/license.public.pem` Generation is safe to re-run. Auto-generated secrets (the PostgreSQL password, credential encryption key, engine API key, auth secret, and log encryption key) are created once and read back from the existing YAML on subsequent runs, and values you typed into the files by hand are preserved. An empty prompt answer never overwrites a previously supplied value. ## Applying ```bash theme={null} kubectl apply -f /kubernetes/ -R ``` (Substitute your distribution's CLI if you don't use `kubectl` — `oc`, `k3s kubectl`, etc.) The registry pull secret is a short-lived ECR token that expires after roughly 12 hours. Recreate it before each apply: ```bash theme={null} kubectl create secret docker-registry ecr-pull-secret \ --docker-server= --docker-username=AWS \ --docker-password=$(aws ecr get-login-password --region ) \ -n --dry-run=client -o yaml | kubectl apply -f - ``` ## Executor backend selection The engine picks an executor backend automatically: * When it detects that it is running **inside a Kubernetes cluster**, it uses the Kubernetes pod backend. * Otherwise it falls back to **Docker / Swarm**. You can force a backend with the `EXECUTOR_MCP_BACKEND` environment variable, set to `kube` or `docker`. The engine talks to the in-cluster Kubernetes API directly using the pod's auto-mounted service-account token. It does not shell out to `kubectl`, and you do not supply a kubeconfig. ## Required access (RBAC) The generated `rbac/executor.yml` provisions a dedicated `engine-executor` ServiceAccount with a namespace-scoped Role and RoleBinding granting: | Resource | Verbs | | ------------- | ------------------------------------------ | | `pods` | `create`, `get`, `list`, `watch`, `delete` | | `pods/attach` | `create`, `get` | | `pods/status` | `get` | | `pods/log` | `get` | Executor pods spawn in the same namespace as the engine. To use a separate sandbox namespace, change the RoleBinding's namespace and override `EXECUTOR_MCP_K8S_NAMESPACE` on the engine pod, which is otherwise set to the engine's own namespace via the Downward API. gMSA-bound Windows targets are not supported on the Kubernetes executor backend. They depend on a Docker-volume Kerberos sidecar that this backend does not provide. Use the Docker or Swarm backend for those targets. See [Docker Swarm](/0.12/deployment/docker-swarm). For the full set of engine and Command Center environment variables (database, security keys, LLM providers, optional integrations), see [Configuration](/0.12/deployment/configuration). # LLM Providers Source: https://docs.2501.ai/0.12/deployment/llm-providers Seed LLM providers at deploy time and manage them in Command Center Models and providers are managed per tenant in a catalog in Command Center. At deploy time, the installer gives you a head start by seeding that catalog from the API keys already present in your engine environment. After the initial seed, ongoing management moves to the UI. Contact your Account Executive at 2501 for the complete list of supported providers and models. ## Seeding at Deploy Time When you deploy, the installer detects provider API keys present in the engine environment and offers to seed matching catalog providers and models for you. Accept the prompt to create those providers and models in the tenant catalog automatically. After this initial seed, you manage providers and models in **Command Center** under **Settings** > **Providers** and **Settings** > **Models**. From there you can add providers, point at custom or Azure endpoints, enable or disable models, set pricing, and run performance tests, all without restarting the engine. See [Providers](/0.12/configure/providers) and [Models](/0.12/configure/models) for the full catalog reference. Existing model configurations are preserved on upgrade. Per-provider feature toggles have been removed; whether a provider or model is available is now controlled by its **Enabled** toggle in the catalog. ## Setting Tenant Models When creating a tenant, specify the default LLM and multimodal models: ```bash theme={null} 2501 infra tenant create \ --name "My Company" \ --llm-model "" \ --multimodal-model "" \ --timezone "Europe/Paris" ``` You can change these defaults later in **Command Center** under **Settings** > **Tenant**, choosing from the catalog's enabled models. # Overview Source: https://docs.2501.ai/0.12/deployment/overview Deploy 2501 on your infrastructure Deploy and configure 2501 on your infrastructure using the `2501 infra` CLI. ## Install the CLI Install the `2501 infra` CLI on the machine that will run the deployment. Your 2501 account team provides the installer. Installation is a single self-contained step: it places a checksum-verified static binary, sets up a sudo wrapper so `2501 infra deploy` runs privileged without you typing `sudo`, and runs a non-interactive setup so you finish with a ready-to-edit workspace. No Docker, AWS CLI, or registry login is required on the machine to install. Common installer options: `--target kubernetes` (instead of the default Docker Swarm), `--user` (install under your home directory with no sudo), and `--version ` (pin a specific version). ### Keeping the CLI current There is no separate update command. The CLI stays in lockstep with the deployed platform version: * **During deploy.** Interactive `2501 infra deploy` lists published releases newest-first. When you pick a version, the CLI self-updates its own binary to match (checksum-verified, then re-executed) before deploying. * **CLI only.** To refresh just the binary, re-run the installer. It is idempotent and never touches an existing workspace. ## Quick start After install, you typically only need to edit two environment files before deploying. Open `/etc/2501/env.engine` and set the human-required values, such as your LLM provider API keys and the external `DATABASE_URL`. Open `/etc/2501/env.command-center` and confirm its values. ```bash theme={null} 2501 infra deploy ``` `deploy` syncs your env files to the current version's spec, runs database migrations, and brings up the stack. Most settings are defaulted for you. A plain Docker Swarm deployment with an external database needs no configuration file at all, just the two environment files above. See [Configuration](/0.12/deployment/configuration) for when an optional `2501-infra.yml` is written. ## Upgrading Upgrading is the same `deploy` command at a newer version. Point your Kubernetes manifests and/or `2501-infra.yml` at the new image tag, depending on how you deployed, then apply. The Engine, the Command Center, and the executor images the Engine spawns all move to that tag together. Read the [release notes](https://www.2501.ai/changelog) for every version you are crossing before you start. Some releases add environment variables or carry breaking changes, which matters most when you are coming from an old version. Back up PostgreSQL first, schema and data both: ```bash theme={null} pg_dump "$DATABASE_URL" --format=custom --file=2501-preupgrade.dump ``` Restoring that dump is what makes a downgrade possible. An Elasticsearch snapshot is optional. Keep automated backups for disaster recovery too, not just one dump before an upgrade. This matters most when 2501 manages PostgreSQL for you, rather than connecting to a database you already run and already back up. Commands worth knowing around an upgrade: | Command | What it gives you | Targets | | ----------------------------- | --------------------------------------------------------------------------------------------- | -------------------- | | `2501 infra status` | Running services, with configured against deployed versions and a warning when they disagree. | Swarm and Kubernetes | | `2501 infra logs` | Service logs, for when something does not come back up. | Swarm and Kubernetes | | `2501 infra deploy --history` | The last 10 deployments, newest first. The current tag is your rollback target. | Swarm only | On Kubernetes the CLI generates manifests rather than applying them, so it does not record what reached the cluster. Take your rollback target from the image tag in your manifests or from your GitOps history instead. Once the new version is up, check both health endpoints at the address your containers are exposed on: ```bash theme={null} curl http://:/health curl http://:/health ``` The interactive version picker reads the release list from `https://2501-public.s3.eu-west-3.amazonaws.com/cli-2501/versions.json`. Allow egress to it if you want the picker; otherwise pass `--tag ` explicitly. ### Rolling back If the new version misbehaves, go back to the last one that worked and restore the data that went with it: * Redeploy the previous image tag. * Restore your PostgreSQL backup. If the failed upgrade already part-migrated the database, recreate it empty and restore into that. * Verify the stack is healthy before handing it back to users. * Contact 2501 with what went wrong and any details you still have, so we can debug it and ship a patch. A restore rewinds your data to the moment the backup was taken. ## Getting help The CLI includes built-in documentation for all commands: ```bash theme={null} # List all commands 2501 infra -h # Help for specific commands 2501 infra deploy -h 2501 infra config -h 2501 infra tenant -h 2501 infra org -h 2501 infra user -h ``` ## Deployment workflow 1. **Install** the `2501 infra` CLI (setup runs automatically) 2. **Edit** `env.engine` and `env.command-center` with your human-required values 3. **Deploy** with `2501 infra deploy` 4. **Create entities** (the CLI prompts you interactively; skipped with `--yes`) 5. **Access** the Command Center UI ## What's Next Initialize your deployment Overview of configuration files Deploy to production Common issues and solutions # Troubleshooting Source: https://docs.2501.ai/0.12/deployment/troubleshooting Troubleshooting guide and feature toggles reference for 2501 Common issues and solutions for 2501 deployments. *** ## Common Issues ### Registry Authentication Failed Registry authentication is handled by the CLI on every deploy, so there is no credential helper or `docker login` to fix. If a deploy reports an image pull or authentication failure, confirm the pull-only registry key from 2501 was supplied at install (or that the node's AWS credentials resolve), then re-run the deploy to fetch a fresh token. Run `2501 infra config` to inspect the effective registry configuration. ### Database Connection Failed ```bash theme={null} # Test connection psql "postgresql://:@:5432/2501" # Check DATABASE_URL format # postgresql://:@:/?schema=public ``` ### Services Not Starting ```bash theme={null} # View logs docker service logs -f 2501_engine docker service logs -f 2501_command-center # Check resources docker stats ``` ### Migration Failed ```bash theme={null} # Check database is accessible # Check DATABASE_URL and DIRECT_URL are correct # Retry deployment 2501 infra deploy ``` *** ## Viewing Logs ```bash theme={null} docker service logs -f 2501_engine docker service logs -f 2501_command-center # All services status docker stack services 2501 ``` ### Exporting logs for a bug report The Command Center's **Debug** page (admin-only) lets you browse structured logs and download them: an NDJSON export of whatever the current filters show, or a bundle with everything about one trace, task or job. Downloads are **anonymized by default**: IP addresses, hostnames, MAC addresses and emails are replaced with stable placeholders (`[ip-1]`, `[host-2]`, …). The same value always maps to the same placeholder within one file, so a connection chain or a repeated failure stays readable without revealing your infrastructure. Anonymized files carry an `-anon` suffix in their name. Turn the **Anonymize** toggle off to download the raw records instead - for example when the file stays inside your organization. *** ## Health Checks ```bash theme={null} curl http://localhost:1337/health # Engine curl http://localhost:3000/health # Command Center ``` *** ## Reset Everything This will delete all data. Only use this for development/testing environments. ```bash theme={null} # Stop and remove 2501 infra deploy --remove # Remove volumes (WARNING: deletes data) docker volume rm $(docker volume ls -q | grep 2501) # Redeploy 2501 infra deploy ``` *** ## Upgrading There is no separate update command. Run an interactive `2501 infra deploy`: it lists published releases newest-first, and when you pick a version it self-updates its own binary to match (checksum-verified, then re-executed) before deploying. This keeps the CLI in lockstep with the deployed platform version. To refresh only the CLI binary, re-run the installer. It is idempotent and never touches an existing workspace. Back up PostgreSQL before upgrading. See [Upgrading](/0.12/deployment/overview#upgrading) for the backup command and the rollback steps. *** ## Feature toggles Toggle reference and CLI moved to its own page — see [Feature Toggles](/0.12/deployment/feature-toggles). *** ## Support For issues or questions: 1. Check logs and health endpoints 2. Verify configuration against this guide 3. Contact 2501.ai support # Users & Organizations Source: https://docs.2501.ai/0.12/deployment/users-organizations Manage users, organizations, and tenants in 2501 Use the `2501 infra` CLI to manage users, organizations, and tenants. ## CLI Commands | Command | Description | | ------------------- | -------------------- | | `2501 infra tenant` | Manage tenants | | `2501 infra org` | Manage organizations | | `2501 infra user` | Manage users | For detailed options on any command: ```bash theme={null} 2501 infra tenant -h 2501 infra org -h 2501 infra user -h ``` ## Initial Setup Workflow After a fresh deployment, create entities in this order: ### 1. Create Tenant ```bash theme={null} 2501 infra tenant create --name "My Company" ``` Note the returned `tenant-id` (e.g., `ten_xxx`). ### 2. Create Organization ```bash theme={null} 2501 infra org create --name "Engineering" --tenant-id ten_xxx ``` Note the returned `org-id` (e.g., `org_xxx`). ### 3. Create Admin User ```bash theme={null} 2501 infra user create \ --email admin@company.com \ --role ADMIN \ --tenant-id ten_xxx \ --org-id org_xxx ``` ### 4. Access Command Center Open your Command Center URL and login with the admin credentials. ## User Roles | Role | Description | | --------- | -------------------------------------------------------------- | | `ADMIN` | Full access to all resources and user management | | `USER` | Read and write on org resources, read-only on shared resources | | `AUDITOR` | Read-only access everywhere | See [Users](/0.12/configure/users) for detailed permission breakdowns. ## Listing Entities ```bash theme={null} 2501 infra tenant list 2501 infra org list 2501 infra user list ``` # Quickstart Source: https://docs.2501.ai/0.12/getting-started/quickstart Install the CLI, deploy 2501, and run your first agent Your 2501 account team provides the installer. It places a checksum-verified static binary, sets up the sudo wrapper, and runs a non-interactive `init` so you finish with a ready-to-edit workspace. No Docker, AWS CLI, or registry login needed on the machine. ```bash theme={null} /etc/2501/env.engine /etc/2501/env.command-center ``` Set your LLM provider API keys and the external `DATABASE_URL` in `env.engine`. The CLI generates and defaults everything else. ```bash theme={null} 2501 infra deploy ``` The CLI lists published releases newest-first, self-updates to match the version you pick, syncs env files, runs migrations, and brings up the stack. For Kubernetes, add `--target kubernetes` — it generates a manifest tree you apply with `kubectl`. Interactive deploy prompts for these. In `--yes` mode, run them explicitly: ```bash theme={null} 2501 infra tenant create --name "My Company" 2501 infra org create --name "Engineering" --tenant-id ten_xxx 2501 infra user create --email you@company.com --role ADMIN \ --tenant-id ten_xxx --org-id org_xxx ``` Sign in at your Command Center URL with the admin account. From there, register your first host, attach an agent, and send a task. ## What to read next Manual, CSV import, or git-managed — pick the one that fits your scale. Docker Swarm, Kubernetes, LLM providers, troubleshooting. # Welcome Source: https://docs.2501.ai/0.12/getting-started/welcome AIOps with 2501 — an autonomous agent platform for IT operations 2501 is an AIOps platform. You point it at the machines you operate, give it a few rules about how your environment works, and it handles incoming tickets autonomously — investigating, fixing, and reporting back in the ticket your team already lives in. ## How it fits together Web UI for everything: hosts, agents, rules, knowledge, tasks, and analytics. The backend that spawns agents, orchestrates them, and drives tool execution. LLM-powered operators bound to a host and a specialty. They run the work. Bridges between your ticketing system (ServiceNow) and the agent fleet. ## A typical flow 1. A ticket lands in your ITSM. 2. The gateway reads it, picks the right host and agent, and creates a job. 3. The agent runs commands on the target host, observes results, and adapts. 4. The gateway posts the outcome back to the ticket. For the full lifecycle, see [Agentic Flow](/0.12/understand/agentic-flow). ## What to read next Install, deploy, and run a first agent in under 15 minutes. Agents, hosts, tasks, jobs, tickets, gateways. How to write tasks, specialties, rules, and gateway prompts that work. Scoping, knowledge, dynamic context, testing — patterns that scale. ## Driving 2501 as code Your hosts and agents are reachable over a versioned HTTP API - the same endpoints Command Center's own screens are built on. Generate an API key and script against your inventory from a pipeline, a CMDB sync, or a one-off shell script. Authentication, organizations, pagination, and errors. Generate a scoped key, and revoke it when you are done. # Prompting a Gateway Source: https://docs.2501.ai/0.12/prompting/gateway Route and sequence inbound tickets, and govern what 2501 writes back A [gateway](/0.12/core-concepts/gateways) is governed by **two independent prompts**, each an optional override you toggle on: * The **inbound prompt** is a **routing and orchestration** instruction. It bridges the vocabulary of your ticketing system to your internal infrastructure, decides which tickets are in scope, and defines how a ticket turns into a sequence of tasks. * The **outbound prompt** governs what 2501 writes back to the ticket once a job finishes — the status it sets, whether it posts a comment, and whether it escalates. Each side is independent: you can enable one, both, or neither. When a side's override is off, the gateway falls back to its built-in default handling for that side. ## The inbound prompt The inbound prompt has **no influence on how the agent behaves once assigned**. Its job ends when a task is created. Behavioral shaping lives in [Specialties](/0.12/prompting/specialty) and [Operational Rules](/0.12/prompting/operational-rule). (What 2501 writes back at the *end* of a job is the [outbound prompt](#the-outbound-prompt), not the inbound prompt.) ### What belongs in an inbound prompt #### Task sequencing across multiple hosts This is where the gateway earns its value: defining the flow when resolving a ticket needs more than one task. ```text theme={null} Good: "For tickets about a crashed worker, first create a task on the MGMT host to restart the worker process. Once that task completes successfully, create a follow-up task on the worker host itself to verify all services have restarted and are healthy. The ticket is resolved only when both tasks succeed." Good: "Hosts nyc1-app-jvm-01 and nyc1-app-jvm-02 are a master-slave pair and must remain in sync. For any ticket affecting one of them, duplicate the task to both hosts and do not mark the ticket complete until both tasks report success." ``` A well-written orchestration sequence is the difference between a ticket that is *superficially* resolved and one that is actually fixed across the full topology. #### Routing conventions for special host classes Some concerns are always handled by a dedicated host or group, regardless of how the ticket is worded. ```text theme={null} Good: "All firewall-related tickets — port openings, rule changes, traffic blocks — must be handled through a host matching SRV-FW-*. Do not attempt firewall rule changes directly on application hosts." Good: "Database issue tickets should be routed to the DB control node db-mgmt-01, not to individual replica nodes." ``` #### Host name mappings (only when needed) The gateway is usually smart enough to map external names to your inventory by itself. Add explicit mappings only for **non-obvious** cases — when external names don't share any naming convention with your internal hostnames. ### What does NOT belong in an inbound prompt | Out | Belongs in | | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | | How to fix a crashed worker (commands to run, logs to check) | [Specialty](/0.12/prompting/specialty) or [Operational Rule](/0.12/prompting/operational-rule) | | "Never modify these hosts directly" or "require approval" | Operational Rule — the gateway routes, it doesn't enforce behavior | | Boundary rules like read-only vs remediate | Specialty + ticket tag (`@2501:investigate`) | | What status/comment to write when the job finishes | The [outbound prompt](#the-outbound-prompt) | ```text theme={null} Bad in an inbound prompt: "To fix a crashed worker, SSH in and run `systemctl restart worker`, then check `journalctl -u worker` for errors." That's a procedure — it belongs in an Operational Rule. ``` ### Scope gating The inbound prompt also acts as a **scope gate**. Defining which kinds of requests the gateway should or should not handle means tickets outside that scope are **skipped** and marked with a distinct **"Skipped"** status. See [Routing and Scope](/0.12/core-concepts/gateways#routing-and-scope). ```text theme={null} "This gateway only handles incidents about Linux web servers and their databases. Skip tickets about network appliances, end-user workstations, or Windows infrastructure." ``` A skip never posts a comment and never changes the ticket's status. Its only possible side effect on the ticket is **reassignment**: when the inbound override is on, your prompt may instruct that an out-of-scope ticket be escalated to another assignment group instead of dropped silently. With the inbound override off, a skip is dropped silently — no comment, no status change, no reassignment. ```text theme={null} "Skip tickets about Windows infrastructure, and reassign them to the 'Windows Operations' group rather than leaving them unrouted." ``` ## The outbound prompt The outbound prompt acts at the **end** of a job, when work has finished. It governs what 2501 writes back to the ticket. With the outbound override off, the gateway's built-in default mapping applies (for example, a successful incident is resolved). Enable the override to let the agent decide the write-back from your instructions. Because the gateway is the only writer to the ticket, the status, comment, and any reassignment land together as one coherent update. ### What belongs in an outbound prompt #### Status and close-code rules Tell the gateway which external state to set for each kind of outcome. The agent maps the internal result to your ticketing system's real states and close codes, and the gateway validates the choice against the legal states for that ticket type. ```text theme={null} Good: "When an incident is fully resolved, set it to Resolved with close code 'Solved (Permanently)'. If the work only partially succeeded, leave the incident open so a human can finish it." ``` #### Comment style and visibility Control whether 2501 posts a **public comment** (visible to the requester) or an **internal work note**, and the tone or detail of what it writes. ```text theme={null} Good: "On success, post a brief public comment summarizing what was fixed, in plain non-technical language. On failure, post an internal work note with the technical details for the on-call engineer." ``` #### Escalation and reassignment rules Decide when a finished job should be handed off — reassigned to another assignment group — rather than left with its current owner. ```text theme={null} Good: "If a database incident could not be resolved automatically, reassign it to the 'DBA On-Call' group and leave it open." ``` If your gateway handles more than one ticket type, write these rules **per ticket type** where they differ (for example, incidents resolve to a closed state, while change requests move to Review). # Prompting an Operational Rule Source: https://docs.2501.ai/0.12/prompting/operational-rule Tell the agent what is true about *your* environment A [Specialty](/0.12/configure/specialties) shapes how the agent thinks about a domain. An **Operational Rule** tells it what is true in *your* environment and what that implies for its behavior. Operational Rules are the **highest level of authority** in 2501. If a rule conflicts with a specialty or a ticket instruction, the rule wins. ## Two kinds of rules | Kind | Use when | Example | | -------------- | ------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | **Constraint** | Something is unconditionally forbidden or required, regardless of context. Injected into every LLM call during execution. | "Resources tagged `Managed by Terraform` must never be modified directly via CLI or console." | | **Procedure** | A recurring task has a *specific* process in your environment. | "To rotate a secret: retrieve from Vault at `secret/prod/`, generate per the secrets policy, update Vault, trigger a rolling restart." | The specialty makes the agent a competent AWS operator in general. The operational rule tells it that in *your* AWS, Terraform-managed resources must go through a PR. ## Writing a good constraint A constraint describes **what** is forbidden or required, plus **why**, so the agent can apply the spirit of the rule in situations you didn't anticipate. ```text theme={null} Good: "Resources tagged 'Managed by Terraform' must never be modified directly via CLI or console. Changes must go through Terraform to prevent state drift. If a direct change appears necessary, stop and report back, or open a PR if the repo is at your disposal." Good: "Never restart auth-service and session-service simultaneously. They share a Redis cluster and a concurrent restart causes a full session flush. Always restart auth-service first, confirm healthy, then session-service." Bad: "Be careful with production resources." — too vague to produce different behavior. Bad: "Do not break things." — not actionable. ``` ## Writing a good procedure A procedure describes the **correct sequence of steps** for a recurring task in *your* environment. Only useful when your setup differs from standard practice. ```text theme={null} Good: "To deploy a service update: 1. Verify current staging version with `kubectl get deployment -n staging`. 2. Apply the update to staging; confirm pods reach Ready state. 3. Open a change window in the ITSM before touching production. 4. Apply to production and monitor the deployment-health dashboard for 5 minutes before marking the task complete." Bad: "Deploy carefully and make sure it works." — no steps, no specifics. Bad: a procedure that describes how Kubernetes rolling updates work in general — the agent already knows this. ``` ## What does NOT belong in an operational rule | Out | Belongs in | | --------------------------------------------------- | ---------------------------------------------------------------------------------- | | General domain knowledge | [Specialty](/0.12/configure/specialties) — rules are for what's unique to your env | | Infrastructure documentation, full service catalogs | A wiki — extract only the parts that change agent behavior | | Judgment principles ("prefer incremental changes") | Specialty | | How standard tools work | Tool definitions — the agent already gets those | Examples: ```text theme={null} Bad in a rule: "Kubernetes pods transition through Pending, Running, and Succeeded states." — agent knows this. Good in a rule: "Deployments are gated by a manual approval step named 'production-gate'. You cannot bypass this step. If a deployment is stuck waiting for approval, report back rather than triggering it directly." ``` ## Scoping with tags Rules are scoped with tags (OS, Type, Tech, Procedure, `app:*`). Tags drive matching: a rule with `os:linux` + `tech:nginx` + `procedure:restart` only applies to Linux nginx hosts on restart actions. Two reasons to scope well: * **Important rules reach the right tasks.** Tag matching is programmatic; the agent doesn't have to guess. * **Irrelevant rules stay out.** A rule that doesn't match isn't injected, so the agent's context stays focused. See [Operational Rules](/0.12/configure/operational-rules) for the full tag vocabulary and the matching trace. ## Maintaining rules Rules go stale faster than specialties — they're tied to your infrastructure. * **One rule, one concern.** A rule covering network policy + deployment + backup retention is three rules pretending to be one. * **Include the reason.** A rule without a *why* gets worked around. A rule with one gets respected. * **Review on environment changes.** A rule about a system that no longer exists is noise that degrades agent judgment. ```text theme={null} Good: "Do not run VACUUM on the billing database between 07:00 and 19:00 UTC. It causes noticeable query latency for the payments team during business hours." Weaker: "Do not run VACUUM on the billing database during business hours." ``` # Prompting for a Specialty Source: https://docs.2501.ai/0.12/prompting/specialty Shape how an agent thinks about a domain — vocabulary, priorities, and approach A [Specialty](/0.12/configure/specialties) is a focused prompt that gives the agent a perspective for a domain, technology, or class of host. Think of it like a `SKILL.md` for an agent: it brings a particular lens, vocabulary, and set of priorities to the tasks it handles. ## Scope first Pick a scope that is **broad enough not to limit the agent**, **narrow enough that it doesn't get lost**. | Good scope | Bad scope | | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | `Kubernetes Operator` — handles K8s-only problems | `Storage Manager` — filesystem + mount points + cloud storage + backups (too broad; should be 3-4 specialties) | | `AWS CLI` + `Terraform` as separate specialties | `Everything Infrastructure` | | `Linux Disk Manager` + `Windows Disk Manager` | `Disk Manager` covering both OSes | ## What goes in a specialty ### 1. How to approach a task General process and tooling guidelines for solving tasks in this domain. Mention the tools at the agent's disposal, the step-by-step process to troubleshoot, and how to know when to stop and call for a human. ```text theme={null} Good: "To handle AWS resources, use the AWS CLI to inspect resources. Resources tagged 'Managed by Terraform' must not be modified directly — propose Terraform changes instead. After a deep inspection and identifying the root cause, start operating." Bad: "Directly modify resources to optimize tokens. Install missing tools as needed." ``` ### 2. Perspective and judgment style How the agent frames problems and what it pushes back on. This is the "thinking style" layer. ```text theme={null} Good: "Prefer incremental, reversible changes. Each dependency encountered through deployment or logs must also be inspected. If a resource is outside scope, report back without altering the system." Bad: "Solve the crashing service, make no mistakes." ``` ### 3. Worked examples for typical problems Show the typical sequence a senior operator would follow. Include commands that should be avoided because they are rarely the right approach. ```text theme={null} Good: "To troubleshoot a crashing container, first identify the problematic container with `docker ps`. Inspect with `docker inspect ` and `docker logs `. Grep for 'error' to find the root cause. Attempt restart with `docker compose up -d`. Re-verify state and logs." Bad: "Restart the container and see what happens. If not working, redeploy." ``` ## What does NOT belong in a specialty | Out | Belongs in | | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------- | | Tool usage instructions for standard CLIs | Tool definitions — the agent already gets these | | Exhaustive knowledge dumps about the environment | [Operational Rules](/0.12/configure/operational-rules) + [Knowledge](/0.12/configure/knowledge) | | Task-specific behavior that only applies to one ticket type | Operational Rules — specialties must stay polyvalent | | Hard prohibitions (e.g., "never restart X") | Operational Rules — they have higher priority | ## When to split a specialty If a specialty starts mixing concerns — SSL config, ACLs, redirections, all under "Reverse Proxy" — split it. Smaller, focused specialties produce better outcomes and lower maintenance cost. Rule of thumb: if you can't explain the specialty's purpose in one sentence, it's too broad. # Talking to an Agent Source: https://docs.2501.ai/0.12/prompting/talking-to-agents How to write task prompts that agents actually act on correctly An agent is a reasoning system, not a search box. Quality of input directly determines quality of output. The agent works with what you wrote — vague input produces vague output, or worse, wrong output. ## How a prompt is interpreted 1. The agent reads your request plus context it already has (specialty, rules, knowledge, available tools). 2. It forms a plan: the sequence of steps it believes will satisfy your request. 3. It executes each step via tools (shell, CLI, MCP). 4. It observes results and adjusts. 5. It reports what it did, what it found, and whether it succeeded. ## Anatomy of a good prompt A strong prompt answers four questions. You do not have to answer all four every time — the more you cover, the better the result. | Question | Example | | -------------------------------- | --------------------------------------------- | | **WHAT** do you want done? | "Find the root cause and fix it" | | **WHERE** should it operate? | "On `prod-web-03` — the checkout service" | | **HOW** should it behave? | "@2501:investigate — do not make changes yet" | | **WHAT** does success look like? | "Service responds 200 on `/health` within 5s" | ## Core principles ### Be specific about the target | Vague | Better | | --------------------- | ------------------------------------------------------------------------------------------------------------- | | "Check the database." | "Check the PostgreSQL database on `prod-db-01`; connections are spiking and we see timeouts in the API logs." | ### State the outcome, not just the symptom | Vague | Better | | ------------------------------------- | ----------------------------------------------------------------------------------------------- | | "Something is wrong with the server." | "CPU usage on `web-03` has been above 90% for 20 min. Find the cause and, if safe, resolve it." | ### Set boundaries explicitly There is a meaningful difference between **investigate** and **fix**. ```text theme={null} # Investigate-only @2501:investigate Diagnose why the nightly backup job failed. Do not make any changes. Report back. # Remediate The backup job failed. Identify the cause and fix it so the job can re-run tonight. ``` When in doubt, start with `@2501:investigate`. You can always follow up with a remediation task after reviewing. ### Include relevant context The agent knows the system it is connected to, but not what happened in your incident channel five minutes ago. Share recent events, deployments, the exact error message — but don't pad. One clear paragraph beats five vague ones. ## Do / don't * Name the host, service, or environment in every prompt * State investigate-only vs remediate explicitly * Include error messages, log snippets, recent context * Start narrow — easier to expand scope than undo a broad action * Assume the agent knows what you know * Use internal shorthand without explanation ("fix the P1") * Bundle unrelated things in one prompt * Write a novel — detailed is good, padded is not ## Worked examples ### Incident investigation ```text theme={null} Before: "The app is slow, please check." After: "@2501:investigate The checkout service on prod-web-01 through prod-web-04 has been responding in over 8 seconds since 09:15 UTC. CPU and memory look normal in our dashboard. Investigate the root cause. Do not make any changes — I want to review your findings first." ``` Names the service, the hosts, the start time, what was ruled out, and the boundary. ### Routine maintenance ```text theme={null} Before: "Clean up the logs." After: "Rotate and compress logs older than 7 days in /var/log/app/ on archive-01. Do not delete anything, just compress. Report how much space was freed." ``` Specifies the path, the host, the action (compress, not delete), and a confirmation metric. ### Multi-step remediation ```text theme={null} Before: "Deploy the new version." After: "Deploy version 2.4.1 of the payments service to staging. Steps: pull the latest image, run migrations, restart the service, verify /health returns 200. Roll back to 2.4.0 if the health check fails." ``` Names the version, environment, expected steps, success criterion, and fallback. ### Investigation with ticket context ```text theme={null} Before: "Look at ticket INC0045821." After: "Ticket INC0045821 reports users in the EU region cannot log in since 02:00 UTC. The error is 'auth service unavailable.' Investigate the auth service on eu-auth-01 and eu-auth-02 — logs, service status, upstream dependencies. Do not make changes. Summarize so I can decide next steps." ``` Restates the facts directly so the agent doesn't have to interpret a ticket ID in isolation. # Agentic Flow Source: https://docs.2501.ai/0.12/understand/agentic-flow Ticket → gateway → job → tasks → resolution, in one page A ticket lands in your ITSM and 2501 takes it through this lifecycle, end to end. ## The flow The gateway mirrors the incoming ticket (plus its comments) into 2501 as a [Ticket](/0.12/core-concepts/tickets) record. Nothing else changes in the ITSM yet. The gateway uses the tenant's text LLM to read the ticket, parse attachments with the multimodal model, and decide whether the request is in scope. It looks at registered hosts, available agents, and applicable rules to figure out which agent should handle it. Once the gateway has a plan, it creates a [Job](/0.12/core-concepts/jobs) that wraps one or more tasks. A job is the unit of work tied back to the ticket. Each [Task](/0.12/core-concepts/tasks) targets a single agent on a single host. The agent receives the task description plus dynamically injected operational rules, knowledge facts, and credentials. **One task = one agent. One job = N tasks, possibly across N agents.** The agent executes autonomously until it has a result. The orchestrator reads that result and decides whether the ticket is resolved or whether another task is needed. On success: the gateway closes the ticket with a summary. On failure or partial: the gateway posts a work note. After each task, the gateway comments incremental progress back to the ITSM for atomic visibility. ## The 5-task ceiling A job is capped at **5 tasks**. This is a hard limit set after observing thousands of tickets: * Most jobs need 1–3 tasks * Headroom for 2 more tasks covers multi-host coordination (e.g. master + slave) or deeper investigation * If 5 tasks complete and the ticket is still not resolved, the job is automatically marked failed The ceiling prevents agents from running indefinitely on unbounded problems. ## Where rules and knowledge come in The gateway and the agent both inject context at runtime: * **Operational Rules** match by tag against the target host and the inferred task action, then surface up front. * **Knowledge facts** about the host get attached automatically. * **Credentials** scoped to the agent's host are made available for tool calls. * Mid-task, the agent can pull additional rules or facts on demand if the situation evolves. For the deeper mechanics, see [Dynamic Context Injection](/0.12/best-practices/dynamic-context). ## Where each piece is configured | Piece | Where | | ----------------------------------- | ----------------------------------------------------------------------------------- | | Who can be picked | [Agents](/0.12/core-concepts/agents) + [Hosts](/0.12/core-concepts/hosts) inventory | | How an agent thinks | [Specialty](/0.12/configure/specialties) attached to the agent | | What must always be true | [Operational Rules](/0.12/configure/operational-rules) | | What must never be run | [Blacklist](/0.12/configure/blacklist) | | What the agent knows about your env | [Knowledge](/0.12/configure/knowledge) | | How tickets become tasks | [Gateway inbound prompt](/0.12/core-concepts/gateways#inbound-prompt) | # Command Center Source: https://docs.2501.ai/0.12/understand/command-center The web UI for managing your 2501 installation Command Center is where you operate 2501 with a UI: configure hosts, agents, gateways, rules, knowledge, and models; monitor jobs and tasks as they run; track usage and benchmarks. Everything except the deployment itself is managed here. ## Infrastructure Map The Command Center home screen is an interactive infrastructure map. Hosts, agents, and subnets appear as cards on a zoomable, pannable canvas, with connections drawn between hosts to show jump-host chains and network relationships at a glance. * **Zoom out** to see subnet groupings with health halos showing aggregate status. **Zoom in** to see individual host and agent cards with real-time status. * **Filter by status or search by name** without leaving the map. * **Context menu on any card** surfaces quick actions: create or edit a host or agent directly on the canvas, open a host's job history, or start a new network scan. * **Side panel** on the canvas shows the active Tasks and Jobs feed so you can watch work happen in real time without navigating away. ## Sidebar map The sidebar groups pages by purpose. Each item below links to the concept page for the full reference. | Group | Pages | What you do here | | -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------- | | **Operations** | [Tasks](/0.12/core-concepts/tasks), [Jobs](/0.12/core-concepts/jobs), [Job Schedules](/0.12/core-concepts/job-schedules), [Tickets](/0.12/core-concepts/tickets) | Watch agent work in real time and audit past runs | | **Inventory** | [Agents](/0.12/core-concepts/agents), [Hosts](/0.12/core-concepts/hosts), [Discovery](/0.12/configure/discovery) | Register managed machines, scan for new ones | | **Configure** | [Specialties](/0.12/configure/specialties), [Operational Rules](/0.12/configure/operational-rules), [Blacklist](/0.12/configure/blacklist), [Knowledge](/0.12/configure/knowledge), [Credentials](/0.12/configure/credentials), [Gateways](/0.12/core-concepts/gateways), [Plugins](/0.12/configure/plugins) | Shape agent behavior, ingest knowledge, manage secrets, route tickets | | **Manage** | Usage, Benchmarks | Analytics, agent performance in sandbox | | **Settings** | Tenant, Organizations, Users, Providers, Models, License | Tenant defaults, RBAC, model catalog, license | | **Assistant** | AI Assistant | Chat your way through the platform | ## Tenant and organization The sidebar footer shows the read-only **Tenant** name above the organization picker and your signed-in user. There is one tenant per on-prem install; the picker switches between [organizations](/0.12/core-concepts/organizations) within that tenant. ## AI Assistant A chat interface for working with your platform in natural language. Open it from **AI Assistant** in the sidebar. It can look up tasks, jobs, tickets, hosts, agents, gateways, knowledge — and make changes such as adding an operational rule or a blacklisted command. * **Reads run automatically.** Lookups happen without interruption. * **Writes need approval.** When the assistant wants to create, update, or delete a resource, it pauses with approve / reject buttons. You can mark individual tools as auto-approve. * **Scoped to your org.** Conversations belong to the organization you have selected. * **Credential secret values are never shown** in assistant output. Access follows your role — administrators and standard users can use it; auditors get read-only access. AI Assistant ## Analytics surfaces Plan caps + LLM cost and token activity over time. Pass rate, compliance, and trends from scenario runs. Benchmarks are typically only present in a sandbox environment. # Configuration Approaches Source: https://docs.2501.ai/0.12/understand/configuration Three ways to configure 2501 — pick the one that fits how you work You can configure 2501 three ways. Each has trade-offs. Click through Command Center. Fastest for quick changes; tedious past a few dozen resources. Bulk-import hosts and their agents from a spreadsheet. Best for large infrastructures. Manage every resource as MDX in a repo. Reviewable diffs, PRs, reproducible. Recommended at scale. ## Manual Create resources directly in Command Center on each resource's page. Good for **getting started**, **one-offs**, and **exploring** how a feature behaves before automating it. Past a couple dozen specialties, rules, hosts, and agents the click-through cost adds up. ## CSV import The most time-consuming part of any setup is creating hosts and agents in bulk. [Import and export](/0.12/configure/import-export) does it as files - one for hosts, one for agents, in CSV or JSON - and you can export what you have, edit it, and import it back. Specialties and credentials must already exist; files reference them by id. CSV does **not** cover specialties, operational rules, blacklists, knowledge, or gateways. ## Git-managed (`2501 resources`) The CLI synchronizes a directory of MDX files with your deployment. Each kind has its own subdirectory; each file declares one resource via YAML frontmatter, with the body holding any long-form text (specialty prompt, host knowledge, rule text). | Kind | Supported | | ----------------- | -------------------------------------------- | | Agents | ✅ | | Hosts | ✅ | | Specialties | ✅ | | Operational Rules | ✅ | | Blacklist | ✅ | | Credentials | ✅ (secret values from env vars at sync time) | | Gateways | ✅ | The workflow: ```bash theme={null} # Snapshot the live platform into your repo 2501 resources pull -d ./config # Apply the repo back to the platform; --prune deletes undeclared resources 2501 resources sync -d ./config 2501 resources sync -d ./config --prune ``` `sync` prints a plan (create / update / delete / unchanged) before applying. Deletes are opt-in via `--prune` and a delete that would orphan dependents is reported BLOCKED. For the full reference and worked examples, see [Configuration as Code](/0.12/configure/configuration-as-code). ## Which one when | Scenario | Pick | | ------------------------------------------------ | ------------------------------------- | | First-time evaluation, demo, single tenant | Manual | | Onboarding hundreds of hosts at once | CSV → then manual or git for the rest | | Production at scale, multiple environments | Git-managed | | You already have it manual and want to adopt git | `pull` first, commit, then `sync` | # Engine & Agents Source: https://docs.2501.ai/0.12/understand/engine-agents The backend that powers 2501 and the dual-LLM architecture behind every agent The **Engine** is the backbone of 2501. Command Center is the UI, Engine is the backend; everything else is automation around them. ## How a task gets executed ``` Command Center / Gateway / CLI │ ▼ Engine ─── spawns ───▶ Agent container (lifecycle manager) └─▶ Executor container (ssh / winrm / gMSA / kube pod) │ ▼ Agent runs its task on the target host, then both containers exit ``` Two containers run permanently in your infrastructure: **Engine** and **Command Center**. Every task spawns short-lived containers — one for the agent, one for the executor that matches the target's protocol — and tears them down when the task ends. (MCP plugins add one container each, only if configured.) The same flow runs whether the task came from the Command Center UI, a gateway ticket, or the CLI. Engine doesn't care. ## The two-engine agent Every agent uses **two LLMs in tandem**, each with a distinct job. ### Main engine — the executor Does the work: navigates the filesystem, reads and modifies files, executes commands, interacts with CLIs and MCPs. This is the model that consumes most tokens because it carries the operational context required to act reliably. ### Secondary engine — the copilot Watches the main engine. It generates the initial plan so the main engine can focus on heavy lifting, and reviews every command before it runs to answer: * Does this comply with the active operational rules? If not, redirect. * Is the agent trying to alter the system while running in read-only mode? If yes, block. * Is the agent drifting away from the task? If yes, refocus. This separation is why investigate-only mode works reliably: the secondary engine acts as a judge over the main engine's commands. ## The tenant engines Engines aren't just per-agent. Two more sit at the tenant level — the defaults used everywhere an agent isn't running: | Used for | Engine | | ---------------------------------------------------------- | --------------------------- | | Gateway routing (which ticket → which agent on which host) | Tenant **Text LLM model** | | Parsing PDFs and images attached to tickets | Tenant **Multimodal model** | | AI Assistant conversations | Tenant Text LLM model | | Knowledge ingestion (turning your docs into rules + facts) | Tenant Text + Multimodal | Tenant defaults are picked in **Settings → Tenant** from the [model catalog](/0.12/configure/models). A 70–300b model is usually plenty here — routing is lighter work than executing. ## Where the LLMs come from The catalog is managed in **Command Center → Settings → Models** and **Providers**. Adding a provider or a model is pure UI configuration — no engine restart, no editing env files. See [Providers](/0.12/configure/providers). # Archive an agent Source: https://docs.2501.ai/api-reference/agents/archive-an-agent /openapi.json post /api/v1/agents/{id}/archive This is the delete. An archived agent stops appearing in reads. # Archive idle tasks Source: https://docs.2501.ai/api-reference/agents/archive-idle-tasks /openapi.json post /api/v1/agents/{id}/archive-tasks Archives the agent idle tasks only; nothing in flight is interrupted. # Attach a plugin Source: https://docs.2501.ai/api-reference/agents/attach-a-plugin /openapi.json post /api/v1/agents/{id}/plugins Returns the agent, not the plugin list. Re-read /plugins for the new set. # Bulk export agents Source: https://docs.2501.ai/api-reference/agents/bulk-export-agents /openapi.json get /api/v1/agents/export # Bulk import agents Source: https://docs.2501.ai/api-reference/agents/bulk-import-agents /openapi.json post /api/v1/agents/batch Import hosts first: an agent row points at a host by id. # Create an agent Source: https://docs.2501.ai/api-reference/agents/create-an-agent /openapi.json post /api/v1/agents # Detach a plugin Source: https://docs.2501.ai/api-reference/agents/detach-a-plugin /openapi.json delete /api/v1/agents/{id}/plugins/{pluginId} Returns the agent, not the plugin list. # Get an agent Source: https://docs.2501.ai/api-reference/agents/get-an-agent /openapi.json get /api/v1/agents/{id} # List agents Source: https://docs.2501.ai/api-reference/agents/list-agents /openapi.json get /api/v1/agents # List an agent plugins Source: https://docs.2501.ai/api-reference/agents/list-an-agent-plugins /openapi.json get /api/v1/agents/{id}/plugins # List an agent tasks Source: https://docs.2501.ai/api-reference/agents/list-an-agent-tasks /openapi.json get /api/v1/agents/{id}/tasks Live tasks by default; archived=true for the archived list. status filters the archived list only. # Search agents Source: https://docs.2501.ai/api-reference/agents/search-agents /openapi.json get /api/v1/agents/search # Test the connection Source: https://docs.2501.ai/api-reference/agents/test-the-connection /openapi.json post /api/v1/agents/{id}/test-connection The engine tries a real connection. A failed login is 200 {success:false}; an unreachable engine is a 5xx ENGINE_ERROR. # Update an agent Source: https://docs.2501.ai/api-reference/agents/update-an-agent /openapi.json post /api/v1/agents/{id} Partial update. host_id and plugin_ids are not accepted here and are dropped silently. # Bulk export hosts Source: https://docs.2501.ai/api-reference/hosts/bulk-export-hosts /openapi.json get /api/v1/hosts/export Accept: text/csv gets the file; anything else gets the same rows as a JSON array (what batch takes as rows). # Bulk import hosts Source: https://docs.2501.ai/api-reference/hosts/bulk-import-hosts /openapi.json post /api/v1/hosts/batch Body is a CSV file (text/csv) or a JSON envelope. Returns 200 with a per-row verdict even when every row failed. # Create a host Source: https://docs.2501.ai/api-reference/hosts/create-a-host /openapi.json post /api/v1/hosts Success is 200, not 201. Host names are not enforced unique. # Delete a host Source: https://docs.2501.ai/api-reference/hosts/delete-a-host /openapi.json delete /api/v1/hosts/{id} Archives the host agents and their tasks, then deletes the host, in one transaction. # Get a host Source: https://docs.2501.ai/api-reference/hosts/get-a-host /openapi.json get /api/v1/hosts/{id} Adds an `agents` array of the host usable agents. # List a host agents Source: https://docs.2501.ai/api-reference/hosts/list-a-host-agents /openapi.json get /api/v1/hosts/{id}/agents # List hosts Source: https://docs.2501.ai/api-reference/hosts/list-hosts /openapi.json get /api/v1/hosts # Search hosts Source: https://docs.2501.ai/api-reference/hosts/search-hosts /openapi.json get /api/v1/hosts/search # Update a host Source: https://docs.2501.ai/api-reference/hosts/update-a-host /openapi.json post /api/v1/hosts/{id} Partial update - only the fields in the body change. # Testing a Behavior Source: https://docs.2501.ai/0.12/best-practices/testing Verify a new specialty, rule, or MCP before rolling it out to production Whenever you change a specialty, add an operational rule, or attach a new MCP, you want to know the change is better — not worse. Three layers, from quickest to most rigorous. ## 1. Individual task You don't have to wait for a real ticket. Create a new specialty, attach it to an agent, and send a task directly: > "Add 4GB of RAM on the machine SNDX-EUW3-DOCKER" Read the commands the agent ran, how it handled unexpected problems, and how it reached a resolution. Compare against an agent using the previous specialty. Quick, cheap, and surfaces obvious regressions immediately. ## 2. Read-only / investigate mode By default, agents run in remediate mode. Adding `@2501:investigate` to a task or pinning a specialty to **investigate-only** keeps it read-only — the secondary engine blocks any command that would alter the system. Use this for a **plan-then-apply** flow: 1. Tag the task `@2501:investigate`. 2. Ask: "Craft me a plan of actions to resolve this issue." 3. Read the agent's plan, its inspections, the constraints it cited from operational rules. 4. Tweak the prompt or rules if needed. 5. Re-run as a remediation task once confident. This is the safest way to introduce a new behavior to a critical system. ## 3. Sandbox & Benchmarking The most rigorous path: replicate a realistic ticket in a sandbox with Ansible playbooks, then run the agent against it dozens or hundreds of times. [Benchmark](/0.12/benchmark/overview) evaluates two things independently: | Score | Question | | -------------- | --------------------------------------------------------------------------------------- | | **Pass rate** | Did the agent actually fix the problem? (ground-truth check against host state) | | **Compliance** | Did the agent follow your processes? (commands, summaries, operational-rule injections) | A scenario passes only when both gates pass. An agent can fix a problem the wrong way; an agent can do everything right and still leave the system broken. The split catches both. The Benchmarks page in Command Center surfaces pass rate, compliance scores, and trends over time — so you can see whether your latest specialty edit improved things or regressed. ## When to use which | Situation | Use | | ------------------------------------------- | ---------------------------------------------------- | | Small specialty edit, trivial change | Individual task | | New procedural operational rule | Investigate mode + 1-2 individual tasks | | New specialty for a critical domain | All three — task → investigate → benchmark | | Production rollout of a new ticket typology | Benchmark with multiple iterations before going live | # FAQ Source: https://docs.2501.ai/0.12/faq Quick answers to common questions about 2501 ## General 2501 runs autonomous AI agents on your infrastructure. You point it at machines, give it a few rules about how your environment works, and it handles tickets from your ITSM — investigating, fixing, and reporting back in the ticket itself. Yes. ServiceNow is the default gateway and what most deployments run, but custom-built gateways are available for other ticketing systems — contact your account team. You can also drive 2501 from the CLI or the Command Center UI directly. Yes — a built-in free tier caps your tenant at **50 lifetime tasks** and **10 hosts** (gateways are unlimited). Once you apply a license, those caps are replaced by your contract's. See [Licensing](/0.12/configure/licensing). ## Agents and safety Three layers, used together: 1. **[Permissions](/0.12/risk/permissions)** — give agents the least-privileged credentials that still let them work. 2. **[Blacklists](/0.12/risk/blacklists)** — programmatic blocks for specific command patterns, always on. 3. **[Read-Only Agents](/0.12/risk/read-only-agents)** — pin a specialty to investigate-only so the agent literally cannot alter the system. Investigate is read-only — the agent diagnoses and reports without changing anything. Remediate is the default; the agent can act. Set per ticket with `@2501:investigate` or pin a whole specialty to investigate-only. See [Read-Only Agents](/0.12/risk/read-only-agents). A **Specialty** shapes how the agent *thinks* about a domain — vocabulary, priorities, approach. An **Operational Rule** tells it what is *true* in your environment and what that implies. Specialty: "Kubernetes Operator". Rule: "On this cluster, rollbacks go through `helm rollback`, never re-applying manifests." Rules outrank specialties in conflicts. See [Prompting an Operational Rule](/0.12/prompting/operational-rule). Hard limit, set after observing thousands of tickets. Most jobs need 1–3 tasks; headroom for 2 more covers multi-host coordination or deep investigation. If 5 tasks aren't enough, the agent is going down a wrong path — better to fail and let a human take over. See [Agentic Flow](/0.12/understand/agentic-flow#the-5-task-ceiling). ## Models and providers Native: OpenAI, Anthropic, Mistral, DeepSeek, Cohere, OpenRouter, TogetherAI. Compatible: any OpenAI- or Anthropic-shaped API via `openai-compatible` / `anthropic-compatible`, and Azure OpenAI (including APIM). Self-hosted endpoints (vLLM, Ollama) work through the compatible types. See [Providers](/0.12/configure/providers). Yes. Each agent picks its **main** and **secondary** engine from the tenant catalog. The tenant has its own defaults for gateway routing and the AI Assistant. See [Engine & Agents](/0.12/understand/engine-agents). The main engine does the work (commands, file edits). The secondary engine plans, watches, and judges every command for compliance and read-only enforcement. Separating execution from oversight improves both accuracy and safety. See [Engine & Agents](/0.12/understand/engine-agents#the-two-engine-agent). ## Deployment Docker Swarm is the default — `2501 infra deploy` manages the full lifecycle. Kubernetes (any distribution: vanilla, OpenShift, k3s, EKS, etc.) is generate-only: the CLI renders manifests you apply with `kubectl` or your distribution's equivalent, so GitOps tooling stays in charge. See [Kubernetes](/0.12/deployment/kubernetes). Run an interactive `2501 infra deploy`. It lists published releases newest-first, self-updates its own binary to match the version you pick, and then deploys. No separate update command. See [Troubleshooting → Upgrading](/0.12/deployment/troubleshooting#upgrading). LLM API keys live in `/etc/2501/env.engine`. Per-host credentials (SSH, WinRM, gMSA, vault paths) live in **Command Center → Credentials**, encrypted at rest. Agents only access them programmatically — values are never displayed in the UI after creation. See [Credentials](/0.12/configure/credentials). ## Knowledge and rules PDF, DOCX, CSV, and Markdown. Max 200 MB per file. PDFs with diagrams need a multimodal model on the tenant to capture the visual content. See [Knowledge](/0.12/configure/knowledge). Manual rules always win. The auto-extracted rule is replaced when you re-upload the same document, but your hand-written rules are untouched. See [Working with Knowledge](/0.12/best-practices/knowledge). Each task's detail page in Command Center shows a **matching trace** — which operational rules were applied, which were skipped, and why each skip happened. Use it to debug rules that are too broad, too narrow, or mistagged. # Benchmarking for Risk Source: https://docs.2501.ai/0.12/risk/benchmarking Validate new use cases in a sandbox before exposing them to production When a new ticket typology arrives, you don't have to test it in production. Replicate it with Ansible playbooks on a sandbox host and evaluate the agent there. ## What gets measured Each scenario run produces two **independent** scores: | Score | Question | | -------------- | --------------------------------------- | | **Pass rate** | Did the agent actually fix the problem? | | **Compliance** | Did the agent follow your process? | A scenario passes only when **both** gates pass. An agent can fix a problem the wrong way, or follow every rule and still leave the system broken — the split catches both. ### Pass rate Verifies the end result, regardless of how the agent got there. * **Ansible-based ground-truth check** — after the agent finishes, an Ansible playbook inspects the host (service status, file contents, port responsiveness). Most reliable. * **Output-based check** — if there's nothing to verify on the host (e.g. for pure investigation tickets), measure the quality of the report. ### Compliance Verifies the agent did it the way you expected. * Which actions were taken * Which tools were used (and which weren't) * Specific words in the task summary * Which operational rules were injected * How many tasks the job needed Compliance lets auditors confirm agents follow company practice — useful for regulated industries. ## Automation The benchmark runner can execute scenarios on a regular schedule. Results land in **Command Center → Benchmarks** with pass rate, compliance, and trends over time, so you can spot regressions early. Recommended cadence: **weekly nightly run** of your full scenario suite, plus **per-PR runs** of the scenarios touching changed specialties or rules. ## What benchmarking is good for | Use case | Why | | -------------------------------------------------------- | ------------------------------------------------------------------------------------ | | Validating a new ticket typology before exposing to prod | Replicate the failure mode, see how the agent handles it | | Comparing two specialties side-by-side | Same scenario, different specialty, see which passes faster + cleaner | | Catching regressions after a rule edit | Trend lines surface failures the moment a previously-passing scenario starts failing | | Tuning LLM models — Sonnet vs Opus vs your own | Same scenario across multiple `--main-engine` overrides | See [Benchmark](/0.12/benchmark/overview) for the runner CLI, scenario format, and validators. # Blacklists in Practice Source: https://docs.2501.ai/0.12/risk/blacklists The programmatic kill-switch — always on, no matter what The [Blacklist](/0.12/configure/blacklist) is the one safety layer that **does not depend on an LLM judgment**. Every command an agent tries to run goes through it first. If the command matches a pattern, it's blocked. ## Always on Blacklists are checked regardless of: * The host the agent is on * The specialty attached to the agent * The operational rules in context * Whether the task is `@2501:investigate` or remediate * Whether the agent is mid-task or starting If the agent retries a blocked command **3 times**, the task is terminated. ## What to blacklist by default Even in non-production environments, block the most destructive primitives: | Pattern | Why | | ------------------------------ | -------------------------------------------------------------- | | `rm -rf /` | Catastrophic by definition | | `rm -rf *` | Scope depends on CWD — agents don't always know where they are | | `shutdown`, `halt`, `poweroff` | Removes the host from the agent's reach | | `mkfs.*`, `dd if=*` | Destroys storage | | `vgremove`, `lvremove` | Destroys logical volumes | | `aws ec2 terminate-instances` | Destroys cloud infrastructure | | `kubectl delete namespace` | Wipes whole environments | Glob wildcards (`*`, `?`) are supported and patterns match anywhere in the command (substring match by default). ## Block interactive tools Agents operate on a prompt, not an interactive shell. Anything that opens a TUI, asks for a password, or expects keystrokes during execution will hang the agent. | Pattern | Why | | ---------------------------- | -------------------------------------------------------- | | `vim`, `nano`, `vi`, `emacs` | TUI editors — agents can't navigate them | | `mysql`, `psql`, `redis-cli` | Interactive REPLs — use `-e ""` or `--batch` | | `python` (alone) | REPL — agents should run scripts with `python script.py` | | `docker exec` (without `-T`) | Allocates a TTY | | `logs -f`, `tail -f` | Streams indefinitely — use bounded reads instead | | `ssh` (without command) | Sub-shell | ## Block calls into known-broken paths If an agent has repeatedly failed with a tool that has a known-bad UX (a CLI with confusing flags, an internal tool that errors silently), blacklist the bad invocation pattern and steer the agent elsewhere via a specialty or operational rule. ## What blacklists are NOT * **Not behavioral guidance.** Use [operational rules](/0.12/configure/operational-rules) to *guide* behavior; use blacklists to **stop** specific commands. * **Not a substitute for permissions.** If the credential can run a destructive command, the blacklist is your last line — but [scoping permissions](/0.12/risk/permissions) is the better first line. * **Not for nuanced cases.** A pattern either matches or it doesn't. If you need "block this in prod but allow in staging," that's an operational rule, not a blacklist entry. # Handling Permissions Source: https://docs.2501.ai/0.12/risk/permissions Limit agents at the credential and provisioning layer, not just the prompt Via [Credentials](/0.12/configure/credentials) and controlled provisioning on target machines, you can hard-cap an agent's blast radius — independent of what the prompt says. ## The credential layer When an agent SSHs into a Linux machine, it does so as a specific user. Give it the **least-privileged user** that still gets the job done. A common pattern: create an `agent-2501` user on each managed host with only the permissions agents need. Critical commands (package installs, service restarts on specific units) go via `sudoers` allowlists. Every authentication layer — SSH, WinRM, MCP, API keys — can be scoped: | Layer | How to limit | | --------------------------- | ----------------------------------------------------------------------------------- | | SSH | Dedicated user + restrictive `sudoers` | | WinRM | Local user with explicit RBAC; gMSA for fine-grained AD-bound privileges | | Cloud CLI (AWS, GCP, Azure) | IAM role attached to the credential — e.g. `ec2:ReadOnly*` for read-only EC2 agents | | API tokens (3rd-party) | Scoped tokens per integration; never reuse a production token in a read-only agent | | MCP servers | RBAC inside the MCP — the MCP itself can enforce read-only tool variants | ## Match permissions to the agent's mission If an agent only needs to read AWS EC2 to produce a resource report, give it an IAM role with **`ec2:Describe*`** only. The agent literally cannot terminate an instance with that role, no matter what the prompt says. This is the most reliable safety layer in the system. Prompts can be jailbroken; IAM cannot. ## When to relax As you grow confident in an agent, you may want to give it more capability. Two things to update **together**: 1. **The credential's permissions** — broaden the IAM role, add `sudoers` entries. 2. **The specialty and rules** — the agent was previously prompted that dangerous operations are out of scope. Update that text to reflect what's now allowed and what's still off-limits. Promote permissions and prompt together. A relaxed prompt with locked-down creds frustrates the agent into retry loops; relaxed creds with a strict prompt is a footgun waiting to fire. ## Quick checklist * [ ] Each host has a dedicated `agent-2501` (or equivalent) user — not root, not the human admin * [ ] Sudo entries are explicit allowlists, not `NOPASSWD: ALL` * [ ] Cloud credentials use scoped IAM roles, not long-lived admin keys * [ ] Read-only investigation agents have IAM/RBAC matching their mission * [ ] When you promote an agent's capability, you update both creds **and** prompts # Read-Only Agents Source: https://docs.2501.ai/0.12/risk/read-only-agents Use the investigate-only ceiling for critical systems and high-risk new use cases Read-only mode constrains an agent to **inspection commands only** — it can describe state, read logs, query APIs, but it cannot change anything. Useful when you want a recommendation, not a remediation. ## Two ways to enable it | Scope | How | Effect | | --------------------- | -------------------------------------------------------------------- | ---------------------------------------------------- | | **One ticket** | Add `@2501:investigate` to the ticket body or a comment | Just this ticket runs read-only | | **A whole specialty** | Pin the [specialty](/0.12/configure/specialties) to investigate-only | Every agent using this specialty is always read-only | Use the specialty ceiling for **large fleets of investigative agents** — for example, 1000 Oracle DB agents that should default to read-only. You can selectively flip individual specialties to remediation later. ## How it works The agent **knows** it's in read-only mode and tries to comply. But it can still emit a command that would alter state. That's where the secondary engine — the [LLM-as-judge](/0.12/understand/engine-agents#secondary-engine--the-copilot) — steps in: it reviews every command before execution. If the command would alter the system in read-only mode, the secondary engine blocks it. Two independent layers of read-only protection: 1. The main engine is **told** it's in read-only mode (prompt-level). 2. The secondary engine **enforces** read-only at every command (runtime check). ### Authentication commands are not blocked Commands that only affect the agent's own credential context — `oc login`, `aws sso login`, `vault login`, and similar — are **not** blocked in read-only mode. These commands do not modify the target system; they configure how the agent authenticates to reach it. Blocking them would prevent the agent from acquiring the credentials it needs to inspect anything. ## Partial resolution When a ticket asks for remediation but a specialty ceiling forced read-only, the job ends as **partial** rather than failed. The ticket gets a public comment explaining what the agent found and why it didn't act, so a human can pick up cleanly. See the [escalation policy](/0.12/core-concepts/gateways#escalation-groups) for how partials route in your ITSM. ## Patterns ### Plan-then-apply Tag the first task `@2501:investigate` and ask for a plan. Read the plan, tweak the rules or prompt if needed, then re-run as a remediation task. ### Critical fleet, safe by default Pin all critical-system specialties to investigate-only. Operators can read every diagnostic the agent produces. When you want a specific repair, drop the investigate ceiling **for one specialty** or one ticket. ### Audit-mode rollout When introducing a new specialty or operational rule for the first time, **always start in investigate-only**. Watch the agent's plans for a week. Lift the ceiling only after the plans look right consistently.