> ## Documentation Index
> Fetch the complete documentation index at: https://docs.2501.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Investigate monitoring alerts automatically

> [!WARNING]
>
> Observability is experimental: its behavior and API may change in later releases.

Observability turns your monitoring alerts into Investigations. Splunk can send alerts to 2501, which stores them as **signals**. Two parts do the work:

* **Sentinel** watches the signals. It runs on a schedule, looks for unusual patterns, and decides for each one whether it needs an Investigation or can wait.
* An **Investigation** works on one problem. It diagnoses with read-only Jobs, fixes the cause with a remediation Job only once the evidence shows it, and checks that the fix worked with another read-only Job before it closes as resolved. When it cannot get there, it is dismissed or escalated to a person.

```mermaid theme={null}
flowchart TD
  A["Splunk alert"] --> S["signals"]
  S --> D["1. Detection and triage<br/>detectors flag unusual patterns"]
  D --> T{"triage (llm)"}
  T -->|snooze| Z["snoozed"]
  T -->|"open an Investigation"| I["2. Investigation<br/>assess (llm)"]
  I <--> J["investigate with read-only Jobs (agentic)"]
  I --> R["3. Remediation<br/>evidence review (llm)"]
  R --> F["Jobs that fix the host (agentic)"]
  F --> E{"4. Resolution<br/>decide (llm)"}
  E -->|"resolve, dismiss or escalate"| C["close"]
```

* **(llm)** - a model call
* **(agentic)** - a Job, which an agent carries out on a host

## Signals and Patterns

Each alert becomes a signal. Signals of the same kind - the same alert rule and message, on any host - form a signal pattern.

```mermaid theme={null}
flowchart TD
  A["Splunk alerts"] -->|"grouped by pattern and host"| B
  subgraph B["signals"]
    direction TB
    S1["connection refused: config-svc<br/>web-1 · 09:00 · 4 signals"]
    S2["connection refused: config-svc<br/>web-1 · 09:05 · 7 signals"]
    S3["connection refused: config-svc<br/>web-2 · 09:05 · 2 signals"]
    S1 ~~~ S2 ~~~ S3
  end
  B --> P["one signal pattern"]
```

## Sentinel

Sentinel runs on a schedule. Its detectors look for unusual patterns, and for each one it finds, the model decides whether to open an Investigation or snooze the pattern.

| Detector | What it looks for |
| - | - |
| **Burst** | A pattern suddenly arrives far more often than usual. |
| **New fingerprint** | A pattern not seen recently: brand new, or back after going quiet. |
| **Host spread** | The same pattern shows up on another host. |
| **Error volume** | Errors across the organization rise sharply: all errors, every pattern. |
| **Untriaged severe** | Severe signals are still waiting for a decision. |

## Investigations

An Investigation assesses what it knows, decides what to do next, and waits for the result, until it can close.

```mermaid theme={null}
flowchart TD
  A["1. assess (llm)<br/>update the hypothesis, confidence and blast radius"] --> D{"2. decide (llm)"}
  D -->|"investigate, remediate or wait"| W["3. wait<br/>until a Job ends, a signal match arrives, or a recheck"]
  W --> A
  D -->|"resolve, dismiss or escalate"| C["close"]
```

In step 2, the Investigation chooses one of:

| Choice | What it does | Then |
| - | - | - |
| **investigate** | Starts read-only Jobs, one per host. | wait |
| **remediate** | Runs the evidence review, then Jobs that change a host. | wait |
| **wait** | Starts nothing. | wait |
| **resolve** | The reported problem stopped. | close |
| **dismiss** | The problem was not real. | close |
| **escalate** | A person needs to look. | close |

An Investigation ends in one of three ways:

* **Resolved** - the problem stopped. This does not prove the cause.
* **Dismissed** - the problem was not real.
* **Escalated** - a person needs to take over. Nobody is notified; check the Investigation's status.

### Fixing Problems Safely

An Investigation starts by diagnosing: it only runs read-only Jobs, which look at hosts without changing them. It moves on to fixing only once its evidence shows the cause with enough confidence. Even then, a separate review checks the fix against that evidence before any Job that changes a host can start, and refuses a fix the evidence does not support. [Operational rules](https://docs.2501.ai/0.20/configure/operational-rules.md) still apply to every Job.

The Jobs of an Investigation never post a [plan](https://docs.2501.ai/0.20/core-concepts/jobs.md#plans): each ends with its answer, and the Investigation decides the next step. A Job that fixes a host uses the [execution mode](https://docs.2501.ai/0.20/core-concepts/agents.md#execution-modes-plan-and-auto) of each agent.

## Turning It On

Observability is off by default. See [Splunk alerts](https://docs.2501.ai/0.20/configure/webhooks.md#splunk-alerts) to turn it on.

## Limits

* Splunk is the only alert source.
* You cannot send messages to an Investigation.
* Escalation notifies nobody.
* An Investigation cannot fix a host through an agent in Plan: that agent only reads, and the Job posts no plan.

## Organization Scoping

Signals, Sentinel decisions and Investigations belong to the [organization](https://docs.2501.ai/0.20/core-concepts/organizations.md) that owns the signal key the alert was sent with.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.