Observability

Observability is experimental: its behavior and API may change in later releases.

Observability turns your monitoring alerts into Investigations. Splunk can send alerts to 2501, which stores them as signals. Two parts do the work:

Sentinel watches the signals. It runs on a schedule, looks for unusual patterns, and decides for each one whether it needs an Investigation or can wait.
An Investigation works on one problem. It diagnoses with read-only Jobs, fixes the cause with a remediation Job only once the evidence shows it, and checks that the fix worked with another read-only Job before it closes as resolved. When it cannot get there, it is dismissed or escalated to a person.

Splunk alert ──▶ signals
                    │
                    ▼
1. Detection and triage     detectors flag unusual patterns ──▶ triage (llm): snooze, or open an Investigation
                                                                                                   │
                    ┌──────────────────────────────────────────────────────────────────────────────┘
                    ▼
2. Investigation            assess (llm) ⇄ investigate with read-only Jobs (agentic)
                    │
                    ▼
3. Remediation              evidence review (llm) ──▶ Jobs that fix the host (agentic)
                    │
                    ▼
4. Resolution               decide (llm): resolve, dismiss or escalate ──▶ close

(llm)      a model call
(agentic)  a Job, which an agent carries out on a host

Signals and Patterns

Each alert becomes a signal. Signals of the same kind - the same alert rule and message, on any host - form a signal pattern.

Splunk alerts ──▶ signals, grouped by pattern and host

    connection refused: config-svc    web-1   09:00   4 signals   ┐
    connection refused: config-svc    web-1   09:05   7 signals   ├──▶  one signal pattern
    connection refused: config-svc    web-2   09:05   2 signals   ┘

Sentinel

Sentinel runs on a schedule. Its detectors look for unusual patterns, and for each one it finds, the model decides whether to open an Investigation or snooze the pattern.

1. Burst - a pattern suddenly arrives far more often than usual

                                  █
                                ▄ █
    ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ █ █

2. New fingerprint - a pattern not seen recently: brand new, or back after going quiet

                ▄ █                                       ▄ █
              ▂ █ █               ▂ ▄ ▂                 ▂ █ █
    · · · · · █ █ █               █ █ █ · · · · · · · · █ █ █
        brand new                   back after going quiet

3. Host spread - the same pattern shows up on another host

    host A    ▂ ▃ ▂ ▃ ▂ ▃ ▂ ▃ ▂ ▃ ▂ ▃
    host B    · · · · · · · · · ▃ ▅ █

4. Error volume - errors across the organization rise sharply

                                  █
                                ▅ █
    ▂ ▂ ▁ ▂ ▂ ▁ ▂ ▂ ▁ ▂ ▂ ▁ ▂ ▃ █ █      all errors, every pattern

5. Untriaged severe - severe signals still have no decision

    ERROR  █ · · · · · · · · · · · · · ▶  still waiting

Investigations

An Investigation assesses what it knows, decides what to do next, and waits for the result, until it can close.

1. assess   (llm)  update the hypothesis, confidence and blast radius
2. decide   (llm)  choose one:
                     investigate   start read-only Jobs, one per host             ──▶ 3
                     remediate     evidence review, then Jobs that change a host  ──▶ 3
                     wait          start nothing                                  ──▶ 3
                     resolve       the reported problem stopped                   ──▶ close
                     dismiss       the problem was not real                       ──▶ close
                     escalate      a person needs to look                         ──▶ close
3. wait            until a Job ends, a signal match arrives, or a recheck ──▶ back to 1

An Investigation ends in one of three ways:

Resolved - the problem stopped. This does not prove the cause.
Dismissed - the problem was not real.
Escalated - a person needs to take over. Nobody is notified; check the Investigation’s status.

Fixing Problems Safely

An Investigation starts by diagnosing: it only runs read-only Jobs, which look at hosts without changing them. It moves on to fixing only once its evidence shows the cause with enough confidence. Even then, a separate review checks the fix against that evidence before any Job that changes a host can start, and refuses a fix the evidence does not support. Operational rules still apply to every Job.

Turning It On

Observability is off by default. See Splunk alerts to turn it on.

Limits

Splunk is the only alert source.
You cannot send messages to an Investigation.
Escalation notifies nobody.

Organization Scoping

Signals, Sentinel decisions and Investigations belong to the organization that owns the signal key the alert was sent with.

Getting Started

Understand 2501

Concepts

Signals and Patterns

Sentinel

Investigations

Fixing Problems Safely

Turning It On

Limits

Organization Scoping

​Signals and Patterns

​Sentinel

​Investigations

​Fixing Problems Safely

​Turning It On

​Limits

​Organization Scoping

Signals and Patterns

Sentinel

Investigations

Fixing Problems Safely

Turning It On

Limits

Organization Scoping