Observability is experimental: its behavior and API may change in later releases.
- Sentinel watches the signals. It runs on a schedule, looks for unusual patterns, and decides for each one whether it needs an Investigation or can wait.
- An Investigation works on one problem. It diagnoses with read-only Jobs, fixes the cause with a remediation Job only once the evidence shows it, and checks that the fix worked with another read-only Job before it closes as resolved. When it cannot get there, it is dismissed or escalated to a person.
Signals and Patterns
Each alert becomes a signal. Signals of the same kind - the same alert rule and message, on any host - form a signal pattern.
Sentinel
Sentinel runs on a schedule. Its detectors look for unusual patterns, and for each one it finds, the model decides whether to open an Investigation or snooze the pattern.
Investigations
An Investigation assesses what it knows, decides what to do next, and waits for the result, until it can close.
An Investigation ends in one of three ways:
- Resolved - the problem stopped. This does not prove the cause.
- Dismissed - the problem was not real.
- Escalated - a person needs to take over. Nobody is notified; check the Investigation’s status.
Fixing Problems Safely
An Investigation starts by diagnosing: it only runs read-only Jobs, which look at hosts without changing them. It moves on to fixing only once its evidence shows the cause with enough confidence. Even then, a separate review checks the fix against that evidence before any Job that changes a host can start, and refuses a fix the evidence does not support. Operational rules still apply to every Job.
Turning It On
Observability is off by default. See Splunk alerts to turn it on.
Limits
- Splunk is the only alert source.
- You cannot send messages to an Investigation.
- Escalation notifies nobody.
Organization Scoping
Signals, Sentinel decisions and Investigations belong to the organization that owns the signal key the alert was sent with.
