> ## Documentation Index > Fetch the complete documentation index at: https://docs.2501.ai/llms.txt > Use this file to discover all available pages before exploring further. # Observability > Investigate monitoring alerts automatically Observability is experimental: its behavior and API may change in later releases. Observability turns your monitoring alerts into Investigations. Splunk can send alerts to 2501, which stores them as **signals**. Two parts do the work: * **Sentinel** watches the signals. It runs on a schedule, looks for unusual patterns, and decides for each one whether it needs an Investigation or can wait. * An **Investigation** works on one problem. It diagnoses with read-only Jobs, fixes the cause with a remediation Job only once the evidence shows it, and checks that the fix worked with another read-only Job before it closes as resolved. When it cannot get there, it is dismissed or escalated to a person. ``` Splunk alert ──▶ signals │ ▼ 1. Detection and triage detectors flag unusual patterns ──▶ triage (llm): snooze, or open an Investigation │ ┌──────────────────────────────────────────────────────────────────────────────┘ ▼ 2. Investigation assess (llm) ⇄ investigate with read-only Jobs (agentic) │ ▼ 3. Remediation evidence review (llm) ──▶ Jobs that fix the host (agentic) │ ▼ 4. Resolution decide (llm): resolve, dismiss or escalate ──▶ close (llm) a model call (agentic) a Job, which an agent carries out on a host ``` ## Signals and Patterns Each alert becomes a signal. Signals of the same kind - the same alert rule and message, on any host - form a signal pattern. ``` Splunk alerts ──▶ signals, grouped by pattern and host connection refused: config-svc web-1 09:00 4 signals ┐ connection refused: config-svc web-1 09:05 7 signals ├──▶ one signal pattern connection refused: config-svc web-2 09:05 2 signals ┘ ``` ## Sentinel Sentinel runs on a schedule. Its detectors look for unusual patterns, and for each one it finds, the model decides whether to open an Investigation or snooze the pattern. ``` 1. Burst - a pattern suddenly arrives far more often than usual █ ▄ █ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ ▂ ▁ █ █ 2. New fingerprint - a pattern not seen recently: brand new, or back after going quiet ▄ █ ▄ █ ▂ █ █ ▂ ▄ ▂ ▂ █ █ · · · · · █ █ █ █ █ █ · · · · · · · · █ █ █ brand new back after going quiet 3. Host spread - the same pattern shows up on another host host A ▂ ▃ ▂ ▃ ▂ ▃ ▂ ▃ ▂ ▃ ▂ ▃ host B · · · · · · · · · ▃ ▅ █ 4. Error volume - errors across the organization rise sharply █ ▅ █ ▂ ▂ ▁ ▂ ▂ ▁ ▂ ▂ ▁ ▂ ▂ ▁ ▂ ▃ █ █ all errors, every pattern 5. Untriaged severe - severe signals still have no decision ERROR █ · · · · · · · · · · · · · ▶ still waiting ``` ## Investigations An Investigation assesses what it knows, decides what to do next, and waits for the result, until it can close. ``` 1. assess (llm) update the hypothesis, confidence and blast radius 2. decide (llm) choose one: investigate start read-only Jobs, one per host ──▶ 3 remediate evidence review, then Jobs that change a host ──▶ 3 wait start nothing ──▶ 3 resolve the reported problem stopped ──▶ close dismiss the problem was not real ──▶ close escalate a person needs to look ──▶ close 3. wait until a Job ends, a signal match arrives, or a recheck ──▶ back to 1 ``` An Investigation ends in one of three ways: * **Resolved** - the problem stopped. This does not prove the cause. * **Dismissed** - the problem was not real. * **Escalated** - a person needs to take over. Nobody is notified; check the Investigation's status. ### Fixing Problems Safely An Investigation starts by diagnosing: it only runs read-only Jobs, which look at hosts without changing them. It moves on to fixing only once its evidence shows the cause with enough confidence. Even then, a separate review checks the fix against that evidence before any Job that changes a host can start, and refuses a fix the evidence does not support. [Operational rules](/0.17/configure/operational-rules) still apply to every Job. ## Turning It On Observability is off by default. See [Splunk alerts](/0.17/configure/webhooks#splunk-alerts) to turn it on. ## Limits * Splunk is the only alert source. * You cannot send messages to an Investigation. * Escalation notifies nobody. ## Organization Scoping Signals, Sentinel decisions and Investigations belong to the [organization](/0.17/core-concepts/organizations) that owns the signal key the alert was sent with. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.