Alerting and incidents

Alert model

The alerter evaluates durable database state and creates one lifecycle record for each active condition. Alert policies are retired; routing now lives in tenant notification settings and each monitor.

On the Alerts page, users who can write work the queue in bulk: tick open alerts (Shift-click selects a range, the header box selects every open alert on the page, Escape clears), then Acknowledge or Resolve from the bar. Each action sends only the selected alerts it applies to, and alerts that changed state meanwhile are reported as skipped rather than failing the batch. Bulk acknowledge and resolve notify and record incident recovery as the single actions do; they skip alerts of deleted monitors (reported as not found), which the list no longer shows. With a row focused, A acknowledges and R resolves that alert; these keys work only while focus is on an alert row, never while typing or with Ctrl, Alt or Cmd held, and a press on an alert that is already past that step says so instead of acting. Every view except Open sorts on the server, so a column sort orders every matching alert, not just the page; those views default to newest first, while Open (which loads the whole queue) defaults to active first. Under a one-status filter (Active, Acknowledged, Resolved) the Status column does not sort, since every row has the same status.

Availability lifecycle

1. A check failure first passes through the monitor's consecutive-failure confirmation and, when locations are selected, its location failure quorum. A check's error and failure are both failing evidence, including a check's own assertions: a SIP test call failing a call-quality threshold (max_loss_pct, min_mos, …) is a failure like any other, while its warn_* thresholds only add warnings to the check and never alert.
2. The alerter opens an availability alert only when effective state is down, the monitor is enabled, and maintenance/rollup rules do not mute it.
3. The initial notification is sent through eligible routes after each route's configured delay.
4. While the condition remains down, reminders are sent at the tenant reminder interval.
5. When the monitor is fully up, the alert resolves and resolution notifications go only to channels that previously received that alert.

suspect, unknown, and a newly degraded monitor do not open an availability alert. If a monitor moves from down to degraded after an alert opened, the alert remains open until the aggregate is fully up.

Location-aware availability alerts include failing-location context so operators can distinguish a regional failure from a broad outage.

Default and custom notification routing

Tenant default routes

An ordered list of active alert channels, each with channel_id, display metadata, and a non-negative delay_seconds.

Monitor default mode

The monitor inherits the tenant's current default routes.

Monitor custom mode

Only the monitor's assigned active channels are considered, with the delay stored on each assignment.

Delays implement escalation: a zero-delay route fires immediately, while a later route fires only if the alert is still open when its delay elapses. Resolution notifications are limited to routes that actually fired, avoiding a recovery message on a channel that never saw the outage.

Routing that resolves to nothing is not an error — an alert still opens, appears on the dashboard, and drives incidents and status pages; it simply notifies nobody. Because that is silent by design, every monitor read reports its effective reachability in alert_routing, and the dashboard counts affected monitors in ops_summary.unrouted_monitors.

Notification channels

Every channel renders the same wording — headline, one-sentence summary, measurement, failing locations, likely root cause and blast radius — so a page, a chat message and an email about one alert never disagree. The opt-in suspected-cause follow-up is worded the same way on every channel that receives it. Deep links into the operator UI appear on every channel when APP_BASE_URL is set and are omitted when it is not.

Generic webhooks include X-Probara-Event-Type and X-Probara-Idempotency-Key. The event type is created, reminder, resolved or, when a workspace opts in, suspected_cause, whose body adds alert.suspected_cause (triage_id, key, title, confidence, actions) and keeps alert.status at the alert's own active or acknowledged. Branch on the event type, not the status, and ignore event types you do not handle. When an HMAC secret is configured they also include X-Probara-Signature in sha256=... form. Receivers should verify the signature against the raw body and deduplicate by the idempotency key. Custom headers cannot set these three in any letter case: saving such a channel is rejected, and a configuration stored before that check delivers without them.

• Create a channel from a registered plugin and save its plugin-specific configuration. Updates are validated by the plugin against the merged configuration, including kept secrets, so an edit cannot move a channel to a host create would have rejected.
• Use the channel test action before assigning production monitors. Tests send a real notification and require write permission. PagerDuty and Opsgenie tests open an informational alert and resolve it immediately, which may still notify whoever is on call. Opsgenie processes requests asynchronously, so its test waits (up to four seconds) for the alert to exist before closing it; if Opsgenie is slower than that, the test reports the alias so you can close the alert by hand. Email tests are served by the API process, so they need the same platform SMTP configuration as the alerter; without it the test reports mailer not configured even when alert email is being delivered.
• Activate or deactivate the channel. Inactive channels remain configured but are skipped for delivery.
• Assign it in tenant defaults or in a monitor's custom routing.

Delivery failures are classified. A rejected configuration — revoked key, deleted webhook, unknown chat, invalid number, any other 4xx — is permanent: it is logged once and not retried, so a broken channel does not re-fail every evaluation cycle. Timeouts, 5xx and 429 responses are retried; a provider Retry-After is honoured (capped at ten minutes) on the asynchronous path. An SMS send is retried whenever any recipient failed transiently, even if another was rejected for good, and waits for the longest delay any recipient's provider response asked for. The retry texts only the recipients Twilio has not accepted yet: each accepted recipient is recorded in the database the moment Twilio answers, so it is skipped whichever alerter replica or delivery worker picks the retry up, after a restart included. Redirects are never followed and count as permanent failures. With asynchronous dispatch, a retried trigger, reminder or acknowledgement is dropped once its alert has resolved, and one already being delivered holds the alert until it finishes, so resolving waits for it (at most the 15-second delivery timeout). Either way the resolve reaches the provider last, so a delayed or concurrent trigger can never leave a PagerDuty or Opsgenie alert open after recovery. Acknowledgement is ordered the same way: if an operator acknowledges while a paging trigger is still waiting on a retry, the acknowledgement that went ahead is a no-op at the provider, so the worker acknowledges again right after the late trigger lands, and the incident does not keep escalating.

Email rendering and templates

With no template overrides, an email channel sends a multipart/alternative message: a branded HTML part and a plain-text part rendered from the same data, so text-only clients and mail archives never see markup. The HTML part is self-contained — inline styles, no remote images, no data: URIs — and adapts to the reader's light or dark theme. Both parts lead with a one-sentence summary of what happened, then the measurement that tripped the alert (observed vs. baseline latency, metric vs. threshold, certificate days remaining), the verbatim probe error, the likely root cause when the dependency graph identifies one, the failing locations of a multi-location monitor, and a metadata block with the trigger time, elapsed duration, and failed-check count.

Set APP_BASE_URL to the public origin of the operator UI and every alert email carries an "open the monitor" button (mesh-edge alerts link to the mesh matrix instead; a suspected-cause follow-up links to that one alert on the Alerts page, in its workspace). Without it the button is omitted rather than pointing at a guessed host. SMTP_FROM_NAME sets the From display name and defaults to Probara Alerts.

Templates receive the raw alert fields (event_type, monitor_name, status, kind, failure_count, last_error, triggered_at, resolved_at, tenant_id, metric_name, metric_value, threshold_value, baseline_latency_ms, observed_latency_ms, anomaly_score, root_cause_monitor_name, source_location_name, target_location_name, failing_locations) plus the presentation values the built-in email uses, so an override can reuse the same wording: label, status_label, summary, duration, triggered_at_human, resolved_at_human, action_url, action_label, and accent_color. A template that fails to parse or execute falls back to the built-in rendering — the alert still goes out, once: the failure is logged (and returned by a test send, so you see it there) but never retried, because retrying would only mail the whole to list the same fallback again. Overrides apply to pages only: a suspected-cause follow-up always uses the built-in rendering and the subject [Suspected Cause] <monitor>, so a template written for outages (status is still active on a follow-up) never turns it into a second DOWN email.

Group alert rollup

A group monitor chooses one of two availability alert strategies. per_monitor lets members alert normally and suppresses an additional group alert. group suppresses member availability alerts covered by the group and emits one group-level alert while any effective member is down.

Under group rollup the group's routing is the routing that matters: a suppressed member's own channels never fire, so its alert_routing.source reports group_rollup and names the group. A member covered by several rollup groups is reachable if any of them delivers, since each group alerts on its own.

Latency anomaly alerts

The anomaly detector compares recent successful monitor-source latency with hourly successful-history baselines. It opens an alert only when latency exceeds the sensitivity threshold, minimum percent delta, and minimum breach duration configured for the tenant.

• The platform-wide latency detector must be enabled.
• The tenant's latency_anomaly_enabled setting must be true.
• The monitor must be enabled, must not be a group, and must not be in maintenance.
• Enough retained successful data must exist to form a baseline.
• The alert resolves automatically when the breach is no longer present.

Latency anomalies are independent from availability. A monitor can be up while unusually slow and therefore have a latency alert without an availability alert.

Host metric, mesh, and TLS expiry alerts

An agent monitor defines metric_rules: up to 50 threshold rules over any metric its collector reports, compared in the metric's native unit (*.utilization metrics are ratios, so 90% is 0.9), with optional attribute filters and an optional sustained-for duration that must hold for the whole window before the alert opens. A rule without filters fans out per matching series — one filesystem rule opens one host_metric alert per breaching mountpoint.

Each alert is keyed by its canonical series key (for example system.filesystem.utilization{device=/dev/sda1,mode=rw,mountpoint=/data,type=ext4}) and resolves independently when the series returns within the rule or the rule is removed. Evaluation is freshness-bounded to three monitor intervals (90-second floor), so a dead collector's last stale readings cannot keep metric alerts open — the availability alert covers that outage. Alerts migrated from the retired metric_thresholds fields keep working; their legacy names still render on existing alert records.

A location mesh edge opens a mesh_edge alert after its directional failure threshold. The reverse direction is a different condition. Mesh alerts use tenant default channels.

An HTTPS or SIP-over-TLS monitor with tls_min_days_valid opens a tls_expiry alert when its most recently observed certificate has fewer remaining validity days than the threshold. The certificate window does not fail the check: the endpoint stays up and the warning is a distinct alert kind, so an aging certificate is distinguishable from an outage. The alert resolves when a renewed certificate is observed or the threshold is removed. A fully expired certificate fails the TLS handshake itself and therefore surfaces as a regular availability alert.

Maintenance windows and snoozes

Maintenance windows are classified as upcoming, active, or past. Checks continue to execute and states continue to change; the alert opening/notification path is muted for covered monitors during the active interval. GET /maintenance-windows?status= lists each class in the order an operator needs it: active windows by soonest end, upcoming by soonest start, past (and the unfiltered list) by most recent end, so a page cut never drops the next window.

A monitor can also be snoozed until an explicit timestamp or for a duration in minutes. Snooze is a concise one-monitor maintenance action, not a pause in data collection.

Dependency-aware alerting

When a downstream availability alert opens, Probara inspects its dependency graph and annotates the alert with the deepest currently down upstream monitor. Ties prefer the upstream condition that became down earlier. The annotation is recomputed every evaluation tick and cleared when no qualifying cause remains; it is always on and always visible on the alert, in notifications ("likely root cause") and in the operator UI.

Whether the annotation also suppresses the downstream page is a policy. It is off by default: enable dependency_suppression_enabled in the workspace notification settings to page root causes only, and override it per monitor with dependency_suppression (inherit, on, off) in the monitor's Alerting section (shown once the monitor has upstream dependencies, or when a non-default value is stored) — off keeps a critical monitor paging whatever the workspace says.

• A suppressed downstream alert still opens, still counts on the dashboard, still resolves and still drives incidents and status pages; only its notifications are withheld. The API reports it with suppression_reason: "dependency", the alert list and dashboard flag it, and GET /api/v1/alerts?suppressed=true lists exactly those alerts.
• The root cause's own DOWN and reminder notifications carry an Also affecting section naming the suppressed downstream monitors (impacted_monitors / impacted_count; the API exposes impacted_count on the alert), recomputed on every send.
• When the upstream recovers but the downstream is still down, the downstream stays quiet for dependency_suppression_grace_seconds (default 120) so a monitor that recovers one check later never pages. After the grace it pages as a normal DOWN, with escalation-tier delays counted from that moment rather than from when the outage began.
• A downstream that recovers while suppressed was never announced, so no recovery notification is sent for it. A downstream that paged before its upstream was detected keeps its recovery notification but stops sending reminders while the upstream explains it.
• Suppression is decided at dispatch, from live state, by the same predicate the API uses to compute suppression_reason; it is not part of alert_routing, which describes static routing configuration.

Incidents

Timeline entries can be system events, internal notes, or public updates. Publishing an incident to a status page exposes only the page-relevant subset: the incident and page must share selected linked monitors. Unpublishing removes it from that page without deleting the incident.

1. Create an incident manually, or enable automatic creation for supported alert paths.
2. Attach the alerts and affected monitors used for internal context.
3. Set severity, owner, and lifecycle status as the response progresses.
4. Add internal notes for operators and public updates for subscribers.
5. Publish to each relevant status page, then unpublish or resolve when appropriate.

AI-assisted incident analysis

When workspace local AI is enabled, incident analysis queues asynchronous work through AI_RCA. Evidence spans 15 minutes before the earliest linked monitor alert (incident creation when none is linked) through 60 minutes after it, capped at analysis request time. Results from jobs that expired before a worker ran them (result_source=platform) are not evidence. Linked mesh-edge alerts are not analysed; they are listed by alert and location ID under incomplete evidence. Full-window summaries, sampled transitions, configured upstream checks, overlapping maintenance, allowlisted timings/certificate facts and fresh host utilization support monitor-level decisions. Results retain summary, suspected cause, contributing factors, actions, confidence and evidence, with structured model/score/reference metadata. Ambiguous or unsupported incidents complete as ready with insufficient evidence. A local AI that is temporarily unavailable (queue full with HTTP 429, model loading or restarting with 502/503/504, connection refused) is retried automatically, 1 and then 2 minutes later or after the service's Retry-After (at most 2 minutes), for up to 3 deliveries; other service failures, and an analysis still unavailable on its third delivery, become failed with a retry action. Each delivery runs under one deadline (LOCAL_AI_TIMEOUT_SECONDS plus 30 seconds); after repeated errors the run is recorded as failed, and database-connected workers fail any analysis still pending after 20 minutes (30 minutes if a run still holds it), which outlasts the unavailable-service retries. With no such worker running, pending analyses wait until one returns. Historical provider results remain visible as legacy analyses.

Below a ready analysis, editors and admins answer Was this analysis right? to record a training label for the local model. Confirm keeps the suspected cause and each monitor's shown answer. Correct picks the actual cause; each monitor's answer follows it only where that monitor's own evidence supports it, otherwise it becomes insufficient evidence. With more than one monitor, per-monitor answers can be set individually. The label stores the exact evidence the model saw, copied server-side, and a later change replaces it. Viewers see the existing label read-only. Legacy analyses, analyses that sent no monitor decisions, and analyses completed before labels existed cannot be labelled; run a new analysis. Labels do not change the incident, alerts or monitor state. See AI label endpoints.

Automatic alert triage

With Automatic alert triage turned on in Settings → AI, each new open alert gets one suspected cause from the same local model, catalogue and evidence checks as incident analysis. Triage is shown next to the alert, and can optionally be sent once to the channels already paged for it (see suspected-cause follow-ups below). SLA reports roll monitor outages up by these suspected causes at medium or high confidence, operator labels included (a low-confidence one counts as inconclusive).

Database-connected platform workers run a relay every 15 seconds. It claims alerts that are open (active or acknowledged, so acknowledging in the first minute still gets a triage), triggered within the last 6 hours, and older than the settle wait. The settle wait is the workspace delay, but at least two check intervals of the alert's monitor, so the alert's own failing checks are part of the evidence. The two-interval part is capped at one hour, so monitors that check less often than every 30 minutes are triaged an hour after the alert opens. An alert that resolves within the wait is never triaged, and alerts from before auto-triage was turned on are not triaged once they are more than 6 hours old. Claiming is oldest alert first. Each workspace has an hourly cap on triages, counted over the last hour with skipped ones excluded. All workspaces also share a platform-wide limit of 200 queued triages, so an alert storm cannot queue more work than the local AI can process before the jobs expire.

Evidence covers the alert's monitor (a group alert expands to its members), its configured upstreams, and the dependency root that suppressed it, if any. The window runs from 15 minutes before the alert triggered to 60 minutes after, capped at claim time. Window, sampling and exclusion rules are the same as for incident analysis. Each alert has one triage. It is not re-run while the alert stays open, and a reopened condition is a new alert.

The Alerts table shows the triage as a pill in the monitor cell, and incident detail shows it next to each linked alert. Pills update the next time the list loads, not live: the alert stream carries alert changes only. Triage runs only while auto-triage and workspace AI are both on. Turning it off, setting AI_TRIAGE_ENABLED=false, or setting LOCAL_AI_DISABLED=true stops new claims, and triages already queued finish as skipped without calling the model. A job that never reached the queue is published again up to twice, 3 minutes apart. Two alerts of different kinds on one monitor triaged at the same moment can both run, because the cooldown only counts triages that are already ready.

Suspected-cause follow-ups. With Send suspected causes to alert channels also on (it needs auto-triage and workspace AI), the alerter sends one advisory follow-up per channel that was already paged for the alert, after the page went out. Pages are never held back for it. The table lists when it is sent and when it is not.

The follow-up is titled Suspected Cause: <monitor> and says which page it follows up on (for example 'Down alert' or 'CPU Usage High', so two host-metric alerts on one host stay distinct), the suspected cause, its confidence, the catalogue's suggested first check, the trigger time and how long the alert has been open, plus the dependency root cause when one is known. It ends with a link to that alert on the Alerts page when APP_BASE_URL is set. Email follow-ups are separate messages with the subject [Suspected Cause] <monitor>, never threaded replies, and ignore channel template overrides. SMS channels send it as one more text per recipient. Reminders, acknowledgements and the resolve keep their own schedule. Under asynchronous dispatch the order is best effort: a page still on the delivery worker's retry backoff after 3 minutes can arrive after its follow-up, and a page the worker dropped for good still counts as paged.

Editors and admins can label a ready triage from the alert's row menu (⋮): Confirm cause or Correct cause…, or Confirm no cause and Name the cause… when the model abstained. Once labelled, a Change label… button (tag icon) takes the menu's place. Correcting opens the same per-monitor control as incident labels. The label stores the evidence the model saw, copied server-side, is included in the training export, and shares a split group with the earliest incident that links the alert. A triage that sent no decisions cannot be labelled. Labels never change the alert; a correction made before the follow-up went out cancels it. A label does replace the model's cause in SLA report suspected causes computed afterwards (an issued report keeps the cause it froze). See AI label endpoints.

Retired alert-policy API

Legacy /alert-policies endpoints return HTTP 410 Gone. Configure tenant notification defaults, monitor-specific routing, consecutive failure thresholds, agent metric thresholds, and latency-anomaly settings instead.