Alerting and incidents

Alert model

The alerter evaluates durable database state and creates one lifecycle record for each active condition. Alert policies are retired; routing now lives in tenant notification settings and each monitor.

Availability lifecycle

1. A check failure first passes through the monitor's consecutive-failure confirmation and, when locations are selected, its location failure quorum.
2. The alerter opens an availability alert only when effective state is down, the monitor is enabled, and maintenance/rollup rules do not mute it.
3. The initial notification is sent through eligible routes after each route's configured delay.
4. While the condition remains down, reminders are sent at the tenant reminder interval.
5. When the monitor is fully up, the alert resolves and resolution notifications go only to channels that previously received that alert.

suspect, unknown, and a newly degraded monitor do not open an availability alert. If a monitor moves from down to degraded after an alert opened, the alert remains open until the aggregate is fully up.

Location-aware availability alerts include failing-location context so operators can distinguish a regional failure from a broad outage.

Default and custom notification routing

Tenant default routes

An ordered list of active alert channels, each with channel_id, display metadata, and a non-negative delay_seconds.

Monitor default mode

The monitor inherits the tenant's current default routes.

Monitor custom mode

Only the monitor's assigned active channels are considered, with the delay stored on each assignment.

Delays implement escalation: a zero-delay route fires immediately, while a later route fires only if the alert is still open when its delay elapses. Resolution notifications are limited to routes that actually fired, avoiding a recovery message on a channel that never saw the outage.

Routing that resolves to nothing is not an error — an alert still opens, appears on the dashboard, and drives incidents and status pages; it simply notifies nobody. Because that is silent by design, every monitor read reports its effective reachability in alert_routing, and the dashboard counts affected monitors in ops_summary.unrouted_monitors.

Notification channels

Generic webhooks include X-Probara-Event-Type and X-Probara-Idempotency-Key. When an HMAC secret is configured they also include X-Probara-Signature in sha256=... form. Receivers should verify the signature against the raw body and deduplicate by the idempotency key.

• Create a channel from a registered plugin and save its plugin-specific configuration.
• Use the channel test action before assigning production monitors. Tests send a real notification and require write permission. Email tests are served by the API process, so they need the same platform SMTP configuration as the alerter; without it the test reports mailer not configured even when alert email is being delivered.
• Activate or deactivate the channel. Inactive channels remain configured but are skipped for delivery.
• Assign it in tenant defaults or in a monitor's custom routing.

Email rendering and templates

With no template overrides, an email channel sends a multipart/alternative message: a branded HTML part and a plain-text part rendered from the same data, so text-only clients and mail archives never see markup. The HTML part is self-contained — inline styles, no remote images, no data: URIs — and adapts to the reader's light or dark theme. Both parts lead with a one-sentence summary of what happened, then the measurement that tripped the alert (observed vs. baseline latency, metric vs. threshold, certificate days remaining), the verbatim probe error, the likely root cause when the dependency graph identifies one, the failing locations of a multi-location monitor, and a metadata block with the trigger time, elapsed duration, and failed-check count.

Set APP_BASE_URL to the public origin of the operator UI and every alert email carries an "open the monitor" button (mesh-edge alerts link to the mesh matrix instead). Without it the button is omitted rather than pointing at a guessed host. SMTP_FROM_NAME sets the From display name and defaults to Probara Alerts.

Templates receive the raw alert fields (monitor_name, status, kind, failure_count, last_error, triggered_at, resolved_at, tenant_id, metric_name, metric_value, threshold_value, baseline_latency_ms, observed_latency_ms, anomaly_score, root_cause_monitor_name, source_location_name, target_location_name, failing_locations) plus the presentation values the built-in email uses, so an override can reuse the same wording: label, status_label, summary, duration, triggered_at_human, resolved_at_human, action_url, action_label, and accent_color. A template that fails to parse or execute falls back to the built-in rendering — the alert still goes out and the error is logged.

Group alert rollup

A group monitor chooses one of two availability alert strategies. per_monitor lets members alert normally and suppresses an additional group alert. group suppresses member availability alerts covered by the group and emits one group-level alert while any effective member is down.

Under group rollup the group's routing is the routing that matters: a suppressed member's own channels never fire, so its alert_routing.source reports group_rollup and names the group. A member covered by several rollup groups is reachable if any of them delivers, since each group alerts on its own.

Latency anomaly alerts

The anomaly detector compares recent successful monitor-source latency with hourly successful-history baselines. It opens an alert only when latency exceeds the sensitivity threshold, minimum percent delta, and minimum breach duration configured for the tenant.

• The platform-wide latency detector must be enabled.
• The tenant's latency_anomaly_enabled setting must be true.
• The monitor must be enabled, must not be a group, and must not be in maintenance.
• Enough retained successful data must exist to form a baseline.
• The alert resolves automatically when the breach is no longer present.

Latency anomalies are independent from availability. A monitor can be up while unusually slow and therefore have a latency alert without an availability alert.

Host metric, mesh, and TLS expiry alerts

An agent monitor defines metric_rules: up to 50 threshold rules over any metric its collector reports, compared in the metric's native unit (*.utilization metrics are ratios, so 90% is 0.9), with optional attribute filters and an optional sustained-for duration that must hold for the whole window before the alert opens. A rule without filters fans out per matching series — one filesystem rule opens one host_metric alert per breaching mountpoint.

Each alert is keyed by its canonical series key (for example system.filesystem.utilization{device=/dev/sda1,mode=rw,mountpoint=/data,type=ext4}) and resolves independently when the series returns within the rule or the rule is removed. Evaluation is freshness-bounded to three monitor intervals (90-second floor), so a dead collector's last stale readings cannot keep metric alerts open — the availability alert covers that outage. Alerts migrated from the retired metric_thresholds fields keep working; their legacy names still render on existing alert records.

A location mesh edge opens a mesh_edge alert after its directional failure threshold. The reverse direction is a different condition. Mesh alerts use tenant default channels.

An HTTP monitor with tls_min_days_valid opens a tls_expiry alert when its most recently observed certificate has fewer remaining validity days than the threshold. The certificate window does not fail the check: the endpoint stays up and the warning is a distinct alert kind, so an aging certificate is distinguishable from an outage. The alert resolves when a renewed certificate is observed or the threshold is removed. A fully expired certificate fails the TLS handshake itself and therefore surfaces as a regular availability alert.

Maintenance windows and snoozes

Maintenance windows are classified as upcoming, active, or past. Checks continue to execute and states continue to change; the alert opening/notification path is muted for covered monitors during the active interval.

A monitor can also be snoozed until an explicit timestamp or for a duration in minutes. Snooze is a concise one-monitor maintenance action, not a pause in data collection.

Dependency-aware alerting

When a downstream availability alert opens, Probara inspects its dependency graph and annotates the alert with the deepest currently down upstream monitor. Ties prefer the upstream condition that became down earlier. The annotation is recomputed every evaluation tick and cleared when no qualifying cause remains; it is always on and always visible on the alert, in notifications ("likely root cause") and in the operator UI.

Whether the annotation also suppresses the downstream page is a policy. It is off by default: enable dependency_suppression_enabled in the workspace notification settings to page root causes only, and override it per monitor with dependency_suppression (inherit, on, off) in the monitor's Alerting section — off keeps a critical monitor paging whatever the workspace says.

• A suppressed downstream alert still opens, still counts on the dashboard, still resolves and still drives incidents and status pages; only its notifications are withheld. The API reports it with suppression_reason: "dependency", the alert list and dashboard flag it, and GET /api/v1/alerts?suppressed=true lists exactly those alerts.
• The root cause's own DOWN and reminder notifications carry an Also affecting section naming the suppressed downstream monitors (impacted_monitors / impacted_count; the API exposes impacted_count on the alert), recomputed on every send.
• When the upstream recovers but the downstream is still down, the downstream stays quiet for dependency_suppression_grace_seconds (default 120) so a monitor that recovers one check later never pages. After the grace it pages as a normal DOWN, with escalation-tier delays counted from that moment rather than from when the outage began.
• A downstream that recovers while suppressed was never announced, so no recovery notification is sent for it. A downstream that paged before its upstream was detected keeps its recovery notification but stops sending reminders while the upstream explains it.
• Suppression is decided at dispatch, from live state, by the same predicate the API uses to compute suppression_reason; it is not part of alert_routing, which describes static routing configuration.

Incidents

Timeline entries can be system events, internal notes, or public updates. Publishing an incident to a status page exposes only the page-relevant subset: the incident and page must share selected linked monitors. Unpublishing removes it from that page without deleting the incident.

1. Create an incident manually, or enable automatic creation for supported alert paths.
2. Attach the alerts and affected monitors used for internal context.
3. Set severity, owner, and lifecycle status as the response progresses.
4. Add internal notes for operators and public updates for subscribers.
5. Publish to each relevant status page, then unpublish or resolve when appropriate.

AI-assisted incident analysis

When an effective tenant LLM configuration is enabled, an incident analysis request queues asynchronous work. Its result progresses through pending, ready, or failed and can include a summary, probable root cause, contributing factors, recommended actions, confidence, and evidence.

Retired alert-policy API

Legacy /alert-policies endpoints return HTTP 410 Gone. Configure tenant notification defaults, monitor-specific routing, consecutive failure thresholds, agent metric thresholds, and latency-anomaly settings instead.