Architecture

System shape

Probara separates control-plane APIs, scheduling, check execution, alert evaluation, and public presentation. PostgreSQL is the source of truth; NATS carries asynchronous work and live invalidation.

Web / API client
      │
      ▼
     API ───────────────► PostgreSQL ◄────────────── Alerter
      │                         ▲                        │
      │                         │                        ├─ notifications
      ▼                         │                        └─ incidents
Scheduler ── check.jobs ──► Worker(s)
      ▲                         │
      └──── check.results ◄─────┘

PostgreSQL + statuspage.updates ──► Status-page service ──► Public visitors

The boundaries are deliberate. The scheduler does not execute checks or evaluate alert policies. Workers do not need direct PostgreSQL access. The alerter derives lifecycle decisions from stored state rather than from an in-memory result stream alone.

Service responsibilities

Persistence and consistency

PostgreSQL

Stores tenant-scoped configuration, check results, current and per-location state, alerts, incidents, maintenance, notification settings, rollups, and audit records.

NATS JetStream

Provides durable streams for check jobs, check results, alert events, and optional asynchronous analysis jobs.

Core NATS

Provides low-latency status-page invalidation and worker location heartbeats where replay is not required.

Schedulers claim due rows with database locking that skips rows already claimed by another scheduler. The queue path is at-least-once, so result ingestion is designed to deduplicate repeated job/location results. Clients must not infer exactly-once execution from a single successful publish.

Monitor execution lifecycle

1. The scheduler selects an enabled, due active monitor and advances its next due time.
2. For a monitor without selected locations, it publishes one job to the default fleet. With selected locations, it publishes one job per enabled location.
3. A worker validates the target and configuration, performs the check within the monitor timeout, and publishes a result.
4. The scheduler persists each result and updates per-location state.
5. For multi-location monitoring, it derives a monitor-level state using the configured failure quorum.
6. The stored state is exposed to dashboards and status pages and evaluated by the alerter.

Run now follows the persisted job/result path. Test is an API request/reply path designed for configuration validation and does not create history or alerts.

Monitor state model

State is intentionally richer than a binary up/down flag. Temporal confirmation and location quorum answer different questions and are applied at different layers.

For a single execution stream, failure and error outcomes increment the consecutive-failure counter. Before the configured threshold the monitor is suspect; at the threshold it is down. A successful result resets the counter and returns the state to up.

For multiple locations, the aggregate is down when the number of down locations reaches quorum, degraded when at least one is down but quorum is not reached, suspect when none is down but at least one is suspect, and up when a healthy report is available without a failing aggregate. Locations that have not reported do not independently trip quorum.

Result sources and analytics

monitor

A result produced by the monitor execution path and eligible for monitor availability and latency analytics.

derived

A state or result produced by aggregation rather than a direct network execution.

platform

A platform-generated operational result, such as a stale passive monitor. These records are excluded from normal uptime analytics.

Recent ranges can be calculated from raw results. Longer windows use hourly and daily rollups plus an unrolled raw tail. Analytics responses identify their source and coverage and can report partial coverage when retained data does not span the requested range.

Alert and public status flow

1. The alerter evaluates stored monitor, location, host-agent, and mesh state.
2. It opens, acknowledges, reminds, and resolves alert records while deduplicating lifecycle notifications.
3. Notification routing comes from tenant defaults or per-monitor custom channel assignments, including per-channel delay.
4. Selected alerts can be attached to incidents. Incidents can publish updates to selected status pages.
5. The status-page service loads durable page data from the API/database path and uses statuspage.updates to invalidate caches and push live SSE refreshes.

Live NATS messages are an acceleration mechanism, not the durable source of truth. Public templates also refresh on a fallback interval so a missed core-NATS message does not permanently freeze a page.

Tenant and authorization boundaries

Application data is tenant-scoped throughout the model. Administrator sessions can select a tenant for which the user has membership; API keys are permanently pinned to the tenant that created them.

Operational service contract

• Standard services expose /healthz, /readyz, and /metrics on their configured listeners.
• Graceful shutdown stops new work, drains or closes queue consumers where applicable, and allows in-flight HTTP work to complete within configured limits.
• Remote workers require NATS but do not receive direct PostgreSQL credentials.
• Location-specific monitor secrets are encrypted for the location credential before they cross the queue; the platform master encryption key is not sent to a worker.
• PostgreSQL schema, queue subjects, credentials, ports, and public origins must be changed consistently across local Compose and Helm deployment configuration.