Monitors

Monitor model

A monitor combines a type-specific check configuration with scheduling, state confirmation, notification routing, dependencies, and optional execution locations.

Read responses also expose identity and timestamps, next_run_at, the effective current state, maintenance status, member and dependency information, passive credentials where applicable, and per-location state on the detailed monitor view.

Supported monitor types

Scheduling, confirmation, and quorum

An active monitor runs on its interval. Both failure and error outcomes count toward consecutive failure confirmation. Before the threshold it is suspect; at the threshold it becomes down. Any successful effective result resets the temporal counter and returns it to up. Confirmed transitions drive the availability alert lifecycle.

A monitor selected for multiple locations maintains independent location state. The aggregate becomes down when down locations reach location_quorum; it becomes degraded when at least one location is down but the quorum is not met. If no location is down but one is suspect, the aggregate is suspect. Locations that have not reported are not counted as down.

HTTP monitors

HTTP monitors support transport diagnostics and layered assertions on one response.

When no status expectation is supplied, the accepted default is the 2xx class. When multiple status criteria are supplied, satisfying any accepted code, range, or class is sufficient. Body, header, JSON, and latency assertions are then evaluated in addition to status. The tls_min_days_valid window is the exception: it never fails the check and instead raises a dedicated tls_expiry alert (the check still fails if the certificate cannot be inspected at all, e.g. a non-HTTPS URL with the field set).

JSON assertions support exists, equals, not_equals, contains, not_contains, regex, number_gt, number_gte, number_lt, number_lte, and bool_is. Result metrics can include the final URL, redirect count, phase timings, TLS version and cipher, certificate validity, issuer, subject names, expiry, and assertion detail.

{
  "url": "https://api.example.com/health",
  "method": "GET",
  "headers": {"Accept": "application/json"},
  "expected_status_classes": ["2xx"],
  "json_assertions": [
    {"path": "status", "op": "equals", "value": "ok"},
    {"path": "queue_depth", "op": "number_lt", "value": 100}
  ],
  "max_latency_ms": 1500,
  "follow_redirects": true,
  "max_redirects": 5,
  "tls_min_days_valid": 14,
  "collect_timing": true
}

Ping, DNS, gRPC, TCP, and SIP

Ping

Set host to a hostname or IP address. The worker performs an ICMP reachability check; firewalls and worker privileges can affect results independently of application health.

DNS

Set host, optional record_type, optional expected_answers, and optional nameserver. Supported record types are A, AAAA, CNAME, TXT, MX, and NS; the record type defaults to A. A nameserver can be a host or host:port, with port 53 used when omitted.

gRPC

Set host, optional port, optional service, and use_tls. Probara calls the standard grpc.health.v1.Health/Check method and succeeds only for SERVING. The default port is 443 with TLS and 80 without it.

TCP

Set host and port; optionally enable TLS and tls_skip_verify. A successful connection, plus a successful TLS handshake when enabled, is healthy. No application payload is exchanged.

SIP

Set host, optional port, transport (udp, tcp, or tls), method, and expected_status. method: options (default) sends a SIP OPTIONS availability ping; method: register sends a query-style REGISTER (no Contact header) that exercises the registrar and its digest authentication without creating, refreshing, or removing bindings. Configure username/password to answer digest challenges (MD5 and SHA-256, qop=auth) on either method, and domain to set the address-of-record for REGISTER. Port defaults to 5060 (5061 for TLS), status to 200; tls_skip_verify accepts lab certificates.

WebSocket monitors

Without a message or expected substring, a successful WebSocket upgrade is sufficient. With an expectation, the worker reads the first response message and performs a substring check.

Prometheus query monitors

A Prometheus monitor turns any PromQL instant query into an uptime signal. The worker POSTs the query to /api/v1/query on the configured base URL and compares every returned float sample against a threshold.

The monitor is healthy only when every returned sample satisfies the comparison, so an instant vector with several series is treated as several assertions. Use PromQL aggregation such as sum, max, or min when one value should decide. Scalar and instant-vector float results are supported; range vectors, string results, and native histogram samples report an error.

• NaN and infinite samples, including division by zero, report an error rather than a threshold verdict.
• Query warnings from Prometheus indicate potentially partial evaluation and report an error, never a healthy result.
• HTTP errors, malformed responses, redirects, and connection failures report an error. Redirects are never followed and HTTPS certificates are always verified.
• Responses are capped at 2 MiB; aggregate large queries rather than relying on truncation.
• The worker's SSRF policy and allowed CIDRs apply, including DNS resolution at connection time. Pick a private location whose worker can reach the endpoint when Prometheus is internal.

Check results record the sample count, the number of samples outside the threshold, and up to twenty numeric values. Labels and raw upstream error bodies are not stored. Test query in the form runs the current configuration through a worker without saving; when editing, saved masked credentials are resolved for that monitor.

Redis, PostgreSQL, MySQL, MongoDB, and RabbitMQ

Data-service monitors perform a real protocol handshake. Database monitors can optionally run a read query and assert its first returned value.

PostgreSQL and MySQL query assertions use query_value_op with equals, not_equals, contains, or numeric comparison operations against the first column of the first returned row. A configured query must return a row.

These monitors share optional max_latency_ms and warn_latency_ms. The maximum is a hard failure; the warning threshold annotates metrics without changing a successful check to failure.

MongoDB monitors can additionally enable per-feature cluster checks, each an individually toggleable read-only admin command: collect_replication runs replSetGetStatus (member states, health, and replication lag), while collect_connections, collect_cache, collect_memory, collect_network, and collect_cpu read their sections from a single serverStatus call (connections, WiredTiger cache, resident/virtual memory, network I/O and opcounters, and — on Linux servers — mongod process CPU time; host-level CPU is agent-monitor territory). Both commands are covered by MongoDB's built-in clusterMonitor role — no clusterAdmin, root, or write privileges. When the monitoring user lacks the role, the affected data is skipped and flagged in the check's metrics (unavailable) without failing the check; on a standalone server the replication check reports "not a replica set". Replication lag supports the same warn/max split as latency: warn_replication_lag_seconds annotates, max_replication_lag_seconds fails the check and flows through normal availability alerting — and it fails closed, so if replication status becomes unreadable (role revoked, command timeout, standalone target, no primary) the check fails rather than silently passing.

The monitor detail page charts the collected cluster metrics over recent checks — replication lag (with the warn/max thresholds drawn as reference lines), connections, WiredTiger cache, memory, and, for the cumulative operation, network, and process-CPU counters, per-second rates derived between consecutive checks.

Synthetic API journeys

A synthetic API monitor runs 1–20 ordered HTTP steps with variables, assertions, and extraction from earlier responses.

URLs may be absolute, or relative when base_url is present. Static and extracted variables are referenced with {{variable_name}} in subsequent supported fields. failure_mode is fail_fast by default; use continue when later diagnostic steps should still execute after a failed step.

Assertions can target status, response headers, body text, or JSON paths. Status supports equality, inequality, and membership. Text and header checks support existence, equality, containment, regular expressions, and negative forms. JSON checks additionally support numeric and boolean operations.

{
  "base_url": "https://api.example.com",
  "failure_mode": "fail_fast",
  "variables": {"user": "monitor@example.com"},
  "steps": [
    {
      "id": "login",
      "request": {
        "method": "POST",
        "url": "/login",
        "headers": {"Content-Type": "application/json"},
        "body": "{\"email\":\"{{user}}\"}"
      },
      "assert": [
        {"target": "status", "op": "equals", "value": 200}
      ],
      "extract": [
        {"name": "token", "from": "json", "path": "token", "sensitive": true}
      ]
    },
    {
      "id": "profile",
      "request": {
        "method": "GET",
        "url": "/me",
        "headers": {"Authorization": "Bearer {{token}}"}
      },
      "assert": [
        {"target": "json", "path": "email", "op": "equals", "value": "{{user}}"}
      ]
    }
  ]
}

Result metrics include completed-step count, total latency, and the failed step identifier. Give every step a stable unique ID so failures remain intelligible after renaming display labels.

Synthetic browser journeys

Browser monitors run 1–12 ordered actions in Chromium through Playwright.

Journey variables include start_url, optional device profile, failure_mode, static variables, artifact settings, and steps. Each step has a unique ID and a 1–180 second timeout. The configuration model exposes screenshot, Playwright trace, and HAR-on-failure switches, but the current worker implements screenshots only; trace and HAR requests produce warnings rather than artifacts.

Failure screenshots are exposed by the authenticated monitor artifact endpoint when the result contains a valid stored artifact path. Artifact availability depends on shared worker/API storage and retention. Do not build automation that expects trace or HAR files until worker support is implemented.

Monitor groups

A group monitor stores member monitor IDs instead of executing a check. It is down when any enabled, non-deleted member is down. It is up when all effective member states are up, suspect, or degraded; otherwise it is unknown.

per_monitor rollup

The default for newly created groups. Member monitors alert independently and the group does not create an additional availability alert.

group rollup

Suppresses member availability alerts covered by that group and creates one group-level availability alert. Suppression ignores the group's own enabled flag, so pausing such a group silences its members too — see group alert rollup.

Secret fields and masked updates

When PROBARA_SECRETS_KEY is configured, verified monitor secret handling covers password, connection_string, and tls_client_key_pem for Redis, PostgreSQL, MySQL, MongoDB, and RabbitMQ, the SIP digest password, the Prometheus password and bearer_token, plus every WebSocket header value. API reads replace protected values with ***.

• Submit *** again to preserve the already stored value.
• Omit the field entirely and the stored value is preserved as well, so an update that resubmits a config read back from the API cannot destroy a credential it was never shown.
• Submit a new value to replace and encrypt it.
• Submit an empty string to clear a protected field. That is the only spelling that removes a stored secret.
• WebSocket header names remain visible while their values are masked. Submitting the headers object edits it key by key, so a key left out of a submitted object is removed; omitting the whole object keeps the stored headers.

Test, run, inspect, and delete

Test configuration

Executes a provided configuration without creating or updating a monitor. It is compute-only and does not write result history or trigger alert lifecycle.

Run now

Queues an existing monitor for normal execution. The returned result is persisted and can change state and alerts.

Results

Raw chronological check records. List calls accept a limit and optional RFC 3339 since timestamp.

Analytics

Summaries, time series, and downtime periods for 1h, 6h, 24h, 7d, 30d, 90d, or 365d.

Delete history

Removes stored results for a monitor without deleting the monitor definition.

Delete monitor

Soft-deletes the monitor immediately. Scheduler cleanup later purges dependent historical data.

Analytics report uptime, SLA/availability summary fields, downtime duration, average/median/p95/latest latency, series data, downtime periods, source, coverage start, and whether the requested window is partially covered. Long ranges combine rollups with the current raw tail. The summary carries has_data: when no checks ran in the window the percentage fields are meaningless zeros and clients must render a no-data state, never 0% or 100%. It also carries method: interval means availability_pct is a time integration over the monitor's state timeline — unknown and paused time leave the denominator and surface as coverage_pct, and downtime inside maintenance windows counts as planned rather than unavailability; sampled means the window predates the timeline and the legacy success/total rate stands in.

Bulk changes, import, and export

• Bulk alerting changes can update notification mode and channel assignments across selected monitors.
• Bulk delete soft-deletes selected monitors and schedules cleanup.
• Import preview parses and validates without creating monitors. Execution reports a per-row outcome rather than treating every file as all-or-nothing.
• JSON, YAML, and CSV formats are accepted, including common wrapper keys and a versioned portable YAML bundle.
• Duplicate matching is case-insensitive on the combination of monitor name and type; duplicates are skipped.
• Groups are resolved in a second pass by member name, and ambiguous or missing names are reported.