Agents and push monitors

Choose an agent or push monitor

Agent monitors are served by probara-collector, a minimal OpenTelemetry Collector distribution that scrapes host metrics and pushes them to Probara's OTLP ingest endpoint. Because the wire protocol is standard OTLP/HTTP, a stock upstream collector build works too — see migrating and extending.

Agent monitor configuration

The monitor receives availability success whenever an OTLP export is accepted. Metric rules are evaluated separately and open host-metric alerts rather than turning the availability result into failure.

Rule metric names must match what the host's scrapers actually emit. The generated collector config enables a per-OS scraper set:

Install the collector

The authenticated agent endpoints generate per-OS install and uninstall instructions containing the monitor's agent ID, API origin, the rendered collector configuration, and a caller-provided API key.

1. Create an agent monitor and choose its expected interval and optional metric rules.
2. Create a tenant API key with write scope. Read scope cannot submit metrics.
3. Fetch the install script with that key (Authorization: Bearer <write-scope key>). The key is embedded only then: a script fetched from a signed-in browser session, an MCP token or a read-scope key carries no key and refuses to run before touching the host.
4. Open the monitor's agent install instructions and select the target platform.
5. Run the generated script with the privileges required by that platform.
6. Confirm the first report appears immediately, then verify periodic reports and current state.

Install scripts download the platform binary from GET /static/collector/probara-collector-{linux,darwin}-{amd64,arm64} (or -windows-amd64.exe), verify its SHA-256 against the published checksums.txt (if checksums.txt cannot be downloaded the installer aborts rather than install an unverified binary; only a binary with no published entry installs with a warning), and first remove any legacy probara-agent installation (systemd/launchd/NSSM service, binary, and config) before installing. Re-running the installer on a host is therefore also the migration path. Re-running the install one-liner also upgrades in place: it replaces the binary, configuration and credentials and restarts the service (systemd restart, launchd kickstart -k; on Windows the binary is swapped and the service re-created), so it is the path for upgrades, key rotation and interval changes too. Uninstall scripts clean up both generations.

GET /api/v1/monitors/{id}/agent/install returns the same material as JSON, including the rendered collector_config YAML and the collector_version. Configuration-management users can fetch the raw collector YAML directly from GET /api/v1/monitors/{id}/agent/config.yaml?platform=linux|darwin|windows and distribute it themselves.

Generated URLs depend on PUBLIC_BASE_URL, which must be an absolute http:// or https:// origin without user information, query, or fragment. The corresponding platform binary must exist in the API's static collector directory. See the public URL configuration for how this value is wired.

Run the collector manually

export PROBARA_API_KEY=<write-scope-api-key>
export PROBARA_AGENT_ID=<agent-id>

probara-collector --config /etc/probara-collector/config.yaml

The generated configuration file contains no secrets: it references credentials exclusively through ${env:…} expansion, so the same file can be committed to configuration management while the environment file (or the Windows service registry value) carries the API key.

Reporting behavior

The collector scrapes host metrics on its configured interval and pushes them as OTLP/HTTP to POST /api/v1/otlp/v1/metrics (protobuf or JSON, gzip supported), authenticated with Authorization: Bearer <tenant API key>. Monitor identity comes from the X-Probara-Agent-Id header, or from a probara.agent.id resource attribute when a gateway collector multiplexes metrics for several hosts through one connection. Responses follow the OTLP specification, including partial_success when only some points were rejected.

Each accepted export doubles as one availability heartbeat: a success check result stamped with the server's receipt time, so collector clock skew can never affect state or history. The heartbeat carries no metrics_data: host readings live only in the metric store, read through GET /api/v1/monitors/{id}/metrics/series and POST /api/v1/monitors/{id}/metrics/query.

Metrics land in a generic metric store: a per-series registry, raw samples in daily partitions kept for METRIC_RAW_RETENTION_DAYS (default 30; a tenant's data_retention_days can tighten it further), and hourly rollups (min/max/avg/sum/first/last plus reset-aware increase for counters) pruned after 400 days. Any OTel metric a collector sends is stored, chartable in the UI's metric explorer, and usable in metric rules — not only the host metrics above. GET /api/v1/monitors/{id}/metrics/series discovers stored series and POST /api/v1/monitors/{id}/metrics/query runs batch range queries.

Agent freshness and recovery

A scheduler-side watchdog marks an agent monitor failed after no accepted report for three times the monitor interval, with a 90-second floor. The watchdog re-fires each window while the silence persists, so consecutive missed windows accumulate toward down exactly like consecutive failed checks. A later valid export creates a success result and recovers availability.

Stale transitions are recorded with result_source = monitor, so they affect current state and availability alerting and are included in standard uptime analytics. Only expired-job results use the platform source that analytics exclude.

Metric rules and host-metric alerts

When a metric rule is breached, Probara opens a separate host-metric alert. Alerts are keyed by the canonical series key, for example system.filesystem.utilization{device=/dev/sda1,mode=rw,mountpoint=/data,type=ext4}, so each series maintains its own independent lifecycle.

• A rule without attribute filters fans out per matching series: one filesystem rule opens one alert per breaching mountpoint, removing the old single-disk-path limitation.
• for_duration_seconds requires the breach to hold for the whole window before the alert opens.
• Evaluation is freshness-bounded (interval×3, 90-second floor): a dead collector's stale readings can no longer keep metric alerts open — availability alerting covers the outage instead.
• A metric-rule breach does not mark agent availability down; a metric alert resolves when the series returns within the rule, and removing a rule resolves its conditions.
• Notification routing follows the monitor's default or custom channel assignments.

Uninstall (remote disable removed)

The legacy agent's opt-in remote-disable mechanism — a 410 Gone response triggering a local self-uninstall — is removed. The collector never executes removal actions on behalf of the backend. Deleting the monitor stops the backend from accepting its reports but leaves the collector service installed and exporting into rejections.

A legacy agent still running after the sunset receives 410 Gone on every report. One explicitly started with --allow-remote-disable reads that as the old self-uninstall signal and runs its remove command; all others log the response and retry until uninstalled.

Migrating from the legacy agent

The custom probara-agent binary is replaced by the OpenTelemetry Collector. Its deprecation window ended on Wednesday, 18 November 2026: POST /api/v1/agent/metrics no longer accepts reports. It answers 410 Gone with a JSON error naming POST /api/v1/otlp/v1/metrics and a Link header (rel="successor-version") pointing at it, stores nothing, and logs each rejected report server-side so operators can find stragglers. A monitor whose host still runs the legacy agent receives no reports, so it goes stale and then down until the collector is installed. Old agent binaries are no longer distributed.

1. Re-run the install one-liner on each host: the installer removes the legacy agent (service, binary, config) and installs the collector in one pass.
2. Verify the monitor keeps reporting; the same monitor, agent ID, and API key carry over unchanged.
3. Check the API logs for legacy agent push rejected warnings (each names the tenant), or for agent monitors that went down, to find hosts still on the legacy path.

Air-gapped or mirror-constrained environments do not need the probara-collector binary at all: the generated configuration is a standard OpenTelemetry Collector config, and a stock otelcol-contrib build runs it unchanged. It uses the current snake_case component names (host_metrics, resource_detection, metrics_transform, otlp_http), which need otelcol-contrib 0.153.0 or newer; an older upstream build rejects them as unknown components. Configs written by earlier installers use the old names (hostmetrics, resourcedetection, metricstransform, otlphttp), which the bundled collector still accepts as deprecated aliases, logging a warning at startup; re-running the install one-liner rewrites the config with the current names. Fetch the YAML from GET /api/v1/monitors/{id}/agent/config.yaml and distribute upstream binaries from your own mirror.

The configuration is also an extension point: add more host_metrics scrapers, or additional receivers entirely (with otelcol-contrib), pushing to the same OTLP endpoint. Any metric that arrives is stored, chartable, and alertable through metric rules — subject to the per-monitor series and rate limits.

Push monitor configuration

Use a push monitor for cron jobs, backup jobs, queue consumers, or external systems that can call an HTTP URL but do not need host metrics. The endpoint accepts both GET and POST without an admin session because the token itself authorizes the report.

Send push heartbeats

curl -fsS 'https://monitoring.example.com/api/v1/push/<token>'

curl -fsS -X POST \
  -H 'Content-Type: application/json' \
  -d '{"status":"down","error":"backup upload failed","files":182,"bytes":987654321}' \
  'https://monitoring.example.com/api/v1/push/<token>'

Reserved JSON keys are status and error; remaining keys are stored as metrics. Query-string values on GET are converted to integers, floating-point values, booleans, or strings when possible.

A disabled push monitor returns HTTP 403 and a missing/deleted token returns 404. A successful push updates freshness and can recover an existing stale/down availability condition.

Push freshness

A push monitor becomes stale after expected_interval_seconds + grace_period_seconds without an accepted report. While it remains stale, the platform limits repeated generated failures to avoid unbounded duplicate history. A new accepted heartbeat restores up state.