Distributed monitoring

Private locations

Deploy credential-scoped workers, run checks from private networks, configure quorum, and monitor the location mesh.

20 min read

On this page

What a private location is

A private location is a tenant-scoped execution target backed by one or more remote worker processes. It lets an active monitor run from a branch, data center, VPC, or other network that the default worker fleet cannot reach.

Field	Meaning
`name`	Required tenant-unique display name, up to 100 characters
`slug`	Cosmetic derived identifier, up to 60 characters; do not use it as authentication
`description`	Optional operator context
`enabled`	Whether the location receives scheduled work
`connected`	Derived from recent worker heartbeats, not a manually editable flag
`last_seen_at`	Most recent accepted worker heartbeat
`mesh_endpoint`	Optional host-and-port endpoint other locations probe for mesh health
`monitor_count`	Current number of monitors assigned to the location

Create and deploy a worker

Create the location in the UI or location API.
Open its deployment information. Creation generates a random location credential that is stored encrypted when platform secret encryption is enabled.
Copy the generated Docker or Kubernetes configuration into the target network.
Start the worker and wait for its heartbeat. The backend considers a location connected when its latest heartbeat is within roughly one minute.
Assign the location to one or more active monitors and choose a failure quorum.

Shape of generated worker environment

image: ghcr.io/yassinebenameur/probara-worker:latest
environment:
  NATS_URL: tls://nats.example.com:4222
  WORKER_LOCATION_ID: <location-uuid>
  LOCATION_CREDENTIAL: <generated-secret>
  HTTP_PORT: "8080"
  METRICS_PORT: "9090"
  HTTP_BLOCK_PRIVATE_IPS: "true"

The generated NATS username is the location UUID and the generated password is the location credential. Treat both the deployment output and any copied manifest as secrets.

Credential and secret boundaries

The NATS authentication callout binds a worker credential to one tenant location and limits its queue access. A worker cannot select another tenant or subscribe to arbitrary location work merely by changing an environment variable.

Before a job containing protected monitor configuration is sent to a private location, the backend re-encrypts those values for that location credential. The worker receives what it needs for execution; it does not receive the platform's master secrets key.

Store the location credential in a Kubernetes Secret or equivalent secret store, not in a committed values file.
Use TLS or WSS for the public NATS endpoint as required by deployment-info generation.
Limit broker network exposure and use the generated credential only for its intended location.
Replace a compromised location rather than treating the display slug as a rotatable secret.

Monitor private targets safely

Workers block private, loopback, link-local, and reserved destinations by default. The generated private-location configuration deliberately keeps HTTP_BLOCK_PRIVATE_IPS=true, so merely moving a monitor to an internal worker does not automatically permit every internal address.

Permit only a trusted application subnet

HTTP_BLOCK_PRIVATE_IPS=true
HTTP_ALLOWED_CIDRS=10.42.16.0/20

Assign monitors and configure quorum

Active monitor types can target one or more locations. Group, agent, and push monitors cannot because they derive state from members or inbound reports. A monitor with no selected locations runs on the default worker fleet.

Reports	Quorum	Aggregate state
Paris down, New York up	1	`down`
Paris down, New York up	2	`degraded`
Paris down, New York down	2	`down`
Paris suspect, New York up	2	`suspect`
Paris unreported, New York up	2	`up` while no down report reaches quorum

The stored quorum is constrained to at least one and at most the selected location count. A down aggregate opens normal availability alerts; degraded does not. Degraded state also does not use the suspect fast-recheck path.

Only locations with fresh evidence vote. A location that stops reporting for three check intervals (minimum 90 seconds) is treated exactly like one that never reported: its last state stops counting, so a silent-while-up location cannot mask a real outage and a silent-while-down one cannot pin the monitor down. When no location has fresh evidence — or a location-less monitor receives no results at all — the platform's absence watchdog moves the monitor to unknown rather than leaving its last state frozen. No data is never rendered as healthy.

Connection and heartbeat behavior

Location workers publish lightweight core-NATS heartbeats approximately every 15 seconds with jitter. The API derives connected from the latest accepted heartbeat and treats a location as disconnected after about one minute without one.

Location mesh monitoring

A location with a mesh_endpoint can participate in directional connectivity checks between private locations.

For each ordered pair of enabled participating locations in the same tenant, Probara creates a directed edge. With N locations, the full directed mesh has N × (N − 1) edges. A source worker sends HTTP to the target's /mesh/echo endpoint and verifies that the response identifies the expected target location.

Mesh setting or state	Behavior
Default interval	30 seconds
Default timeout	5 seconds
Default failure threshold	3 consecutive failures
Suspect recheck	Approximately 20 seconds
States	`unknown`, `up`, `suspect`, and `down`, maintained independently per direction
History	Query by source, target, and time window; includes latency and error information
Staleness	Means no recent edge result; it is not conclusive proof that the target is down

Direction matters

Paris ──probe──► New York   can be DOWN
Paris ◄──probe──── New York   can remain UP

Mesh alerts are directional and use the tenant's default notification channels. They maintain their own lifecycle rather than changing an application monitor's state.

Disable or delete a location

Disabling a location excludes it from new monitor scheduling and quorum participation without deleting the location record. Deleting it soft-deletes the location, detaches it from monitors, removes its per-location state, and reduces affected quorums as needed.

If deletion leaves a monitor with no assigned location, the monitor returns to the default fleet. Its effective state resets to unknown and its failure counter resets before new default-fleet results arrive.