Security

Trust boundaries

Public edge

Frontend, API routes intentionally published through the frontend proxy/ingress, agent ingest, and public status pages.

Control plane

API, scheduler, alerter, PostgreSQL, and platform NATS credentials. This tier can access every tenant’s shared application data.

Execution plane

Workers execute user-configured network checks and synthetic browser code. Isolate them because their purpose is outbound reachability.

Private location

A remote worker fleet with a location-scoped NATS identity and no PostgreSQL access.

Presentation plane

Public status renderer and browser artifact access. Public status content is intentionally anonymous; edit/preview paths are not.

Roles, scopes, and tenant isolation

• Do not accept a tenant ID from a request without reconciling it to the authenticated identity and selected membership.
• API-key tenant identity is resolved from the stored key, not from a client-chosen tenant header.
• Review side-effecting “test,” “preview,” or “probe” endpoints carefully before adding them to the read-only POST allowlist.
• Use separate API keys per integration, give them an expiry, select read unless mutation is necessary, and revoke them when ownership changes.

Passwords, API keys, and browser sessions

Session cookies are HttpOnly and SameSite=Lax. Their Secure attribute is controlled by ADMIN_COOKIE_SECURE, which defaults to false. Production HTTPS deployments must set it to true and preserve one stable JWT secret across API replicas.

OIDC security

• Use an exact HTTPS issuer and callback. The default callback is <PUBLIC_BASE_URL>/api/v1/auth/oidc/callback.
• Register only the needed scopes: default openid profile email.
• Store OIDC_CLIENT_SECRET outside source control and rotate it using the IdP’s overlapping-secret procedure when possible.
• Use JIT role viewer unless a stronger business requirement exists. Validate OIDC_JIT_DEFAULT_TENANT_ID because the loader does not validate UUID syntax.
• Prefer OIDC group mappings over broad JIT defaults when the IdP manages access: roles are re-derived from IdP groups on every login, so removing a user from a group revokes their access at their next sign-in (active sessions are re-checked per request, so demotions bite immediately after the sync).
• Group-mapping misconfiguration cannot fully lock you out: the sync never demotes the last active superadmin, local password login is unaffected, and deleting all mapping rows instantly restores manual role management.
• Disable JIT provisioning when accounts must be approved in advance; pre-created users are matched using verified identity information.
• Keep PUBLIC_BASE_URL stable and canonical to avoid callback confusion behind proxies.
• Review IdP claim and email-verification behavior during every provider change.

Secret inventory and ownership

• Never commit .env, Helm production values, generated location snippets, backup files, or decrypted configuration.
• Kubernetes Secrets are base64 transport objects, not encrypted storage by themselves. Enable etcd encryption and use workload identity/external secret management where available.
• Helm stores release values in-cluster; avoid putting secrets into command history and protect the namespace/release metadata. Every chart credential can instead be read from a Secret you supply, which keeps it out of the release values entirely.
• The local launcher chmods the development JWT file to 0600, but the development encryption-key file relies on the current umask. Verify and restrict its permissions.
• Use separate secrets for preview signing, JWT signing, encryption, webhooks, NATS issuer, and location credentials; do not reuse one master string.

Encryption at rest and key rotation

When a valid PROBARA_SECRETS_KEY keyring is configured, sensitive configuration is stored in self-describing AES-256-GCM JSON envelopes containing algorithm, key version, nonce, and ciphertext. The base key and every rotation key must decode from base64 to exactly 32 bytes.

# Keep V1 configured while introducing V2.
export PROBARA_SECRETS_KEY='existing-base64-key'
export PROBARA_SECRETS_KEY_V2="$(openssl rand -base64 32)"
export POSTGRES_URL='postgres://...'

# Preview and then re-encrypt monitor secret fields with the highest key version.
go run ./cmd/admin/reencrypt_monitor_configs -dry-run
go run ./cmd/admin/reencrypt_monitor_configs

go run ./cmd/admin/encrypt_existing_channels -dry-run
go run ./cmd/admin/encrypt_existing_channels

1. Back up PostgreSQL and all currently configured key versions.
2. Add the next PROBARA_SECRETS_KEY_V<n> to every data-handling service before writing with it.
3. Restart/roll out services and verify each replica reports the expected keyring.
4. Dry-run and execute monitor re-encryption; encrypt any legacy plaintext channels.
5. Inventory ciphertext envelope versions across monitor, alert-channel and location data, plus any retired AI provider key (rollback only).
6. Remove an old key only after every relevant envelope is migrated and a restore test succeeds without it.

SSRF and worker network policy

HTTP_BLOCK_PRIVATE_IPS=true is the recommended setting; the code default is false, and neither the Helm chart nor Compose sets it, so set it on every worker that runs inside a network it should not reach. Despite its historical name, the guard is shared by HTTP, browser, TCP, gRPC, PostgreSQL, MySQL, MongoDB, Redis, broker, SIP, and other worker dial paths.

• Hostname dials resolve candidate IPs and validate them before connecting.
• A SIP test call with `call.media` sends audio to the address the far end's SDP answer names, which the remote server chose rather than the operator, so it is vetted more strictly than a target before a single packet is sent: the port must be 1024-65535 (keeping the stream off DNS, NTP, SNMP and syslog); unspecified addresses and 0.0.0.0/8, multicast and 240.0.0.0/4, link-local addresses including cloud metadata, and the broadcast or network address of any of the worker's own IPv4 subnets (broadcast, which would otherwise reach every host on that segment) are refused whatever the policy or HTTP_ALLOWED_CIDRS say; the signalling peer's own address is allowed; loopback only when the peer is loopback; then the worker's policy applies; and, even with HTTP_BLOCK_PRIVATE_IPS=false, a private or reserved address named by a public peer is refused (private_behind_public) unless HTTP_ALLOWED_CIDRS lists it, while a private peer naming a private address (an on-premises estate) passes. A NAT64 (64:ff9b::/96, 64:ff9b:1::/48) or 6to4 (2002::/16) address is judged on its embedded IPv4 address too, so an IPv6-only worker cannot reach 10.0.0.1 through 64:ff9b::a00:1; Teredo and documentation prefixes count as reserved. A hostname in the answer is refused, never resolved, so the far end cannot steer the worker's DNS lookups.
• Media leaves from the call's bound, unconnected RTP and RTCP sockets, which surface no ICMP errors, so the stream cannot probe which ports are open, and which have broadcast turned off (SO_BROADCAST), so the kernel refuses any broadcast destination the vetting did not catch, such as a subnet the worker is not on but routes to. RTCP goes only to the media destination's IP, at the answer's a=rtcp port or the RTP port + 1, vetted by the same rules; an a=rtcp naming any other address gets no RTCP, so the far end cannot name a second host. The far end's RTCP is read only from the addresses audio is accepted from. Latching is restricted: the probe moves its audio at most once, to another port of the address the answer named, or, when that address was refused, to the signalling peer's IP; the new target is vetted again, and the destination never moves to any other address. Packets from any other source are counted, never analysed. Refusals are reported as coarse reason codes, and no media address ever appears in an error message or warning, which reach notifications; addresses appear only in the check's metrics_data, which the operator UI reads and status pages never show.
• What a hostile far end can do with this is bounded: at most about 100 KB of fixed, probe-generated content (silence, or the test tone) per call, plus at most 3 small RTCP packets, toward one public address it names, never into the worker's private network, to metadata, to a well-known port, or to a second host. A tenant admin who points a monitor at a SIP server of their own controls both the trigger and the address, so they could aim probe audio at a third party: about 80 kbit/s per call, at most WORKER_CONCURRENCY scheduled calls plus 16 Test calls per worker at once, from monitors limited to one call a minute per location and holds of at most 10 s; only a signed-in tenant admin can create or run a test call, and every change and test is in the audit log. A tenant admin who points SIP at a private peer (allowed with HTTP_BLOCK_PRIVATE_IPS=false) can have that peer name any private media address, which is no wider than the SIP egress to that network they already have.
• A SIP monitor's nameserver is dialed under the same policy as its targets. Without one, its RFC 3263 NAPTR query goes to the worker's own /etc/resolv.conf nameservers, the resolver every hostname lookup already uses: the policy lets exactly those addresses through.
• Browser validation rejects a hostname if any returned address is blocked, reducing mixed public/private DNS selection risk.
• Use a dedicated location worker for internal monitoring and grant that location only the internal CIDRs it owns.
• Do not globally disable blocking to monitor one private target.
• Kubernetes discovery marks a suggestion internal from its name alone (Service addresses, IP literals, single-label names and .svc, .cluster.local, .local, .internal hosts) and requires a private location only for those. That flag is a routing hint, not an SSRF control: a public-looking Ingress host that resolves to a private address is classed external and can be approved onto the shared fleet. Keep HTTP_BLOCK_PRIVATE_IPS=true on every fleet that runs discovered checks, with HTTP_ALLOWED_CIDRS granting a private location only the cluster ranges it should reach.
• Treat tls_skip_verify and custom CA settings as scoped exceptions; prefer correct certificate chains and server names.
• Run workers without cloud instance metadata access and with a minimally privileged service account.

Notification channels (generic webhook, Slack, Discord, Teams, PagerDuty, Opsgenie, Telegram, Twilio) have their own policy, because their destinations are tenant-supplied and the requests leave the API, alerter and worker unattended. It uses the same address table and resolve-then-validate dialer as the monitor guard, but it is on by default and configured separately, so enabling internal monitoring never opens internal notification targets.

• Set both on the API (channel test), alerter (synchronous delivery) and worker (asynchronous delivery); a service that never receives the policy fails closed.
• Redirects are never followed, so an allowed host cannot bounce a request to a blocked one.
• Transport errors are logged without the request URL, because webhook paths and Telegram bot tokens are credentials.
• With an HTTPS_PROXY configured, the policy vets the proxy address (allow-list it) and destination control becomes the proxy's job.

Status page browser notifications add a third egress path with its own guard. The subscribe endpoint is the only unauthenticated write in the status-page service, and the endpoint it stores is a URL the platform later POSTs to, unattended, from inside the cluster — so an unguarded version would let any visitor turn the subscription table into a stored-SSRF and outbound spam primitive.

• Stored endpoints must be https and must match STATUS_PAGE_PUSH_ENDPOINT_ALLOWLIST (default: the Google, Mozilla, Microsoft, and Apple push services). Matching is on a dot boundary, so fcm.googleapis.com.attacker.example does not match fcm.googleapis.com.
• Setting the allowlist to * disables the check for a self-hosted push service and re-opens this surface; restrict egress by network policy if you do.
• Request bodies are capped, subscriptions are rate limited per client IP, and each page has a subscription cap so the table cannot grow without bound.
• X-Forwarded-For is honored for rate-limit keying only when STATUS_PAGE_TRUSTED_PROXY=true, because the header is client-settable.
• Subscriptions store only the opaque browser endpoint and its public keys — no visitor identity, no email address, and no page-visit history.

Private locations and NATS hardening

Private locations authenticate to NATS with location UUID as username and a 32-byte random URL-safe credential as password. The API authorization callout validates only active, non-deleted locations and issues a one-hour NATS user claim scoped to that location.

• Use tls:// or wss:// for every public broker endpoint. Deploy generation rejects plaintext schemes and URLs that already contain userinfo.
• Keep the account issuer seed only in API. Configure only its public account key in NATS.
• Use a separate platform NATS user for trusted internal services. Never give it to a location worker.
• Location claims restrict result publication, heartbeat publication, consumer pull/ACK operations, inbox responses, and test-job subscription to that location.
• Monitor and revoke disabled/deleted location credentials. Replace credentials if a generated deploy URL is exposed in logs, shell history, manifests, or tickets.
• Remote workers must not receive PostgreSQL credentials or the platform-wide encryption key. Monitor configuration secrets are re-encrypted with a key derived from the location credential.
• Expose mesh echo endpoints only across intended private networks; they identify location connectivity and should not publish /metrics externally.

Notifications, webhooks, SMTP, and AI

Webhook signing

When a webhook secret is configured, Probara sends X-Probara-Signature: sha256=<HMAC> so receivers can authenticate the raw body.

SMTP TLS

SMTP_USE_TLS=true performs direct implicit TLS. It does not issue STARTTLS, so verify protocol and port with the provider.

Local AI token

LOCAL_AI_TOKEN is installation-owned and shared only by the API, platform worker and CPU service, which compares it in constant time. Workspaces cannot set an inference destination or credential. Plain Compose falls back to a development token that is public in the repository, acceptable only because the port is bound to loopback. When Helm manages it and secrets.localAiToken is empty, the token is derived from secrets.adminJwtSecret, so it is exactly as secret as that value; set it explicitly or from an existing Secret for production.

AI training labels

A label is copied server-side from the decision snapshot the worker stored when the analysis or alert triage completed (dependency pairs: evidence recomputed at labelling time); client-supplied evidence is rejected. It holds the evidence counters and booleans, catalogue keys, opaque decision IDs, the monitor and incident IDs the UI names decisions with, and the labelling user ID. It never holds names or free text. Labels are kept until the workspace is deleted: data_retention_days does not prune them, and they survive monitor purges and incident deletion.

Suspected-cause follow-ups

When a workspace opts in, the alerter sends the alert triage result to the notification providers already paged for the alert: the catalogue key, title and suggested first checks, the confidence level and the triage ID, alongside the usual alert fields (webhooks receive them as alert.suspected_cause). No evidence, model output text or summary leaves the installation this way, and PagerDuty and Opsgenie never receive it. The alerter only reads the catalogue; it never calls local AI.

Label export

Export rows carry evidence, catalogue answers and provenance, and never monitor, incident, analysis, decision, user or tenant identifiers or names. Row and group IDs are HMACs keyed by a per-workspace random salt that is never exported, so they stay stable across exports without revealing the subject. Treat the files as workspace operational data.

• Verify webhook signatures with constant-time comparison, reject stale/replayed events according to your receiver policy, and rotate per channel.
• Do not place secrets in webhook URLs when headers or signing are available; URLs appear in proxy and DNS logs.
• Restrict webhook destinations with egress policy because the worker monitor CIDR guard does not cover the notification plugin.
• Use a dedicated SMTP account with send-only permissions and a controlled From domain.
• Review alert and RCA payloads before sending them to third-party notification providers; error bodies, URLs, headers, and incident context can be sensitive.
• Keep the local AI service reachable only from the API and platform worker. The chart’s NetworkPolicy does this only on a CNI that enforces NetworkPolicy; elsewhere the token is the only barrier. Share label export files only with the people who retrain the model.
• Keep asynchronous dispatch stream/consumer permissions separate from check execution permissions.

Status pages, previews, and browser artifacts

• Published status pages are intentionally public. Do not include private monitor URLs, internal hostnames, customer names, or operational notes unless disclosure is intended.
• Set one strong shared STATUS_PAGE_PREVIEW_SECRET on API and status service. Without it, preview-token enforcement is disabled.
• Preview tokens are HMAC-SHA256 values with a one-hour expiry. Treat preview URLs as temporary bearer links.
• The status service’s optional API proxy is intentionally restricted, but should still be exposed only through the expected status editing/preview flow.
• Synthetic browser screenshots can contain credentials, cookies, personal data, or application content. Store them encrypted, authorize retrieval, and expire them. Apply the same controls if trace or HAR capture is implemented later.
• Never make SYNTHETIC_BROWSER_ARTIFACTS_DIR a public static directory.

Audit evidence and trusted proxies

Audit records capture time, tenant, actor type/ID/label, action, resource, outcome, HTTP status, IP, user agent, and structured details. Writes are asynchronous in batches so audit storage does not block product requests.

• The recorder buffer holds 1,024 events, flushes every two seconds or 100 records, and drops rather than blocking when full. Watch for “Audit buffer full; dropping events” warnings.
• A failed audit insert does not fail the original API request. Forward application logs and audit records to a protected external system when compliance requires stronger durability.
• AUDIT_RETENTION_DAYS defaults to 365; 0 retains forever. This is separate from tenant telemetry retention.
• Restrict audit search/export to appropriate administrative roles and treat IP/user-agent/details as potentially personal or sensitive data.
• Back up audit data and preserve time synchronization across services.

Database, broker, containers, and ingress

Security verification checklist

• Confirm all public URLs use HTTPS and every remote NATS URL uses TLS/WSS with certificate validation.
• Verify ADMIN_COOKIE_SECURE=true, stable random JWT/preview/encryption keys, and exact OIDC callback values.
• Enumerate API keys by owner, tenant, scope, expiry, and last use; revoke unknown or overprivileged keys.
• Test viewer/editor/admin/superadmin and read/write API-key boundaries across representative endpoints.
• Attempt loopback, metadata, RFC1918, IPv6 local, mixed-DNS, redirect, browser, database, and broker SSRF cases from each worker class.
• Inspect effective Kubernetes environment and NATS permissions rather than relying on values files.
• Verify scheduler has every encryption key and can execute encrypted monitors.
• Confirm notification egress cannot reach internal metadata/control-plane addresses.
• Test location disable/revocation and ensure one location cannot consume another location’s jobs.
• Validate audit capture, drop warnings, retention, export restrictions, and external log durability.
• Restore a database backup with the complete keyring and verify encrypted channels, monitors, and locations.
• Scan images/dependencies and run Go race tests, frontend production builds, Helm lint, and integration tests before release.

Credential-exposure response