Security

Trust boundaries

Public edge

Frontend, API routes intentionally published through the frontend proxy/ingress, agent ingest, and public status pages.

Control plane

API, scheduler, alerter, PostgreSQL, and platform NATS credentials. This tier can access every tenant’s shared application data.

Execution plane

Workers execute user-configured network checks and synthetic browser code. Isolate them because their purpose is outbound reachability.

Private location

A remote worker fleet with a location-scoped NATS identity and no PostgreSQL access.

Presentation plane

Public status renderer and browser artifact access. Public status content is intentionally anonymous; edit/preview paths are not.

Roles, scopes, and tenant isolation

• Do not accept a tenant ID from a request without reconciling it to the authenticated identity and selected membership.
• API-key tenant identity is resolved from the stored key, not from a client-chosen tenant header.
• Review side-effecting “test,” “preview,” or “probe” endpoints carefully before adding them to the read-only POST allowlist.
• Use separate API keys per integration, give them an expiry, select read unless mutation is necessary, and revoke them when ownership changes.

Passwords, API keys, and browser sessions

Session cookies are HttpOnly and SameSite=Lax. Their Secure attribute is controlled by ADMIN_COOKIE_SECURE, which defaults to false. Production HTTPS deployments must set it to true and preserve one stable JWT secret across API replicas.

OIDC security

• Use an exact HTTPS issuer and callback. The default callback is <PUBLIC_BASE_URL>/api/v1/auth/oidc/callback.
• Register only the needed scopes: default openid profile email.
• Store OIDC_CLIENT_SECRET outside source control and rotate it using the IdP’s overlapping-secret procedure when possible.
• Use JIT role viewer unless a stronger business requirement exists. Validate OIDC_JIT_DEFAULT_TENANT_ID because the loader does not validate UUID syntax.
• Prefer OIDC group mappings over broad JIT defaults when the IdP manages access: roles are re-derived from IdP groups on every login, so removing a user from a group revokes their access at their next sign-in (active sessions are re-checked per request, so demotions bite immediately after the sync).
• Group-mapping misconfiguration cannot fully lock you out: the sync never demotes the last active superadmin, local password login is unaffected, and deleting all mapping rows instantly restores manual role management.
• Disable JIT provisioning when accounts must be approved in advance; pre-created users are matched using verified identity information.
• Keep PUBLIC_BASE_URL stable and canonical to avoid callback confusion behind proxies.
• Review IdP claim and email-verification behavior during every provider change.

Secret inventory and ownership

• Never commit .env, Helm production values, generated location snippets, backup files, or decrypted configuration.
• Kubernetes Secrets are base64 transport objects, not encrypted storage by themselves. Enable etcd encryption and use workload identity/external secret management where available.
• Helm stores release values in-cluster; avoid putting secrets into command history and protect the namespace/release metadata. Every chart credential can instead be read from a Secret you supply, which keeps it out of the release values entirely.
• The local launcher chmods the development JWT file to 0600, but the development encryption-key file relies on the current umask. Verify and restrict its permissions.
• Use separate secrets for preview signing, JWT signing, encryption, webhooks, NATS issuer, and location credentials; do not reuse one master string.

Encryption at rest and key rotation

When a valid PROBARA_SECRETS_KEY keyring is configured, sensitive configuration is stored in self-describing AES-256-GCM JSON envelopes containing algorithm, key version, nonce, and ciphertext. The base key and every rotation key must decode from base64 to exactly 32 bytes.

# Keep V1 configured while introducing V2.
export PROBARA_SECRETS_KEY='existing-base64-key'
export PROBARA_SECRETS_KEY_V2="$(openssl rand -base64 32)"
export POSTGRES_URL='postgres://...'

# Preview and then re-encrypt monitor secret fields with the highest key version.
go run ./cmd/admin/reencrypt_monitor_configs -dry-run
go run ./cmd/admin/reencrypt_monitor_configs

go run ./cmd/admin/encrypt_existing_channels -dry-run
go run ./cmd/admin/encrypt_existing_channels

1. Back up PostgreSQL and all currently configured key versions.
2. Add the next PROBARA_SECRETS_KEY_V<n> to every data-handling service before writing with it.
3. Restart/roll out services and verify each replica reports the expected keyring.
4. Dry-run and execute monitor re-encryption; encrypt any legacy plaintext channels.
5. Inventory ciphertext envelope versions across monitor, alert-channel, location, and AI/settings data.
6. Remove an old key only after every relevant envelope is migrated and a restore test succeeds without it.

SSRF and worker network policy

HTTP_BLOCK_PRIVATE_IPS=true is the secure default. Despite its historical name, the guard is shared by HTTP, browser, TCP, gRPC, PostgreSQL, MySQL, MongoDB, Redis, broker, SIP, and other worker dial paths.

• Hostname dials resolve candidate IPs and validate them before connecting.
• Browser validation rejects a hostname if any returned address is blocked, reducing mixed public/private DNS selection risk.
• Use a dedicated location worker for internal monitoring and grant that location only the internal CIDRs it owns.
• Do not globally disable blocking to monitor one private target.
• Treat tls_skip_verify and custom CA settings as scoped exceptions; prefer correct certificate chains and server names.
• Run workers without cloud instance metadata access and with a minimally privileged service account.

Status page browser notifications add a third egress path with its own guard. The subscribe endpoint is the only unauthenticated write in the status-page service, and the endpoint it stores is a URL the platform later POSTs to, unattended, from inside the cluster — so an unguarded version would let any visitor turn the subscription table into a stored-SSRF and outbound spam primitive.

• Stored endpoints must be https and must match STATUS_PAGE_PUSH_ENDPOINT_ALLOWLIST (default: the Google, Mozilla, Microsoft, and Apple push services). Matching is on a dot boundary, so fcm.googleapis.com.attacker.example does not match fcm.googleapis.com.
• Setting the allowlist to * disables the check for a self-hosted push service and re-opens this surface; restrict egress by network policy if you do.
• Request bodies are capped, subscriptions are rate limited per client IP, and each page has a subscription cap so the table cannot grow without bound.
• X-Forwarded-For is honored for rate-limit keying only when STATUS_PAGE_TRUSTED_PROXY=true, because the header is client-settable.
• Subscriptions store only the opaque browser endpoint and its public keys — no visitor identity, no email address, and no page-visit history.

Private locations and NATS hardening

Private locations authenticate to NATS with location UUID as username and a 32-byte random URL-safe credential as password. The API authorization callout validates only active, non-deleted locations and issues a one-hour NATS user claim scoped to that location.

• Use tls:// or wss:// for every public broker endpoint. Deploy generation rejects plaintext schemes and URLs that already contain userinfo.
• Keep the account issuer seed only in API. Configure only its public account key in NATS.
• Use a separate platform NATS user for trusted internal services. Never give it to a location worker.
• Location claims restrict result publication, heartbeat publication, consumer pull/ACK operations, inbox responses, and test-job subscription to that location.
• Monitor and revoke disabled/deleted location credentials. Replace credentials if a generated deploy URL is exposed in logs, shell history, manifests, or tickets.
• Remote workers must not receive PostgreSQL credentials or the platform-wide encryption key. Monitor configuration secrets are re-encrypted with a key derived from the location credential.
• Expose mesh echo endpoints only across intended private networks; they identify location connectivity and should not publish /metrics externally.

Notifications, webhooks, SMTP, and AI

Webhook signing

When a webhook secret is configured, Probara sends X-Probara-Signature: sha256=<HMAC> so receivers can authenticate the raw body.

SMTP TLS

SMTP_USE_TLS=true performs direct implicit TLS. It does not issue STARTTLS, so verify protocol and port with the provider.

Tenant AI key

Encrypted at rest when platform secret encryption is configured and never returned to clients; only a “has key” indicator is exposed.

• Verify webhook signatures with constant-time comparison, reject stale/replayed events according to your receiver policy, and rotate per channel.
• Do not place secrets in webhook URLs when headers or signing are available; URLs appear in proxy and DNS logs.
• Restrict webhook destinations with egress policy because the worker monitor CIDR guard does not cover the notification plugin.
• Use a dedicated SMTP account with send-only permissions and a controlled From domain.
• Review alert and RCA payloads before sending them to third-party notification or LLM providers; error bodies, URLs, headers, and incident context can be sensitive.
• Allowlist AI provider origins and use a model/account with retention and data-processing terms appropriate for your environment.
• Keep asynchronous dispatch stream/consumer permissions separate from check execution permissions.

Status pages, previews, and browser artifacts

• Published status pages are intentionally public. Do not include private monitor URLs, internal hostnames, customer names, or operational notes unless disclosure is intended.
• Set one strong shared STATUS_PAGE_PREVIEW_SECRET on API and status service. Without it, preview-token enforcement is disabled.
• Preview tokens are HMAC-SHA256 values with a one-hour expiry. Treat preview URLs as temporary bearer links.
• The status service’s optional API proxy is intentionally restricted, but should still be exposed only through the expected status editing/preview flow.
• Synthetic browser screenshots can contain credentials, cookies, personal data, or application content. Store them encrypted, authorize retrieval, and expire them. Apply the same controls if trace or HAR capture is implemented later.
• Never make SYNTHETIC_BROWSER_ARTIFACTS_DIR a public static directory.

Audit evidence and trusted proxies

Audit records capture time, tenant, actor type/ID/label, action, resource, outcome, HTTP status, IP, user agent, and structured details. Writes are asynchronous in batches so audit storage does not block product requests.

• The recorder buffer holds 1,024 events, flushes every two seconds or 100 records, and drops rather than blocking when full. Watch for “Audit buffer full; dropping events” warnings.
• A failed audit insert does not fail the original API request. Forward application logs and audit records to a protected external system when compliance requires stronger durability.
• AUDIT_RETENTION_DAYS defaults to 365; 0 retains forever. This is separate from tenant telemetry retention.
• Restrict audit search/export to appropriate administrative roles and treat IP/user-agent/details as potentially personal or sensitive data.
• Back up audit data and preserve time synchronization across services.

Database, broker, containers, and ingress

Security verification checklist

• Confirm all public URLs use HTTPS and every remote NATS URL uses TLS/WSS with certificate validation.
• Verify ADMIN_COOKIE_SECURE=true, stable random JWT/preview/encryption keys, and exact OIDC callback values.
• Enumerate API keys by owner, tenant, scope, expiry, and last use; revoke unknown or overprivileged keys.
• Test viewer/editor/admin/superadmin and read/write API-key boundaries across representative endpoints.
• Attempt loopback, metadata, RFC1918, IPv6 local, mixed-DNS, redirect, browser, database, and broker SSRF cases from each worker class.
• Inspect effective Kubernetes environment and NATS permissions rather than relying on values files.
• Verify scheduler has every encryption key and can execute encrypted monitors.
• Confirm webhook and LLM egress cannot reach internal metadata/control-plane addresses.
• Test location disable/revocation and ensure one location cannot consume another location’s jobs.
• Validate audit capture, drop warnings, retention, export restrictions, and external log durability.
• Restore a database backup with the complete keyring and verify encrypted channels, monitors, locations, and AI settings.
• Scan images/dependencies and run Go race tests, frontend production builds, Helm lint, and integration tests before release.

Credential-exposure response