Deployment

Choose a deployment workflow

Prerequisites

• Docker with Compose v2 for PostgreSQL, NATS, migrations, and container workflows.
• GNU Make and a Bash-compatible environment for repository scripts. On Windows, use WSL or another environment that can execute the Makefile’s Bash commands.
• Go 1.26.5 for the main monorepo. The local launcher compares the installed major/minor version with the root go.mod directive.
• Node.js LTS and npm for the frontend. scripts/start-ui.sh can use nvm when it is available.
• Helm 3 and kubectl for Kubernetes deployment.
• A PostgreSQL 16-compatible database and NATS 2.10 with JetStream for production when not using embedded chart dependencies.

The host agent (probara-collector) is not a separate Go module: scripts/build-collector.sh generates and cross-compiles a minimal OpenTelemetry Collector distribution from collector/manifest.yaml with a pinned builder version. Root make test covers the whole repository.

Prepare required values

cp .env.example .env

# Generate stable secrets; keep them across restarts and upgrades.
openssl rand -hex 32       # ADMIN_JWT_SECRET
openssl rand -base64 32    # PROBARA_SECRETS_KEY

• Set ADMIN_JWT_SECRET to a random value of at least 32 characters. Validation only rejects an empty value, so shipped placeholder defaults must be replaced deliberately.
• Set PUBLIC_BASE_URL to the externally reachable API origin. For local use, http://localhost:8080 is appropriate.
• Set PROBARA_SECRETS_KEY before storing monitor credentials, notification secrets, location credentials, or tenant AI keys; see encryption keys for format and rotation.
• Do not set PUBLIC_NATS_URL until NATS authentication and an externally reachable TLS/WSS endpoint are ready.

Local-process stack

make start-all-local

# Logs
tail -f /tmp/probara-*.log
tail -f /tmp/probara-ui.log

# Matching lifecycle
make restart-all-local
make stop-all-local

The target starts PostgreSQL and NATS, bootstraps local database access, validates Go, runs migrations, builds downloadable collector binaries, builds and launches each Go service, and starts Next.js on 0.0.0.0:3000.

.dev-secrets.key

Gitignored, persistent local encryption key created by the launcher.

/tmp/probara-*.pid and /tmp/probara-*.log

Local service process tracking and logs used by the start/stop scripts.

Docker Compose stack

make start-all
make ps
make logs
make healthcheck
make stop-all

docker compose up -d
docker compose ps
docker compose logs -f
docker compose down

Compose contract gaps

• API, scheduler, and worker all use stream check-jobs and subject check.job, so scheduled and on-demand jobs share one subject contract. If you change these values, change them on all three services together.
• The root .env file is not passed as a service env_file; only variables explicitly declared in Compose reach containers. PROBARA_SECRETS_KEY is forwarded from the shell environment to the api, scheduler, worker, and alerter services.
• AI, SMTP, asynchronous notifications, public NATS authorization, scheduler mesh/purge, and many result-ingest settings are not wired.
• The commented private-location worker example omits required LOCATION_CREDENTIAL and should not be enabled as written.
• The API and worker correctly share the synthetic_artifacts volume used for failure screenshots. Trace and HAR capture are configured by the schema but are not implemented by the current worker.

Install with Helm

image:
  tag: "pin-an-immutable-release"

secrets:
  adminJwtSecret: "replace-with-at-least-32-random-characters"
  probaraSecretsKey: "base64-encoded-32-byte-key"

api:
  publicBaseURL: "https://probara.example.com"

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: probara.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: probara-tls
      hosts:
        - probara.example.com

helm lint ./helm/monitoring-platform \
  --set secrets.adminJwtSecret="$(openssl rand -hex 32)" \
  --set api.publicBaseURL="https://probara.example.com"

helm upgrade --install probara ./helm/monitoring-platform \
  --namespace probara \
  --create-namespace \
  --values values.prod.yaml

kubectl -n probara get pods,svc,ingress
kubectl -n probara get jobs

Helm values: images and infrastructure

Helm values: auth and service workloads

Each service group also accepts a standard resources.requests and resources.limits object. Defaults are intentionally small and should be sized from observed check volume, job latency, database load, and notification throughput.

Helm values: status, frontend, ingress, and pod policy

Secrets from an external secret manager

Every credential the chart consumes can come from a Secret you supply instead of a literal in your values file. The chart takes no dependency on any particular tool: External Secrets Operator, the Vault agent injector, sealed-secrets, SOPS, and a hand-written kubectl create secret all end at the same place — a Secret in the release namespace — and the chart only ever names it.

Each field resolves its source in a fixed order: its own <field>ExistingSecret, then the chart-wide secrets.existingSecret, then the Secret the chart renders from your literal values. Sources mix field by field, so an install can keep the JWT chart-managed while the encryption key comes from Vault and the database DSN from a cloud secret store.

secrets:
  # The chart then renders no Secret of its own.
  existingSecret: probara-platform-secrets

postgresql:
  enabled: false
  existingSecret: rds-probara       # key: postgres_url (override with existingSecretKey)

nats:
  existingSecret: nats-platform     # key: nats_url

secrets:
  adminJwtSecret: "kept-in-the-values-file-for-this-install"
  probaraSecretsKeyExistingSecret: vault-probara
  probaraSecretsKeyExistingSecretKey: encryption_key
  oidcClientSecretExistingSecret: vault-probara

worker:
  locations:
    - name: eu-west
      locationId: "location-uuid-from-probara"
      existingSecret: probara-location-eu-west

• Per-field <field>ExistingSecretKey values default to the canonical key names above, so a Secret laid out with those names needs no key overrides.
• A key the chart asks for but the Secret does not carry leaves the pod in CreateContainerConfigError. That is deliberate: a missing encryption key should stop a rollout, not quietly disable encryption.
• Helm cannot read Secrets, so it cannot assemble a URL around a value it does not have: postgresql.auth.existingSecret needs the DSN externalized too, and nats.auth.platformPasswordExistingSecret needs the broker URL externalized. Either the matching per-field reference or the chart-wide secrets.existingSecret satisfies that; setting neither fails the render with that explanation.
• The Secret must exist before the pods start. With External Secrets Operator, that means the ExternalSecret must have synced — the migrations Job runs as a post-install/post-upgrade hook and needs postgres_url at that moment.
• Rotating a value in the Secret does not restart the workloads; the chart sets no checksum annotation. Restart the affected Deployments yourself after a rotation.
• Rotation keys (PROBARA_SECRETS_KEY_V2 and higher) have no dedicated value and go through the top-level extraEnv, which accepts a valueFrom.secretKeyRef entry like any Kubernetes EnvVar.

External PostgreSQL and NATS

postgresql:
  enabled: false
  externalUrl: "postgres://probara:REDACTED@postgres.example.internal:5432/probara?sslmode=require"

nats:
  enabled: false
  externalUrl: "tls://probara-platform:REDACTED@nats.example.internal:4222"

api:
  publicBaseURL: "https://probara.example.com"
  # Set only if private locations can reach this broker address.
  publicNatsURL: "tls://nats.example.com:4222"

• Use TLS certificate verification for both PostgreSQL and NATS. Do not copy the example with sslmode=disable into production.
• Provision JetStream storage for CHECK_JOBS, CHECK_RESULTS, alert, notification, and AI workloads.
• Keep the platform NATS credential separate from per-location credentials and restrict broker/network access.
• External URLs written as literals live in your values file. Point postgresql.existingSecret and nats.existingSecret at Secrets instead to keep them out of it — either way the chart reads both URLs from a Secret at runtime, never from a PodSpec.
• Test migrations against a backup before changing application versions.

Deploy private-location workers

A private-location worker needs only an authenticated NATS endpoint, its location UUID, and its location credential. It must not receive direct PostgreSQL access. Generate deploy information in the Locations UI/API after NATS authorization is configured.

worker:
  locations:
    - name: eu-west
      locationId: "location-uuid-from-probara"
      credential: "generated-location-credential"
      natsUrl: "tls://location-id:credential@nats.example.com:4222"
      # Or replace both with a Secret in the release namespace:
      #   existingSecret: probara-location-eu-west
      #   natsUrlKey: nats_url          # default
      #   credentialKey: credential     # default
      replicas: 1
      concurrency: "10"
      meshService:
        enabled: true
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"

• Use a DNS-safe unique name; each entry creates a separate Deployment.
• Set existingSecret on an entry to take natsUrl and credential from a Secret instead of the values file; the literals are then no longer required and no location credential appears in the PodSpec.
• The chart does not expose HTTP egress policy; provide HTTP_BLOCK_PRIVATE_IPS and HTTP_ALLOWED_CIDRS through extra environment configuration, and keep any allowlist limited to networks that location is explicitly trusted to monitor.
• Expose the mesh echo endpoint only on private inter-location networks.
• For locations outside the Kubernetes cluster, use the generated container/deployment snippet rather than granting database access.
• The embedded NATS Service is ClusterIP-only; an external location needs a separate TLS/WSS exposure path.

Scaling and availability

make scale-workers N=3

Migrations, upgrades, and rollback

1. Back up PostgreSQL and record the running application/chart/image versions.
2. Render or lint the target chart with production values and inspect Secrets, Services, queue variables, and image tags.
3. Run the target migrations against a disposable copy or staging database.
4. Deploy the migration-compatible application version. The chart migration Job is a post-install/post-upgrade hook; API startup also runs shared migrations.
5. Verify /readyz, result ingestion, scheduled and on-demand checks, alert delivery, status invalidation, and private locations.
6. Roll back application images only when the schema remains backward-compatible. Database migrations are not automatically reversed by helm rollback.

Current Helm packaging gaps

Production checklist

• Pin immutable backend, frontend, collector, migration, PostgreSQL, and NATS versions.
• Use managed/HA PostgreSQL and NATS or define tested backup and restore objectives for embedded state.
• Configure HTTPS, ADMIN_COOKIE_SECURE=true, a correct PUBLIC_BASE_URL, and trusted reverse-proxy headers.
• Use stable random JWT, encryption, OIDC, preview, SMTP, webhook, database, and NATS secrets, sourced from your secret manager rather than written into the values file.
• Align every NATS stream, subject, consumer, and status-update subject across services.
• Keep private-destination blocking enabled and allow only narrowly scoped CIDRs.
• Share or externalize synthetic-browser artifact storage.
• Provide secure TLS/WSS NATS exposure before enabling remote private locations.
• Correct ingress routing and verify browser client URLs in the built frontend image.
• Scrape the listeners that actually serve /metrics; alert on readiness failures, result-ingest errors, queue backlog, and migration failure.
• Run the complete verification suite and a restore drill before launch.