Deployment

Choose a deployment workflow

Prerequisites

• Docker with Compose v2 for PostgreSQL, NATS, migrations, and container workflows.
• GNU Make and a Bash-compatible environment for repository scripts. On Windows, use WSL or another environment that can execute the Makefile’s Bash commands.
• Git LFS to build the local AI image from source: run git lfs install and git lfs pull --include 'ai-service/artifacts/*/head.safetensors' once after cloning. Not needed when a release image is pulled or with LOCAL_AI_DISABLED=true.
• Go 1.26.5 for the main monorepo. The local launcher compares the installed major/minor version with the root go.mod directive.
• Node.js LTS and npm for the frontend. scripts/start-ui.sh can use nvm when it is available.
• Helm 3 and kubectl for Kubernetes deployment.
• A PostgreSQL 16-compatible database and NATS 2.10 with JetStream for production when not using embedded chart dependencies.

The host agent (probara-collector) is not a separate Go module: scripts/build-collector.sh generates and cross-compiles a minimal OpenTelemetry Collector distribution from collector/manifest.yaml with a pinned builder version. Root make test covers the whole repository.

Prepare required values

cp .env.example .env

# Generate stable secrets; keep them across restarts and upgrades.
openssl rand -hex 32       # ADMIN_JWT_SECRET
openssl rand -base64 32    # PROBARA_SECRETS_KEY

• Set ADMIN_JWT_SECRET to a random value of at least 32 characters. Validation only rejects an empty value, so shipped placeholder defaults must be replaced deliberately.
• Set PUBLIC_BASE_URL to the externally reachable API origin. For local use, http://localhost:8080 is appropriate.
• Set PROBARA_SECRETS_KEY before storing monitor credentials, notification secrets, location credentials, or tenant AI keys; see encryption keys for format and rotation.
• Do not set PUBLIC_NATS_URL until NATS authentication and an externally reachable TLS/WSS endpoint are ready.

Local-process stack

make start-all-local

# Logs
tail -f /tmp/probara-*.log
tail -f /tmp/probara-ui.log

# Matching lifecycle
make restart-all-local
make stop-all-local

The target starts PostgreSQL and NATS, bootstraps local database access, validates Go, runs migrations, builds downloadable collector binaries, builds and launches each Go service, starts the local-ai container in the background (the first image build takes several minutes; progress in /tmp/probara-local-start.log), and starts Next.js on 0.0.0.0:3000. Until the container is up, or if it fails to start, API and worker report the model unavailable; the platform is never blocked. With no local image and the head weights still a Git LFS pointer, it only tries a pull. A local build is reused as is, so rebuild it with docker compose build local-ai after an ai-service/ change. LOCAL_AI_DISABLED=true skips the container and turns the feature off.

.dev-secrets.key

Gitignored, persistent local encryption key created by the launcher.

.dev-local-ai-token

Gitignored per-checkout local AI bearer token, created on first start when LOCAL_AI_TOKEN is not exported, and passed to the container, API and worker.

/tmp/probara-*.pid and /tmp/probara-*.log

Local service process tracking and logs used by the start/stop scripts.

Docker Compose stack

make start-all
make ps
make logs
make healthcheck
make stop-all

docker compose up -d
docker compose ps
docker compose logs -f
docker compose down

Local AI is on by default. Compose pulls LOCAL_AI_IMAGE and builds it from ai-service/ only when the pull fails; that build fails on purpose unless the committed artifact is release-approved, and a failed build aborts a whole docker compose up. The Make targets (up, build, rebuild, scale-workers, start-all, start-all-local) therefore start the platform first and local-ai separately, tolerating its failure. With plain Compose and no pullable approved image, name the services you want (docker compose up -d postgres nats migrations api scheduler worker status-page alerter) or set LOCAL_AI_IMAGE to an image you already have. LOCAL_AI_DISABLED=true turns the feature off and makes the Make targets skip local-ai, but plain docker compose up still pulls or builds the service.

Compose contract gaps

• API, scheduler, and worker all use stream check-jobs and subject check.job, so scheduled and on-demand jobs share one subject contract. If you change these values, change them on all three services together.
• The root .env file is not passed as a service env_file; only variables explicitly declared in Compose reach containers. PROBARA_SECRETS_KEY is forwarded from the shell environment to the api, scheduler, worker, and alerter services.
• SMTP, asynchronous notifications, public NATS authorization, scheduler mesh/purge, and many result-ingest settings are not wired.
• Without an exported LOCAL_AI_TOKEN (shell or .env), local-ai, API and worker share a fixed development token that is public in the repository. That is tolerable only because port 8090 is bound to loopback; set your own (openssl rand -hex 32) on any shared host. make start-all-local generates a per-checkout token instead.
• The commented private-location worker example omits required LOCATION_CREDENTIAL and should not be enabled as written.
• The API and worker correctly share the synthetic_artifacts volume used for failure screenshots. Trace and HAR capture are configured by the schema but are not implemented by the current worker.

Install with Helm

image:
  tag: "pin-an-immutable-release"

secrets:
  adminJwtSecret: "replace-with-at-least-32-random-characters"
  probaraSecretsKey: "base64-encoded-32-byte-key"

api:
  publicBaseURL: "https://probara.example.com"

ingress:
  enabled: true
  className: nginx
  hosts:
    - host: probara.example.com
      paths:
        - path: /
          pathType: Prefix
  tls:
    - secretName: probara-tls
      hosts:
        - probara.example.com

helm lint ./helm/monitoring-platform \
  --set secrets.adminJwtSecret="$(openssl rand -hex 32)" \
  --set api.publicBaseURL="https://probara.example.com"

helm upgrade --install probara ./helm/monitoring-platform \
  --namespace probara \
  --create-namespace \
  --values values.prod.yaml

kubectl -n probara get pods,svc,ingress
kubectl -n probara get jobs

Helm values: images and infrastructure

Helm values: auth and service workloads

Each service group also accepts a standard resources.requests and resources.limits object. Defaults are intentionally small and should be sized from observed check volume, job latency, database load, and notification throughput.

Helm values: status, frontend, ingress, and pod policy

Helm values: local AI

The chart deploys the bundled CPU inference service by default and configures only the API and the platform worker to call it. The alerter never calls it and receives only the LOCAL_AI_DISABLED and AI_TRIAGE_ENABLED kill switches; scheduler, status page and location workers never receive LOCAL_AI_*.

The token resolves like every other chart credential: secrets.localAiTokenExistingSecret, then the chart-wide secrets.existingSecret, then the chart-managed Secret. When chart-managed and left empty, the chart derives it from secrets.adminJwtSecret (a SHA-256 over a domain-separated copy) instead of generating a random value, so helm template and GitOps renderers produce the same Secret on every render. The derived token is as private as your JWT secret: with the shipped placeholder JWT secret it is a well-known value, and rotating adminJwtSecret rotates it, after which API, worker and local-ai need a restart together.

• The render fails, asking for secrets.localAiToken or an existing Secret, when localAI.externalUrl is set (the operator-run service must hold the same token) or when adminJwtSecret is empty or supplied by an existing Secret (the chart cannot read it to derive from).
• API and worker reference the token as optional, so an existing Secret that predates local AI does not stop them on upgrade: they start and report local AI as not configured correctly. The local AI pod’s reference is required and it waits in CreateContainerConfigError until the key exists. The chart’s install/upgrade notes print this warning when secrets.existingSecret is set without a per-field token Secret.
• Readiness execs the image’s Python against the authenticated /readyz, because an httpGet probe cannot read the token from a Secret; like the service and the clients, it ignores whitespace around the token. The startup probe allows five minutes for the model to load. Liveness polls /healthz every 15 seconds and restarts the pod after three failures, which happens once a batch has run past maxInferenceSeconds: a hung inference cannot be cancelled in the process, and with one replica only a restart recovers.

Secrets from an external secret manager

Every credential the chart consumes can come from a Secret you supply instead of a literal in your values file. The chart takes no dependency on any particular tool: External Secrets Operator, the Vault agent injector, sealed-secrets, SOPS, and a hand-written kubectl create secret all end at the same place — a Secret in the release namespace — and the chart only ever names it.

Each field resolves its source in a fixed order: its own <field>ExistingSecret, then the chart-wide secrets.existingSecret, then the Secret the chart renders from your literal values. Sources mix field by field, so an install can keep the JWT chart-managed while the encryption key comes from Vault and the database DSN from a cloud secret store.

secrets:
  # The chart then renders no Secret of its own.
  existingSecret: probara-platform-secrets

postgresql:
  enabled: false
  existingSecret: rds-probara       # key: postgres_url (override with existingSecretKey)

nats:
  existingSecret: nats-platform     # key: nats_url

secrets:
  adminJwtSecret: "kept-in-the-values-file-for-this-install"
  probaraSecretsKeyExistingSecret: vault-probara
  probaraSecretsKeyExistingSecretKey: encryption_key
  oidcClientSecretExistingSecret: vault-probara

worker:
  locations:
    - name: eu-west
      locationId: "location-uuid-from-probara"
      existingSecret: probara-location-eu-west

• Per-field <field>ExistingSecretKey values default to the canonical key names above, so a Secret laid out with those names needs no key overrides.
• A key the chart asks for but the Secret does not carry leaves the pod in CreateContainerConfigError. That is deliberate: a missing encryption key should stop a rollout, not quietly disable encryption.
• Helm cannot read Secrets, so it cannot assemble a URL around a value it does not have: postgresql.auth.existingSecret needs the DSN externalized too, and nats.auth.platformPasswordExistingSecret needs the broker URL externalized. Either the matching per-field reference or the chart-wide secrets.existingSecret satisfies that; setting neither fails the render with that explanation.
• The Secret must exist before the pods start. With External Secrets Operator, that means the ExternalSecret must have synced — the migrations Job runs as a post-install/post-upgrade hook and needs postgres_url at that moment.
• Rotating a value in the Secret does not restart the workloads; the chart sets no checksum annotation. Restart the affected Deployments yourself after a rotation.
• Rotation keys (PROBARA_SECRETS_KEY_V2 and higher) have no dedicated value and go through the top-level extraEnv, which accepts a valueFrom.secretKeyRef entry like any Kubernetes EnvVar.

External PostgreSQL and NATS

postgresql:
  enabled: false
  externalUrl: "postgres://probara:REDACTED@postgres.example.internal:5432/probara?sslmode=require"

nats:
  enabled: false
  externalUrl: "tls://probara-platform:REDACTED@nats.example.internal:4222"

api:
  publicBaseURL: "https://probara.example.com"
  # Set only if private locations can reach this broker address.
  publicNatsURL: "tls://nats.example.com:4222"

• Use TLS certificate verification for both PostgreSQL and NATS. Do not copy the example with sslmode=disable into production.
• Provision JetStream storage for CHECK_JOBS, CHECK_RESULTS, alert, notification, and AI workloads.
• Keep the platform NATS credential separate from per-location credentials and restrict broker/network access.
• External URLs written as literals live in your values file. Point postgresql.existingSecret and nats.existingSecret at Secrets instead to keep them out of it — either way the chart reads both URLs from a Secret at runtime, never from a PodSpec.
• Test migrations against a backup before changing application versions.

Deploy private-location workers

A private-location worker needs only an authenticated NATS endpoint, its location UUID, and its location credential. It must not receive direct PostgreSQL access. Generate deploy information in the Locations UI/API after NATS authorization is configured.

worker:
  locations:
    - name: eu-west
      locationId: "location-uuid-from-probara"
      credential: "generated-location-credential"
      natsUrl: "tls://location-id:credential@nats.example.com:4222"
      # Or replace both with a Secret in the release namespace:
      #   existingSecret: probara-location-eu-west
      #   natsUrlKey: nats_url          # default
      #   credentialKey: credential     # default
      replicas: 1
      concurrency: "10"
      meshService:
        enabled: true
        type: LoadBalancer
        annotations:
          service.beta.kubernetes.io/aws-load-balancer-internal: "true"

• Use a DNS-safe unique name; each entry creates a separate Deployment.
• Set existingSecret on an entry to take natsUrl and credential from a Secret instead of the values file; the literals are then no longer required and no location credential appears in the PodSpec.
• The chart does not expose HTTP egress policy; provide HTTP_BLOCK_PRIVATE_IPS and HTTP_ALLOWED_CIDRS through extra environment configuration, and keep any allowlist limited to networks that location is explicitly trusted to monitor.
• Expose the mesh echo endpoint only on private inter-location networks.
• For locations outside the Kubernetes cluster, use the generated container/deployment snippet rather than granting database access.
• The embedded NATS Service is ClusterIP-only; an external location needs a separate TLS/WSS exposure path.

Scaling and availability

make scale-workers N=3

Migrations, upgrades, and rollback

1. Back up PostgreSQL and record the running application/chart/image versions.
2. Render or lint the target chart with production values and inspect Secrets, Services, queue variables, and image tags.
3. Run the target migrations against a disposable copy or staging database.
4. Deploy the migration-compatible application version. The chart migration Job is a post-install/post-upgrade hook; API startup also runs shared migrations.
5. Verify /readyz, result ingestion, scheduled and on-demand checks, alert delivery, status invalidation, and private locations.
6. Roll back application images only when the schema remains backward-compatible. Database migrations are not automatically reversed by helm rollback.

Current Helm packaging gaps

Production checklist

• Pin immutable backend, frontend, local AI, collector, migration, PostgreSQL, and NATS versions.
• Use managed/HA PostgreSQL and NATS or define tested backup and restore objectives for embedded state.
• Configure HTTPS, ADMIN_COOKIE_SECURE=true, a correct PUBLIC_BASE_URL, and trusted reverse-proxy headers.
• Use stable random JWT, encryption, local AI token, OIDC, preview, SMTP, webhook, database, and NATS secrets, sourced from your secret manager rather than written into the values file.
• Confirm the cluster CNI enforces the local AI NetworkPolicy, or isolate the service another way.
• Align every NATS stream, subject, consumer, and status-update subject across services.
• Keep private-destination blocking enabled and allow only narrowly scoped CIDRs.
• Share or externalize synthetic-browser artifact storage.
• Provide secure TLS/WSS NATS exposure before enabling remote private locations.
• Correct ingress routing and verify browser client URLs in the built frontend image.
• Scrape the listeners that actually serve /metrics; alert on readiness failures, result-ingest errors, queue backlog, and migration failure.
• Run the complete verification suite and a restore drill before launch.