Operations

Service dependency map

Start, stop, inspect, and restart

make start-all-local
make restart-all-local
make stop-all-local

tail -f /tmp/probara-*.log
tail -f /tmp/probara-ui.log

make start-all
make ps
make logs
make logs-api
make logs-scheduler
make logs-worker
make logs-status
make restart-all
make stop-all

kubectl -n probara get pods,deployments,statefulsets,jobs
kubectl -n probara get events --sort-by=.lastTimestamp
kubectl -n probara logs deployment/probara-api --all-containers --tail=200
kubectl -n probara logs deployment/probara-scheduler --all-containers --tail=200
kubectl -n probara rollout status deployment/probara-api
kubectl -n probara rollout status deployment/probara-worker

Health, readiness, and listener truth

/healthz answers liveness, /readyz checks required dependencies, and /metrics exposes Prometheus data. The port carrying those routes differs by service.

curl -fsS http://localhost:8080/healthz
curl -fsS http://localhost:8080/readyz
curl -fsS http://localhost:9091/readyz
curl -fsS http://localhost:9092/readyz
curl -fsS http://localhost:9093/readyz
curl -fsS http://localhost:9094/readyz
curl -fsS -H "Authorization: Bearer $LOCAL_AI_TOKEN" http://127.0.0.1:8090/readyz

make healthcheck

Worker also serves /mesh/echo on the metrics listener. When its HTTP_PORT differs, it starts a second listener containing only mesh echo and /healthz, allowing mesh exposure without publishing /metrics.

Prometheus metrics

Custom metrics use namespace probara and a sanitized service subsystem, for example probara_scheduler_loops_total. Go runtime and process collectors are also registered.

curl -fsS http://localhost:8080/metrics | grep '^probara_'
curl -fsS http://localhost:9091/metrics | grep '^probara_scheduler_'
curl -fsS http://localhost:9092/metrics | grep '^probara_worker_'
curl -fsS http://localhost:9093/metrics | grep '^probara_status_page_'
curl -fsS http://127.0.0.1:8090/metrics

The local AI service runs one batch at a time and admits at most LOCAL_AI_QUEUE_CAPACITY more. A batch beyond that, or one that waits longer than LOCAL_AI_QUEUE_TIMEOUT_SECONDS (default 60 s, a few full batches) for the executing slot, gets HTTP 429 with Retry-After: 5. The worker treats that as transient: an incident analysis or alert triage is redelivered 1 minute and then 2 minutes later (or after the service’s Retry-After, at most 2 minutes), and the third failed delivery records it as failed; a failed analysis carries the action to check readiness and retry. Dependency suggestions, which the API computes on request, answer 503 unavailable with Retry-After instead of an error. A batch still running after LOCAL_AI_MAX_INFERENCE_SECONDS (default 120 s) is wedged: /healthz answers 503 and probara_ai_wedged is 1 until it finishes, so the Helm liveness probe restarts the pod; under Compose the container turns unhealthy and needs docker compose restart local-ai. Bodies over 512 KiB get 413 and evidence outside the allowlist 422. The chart does not annotate this pod for scraping; see local AI values.

Logs and diagnostic context

• Use LOG_LEVEL=debug temporarily for connection, scheduling, and readiness diagnosis; restore info after the incident.
• Correlate monitor ID, tenant ID, location ID, alert ID, NATS stream/subject, and timestamps across services.
• Never paste location-authenticated NATS URLs, API keys, SMTP passwords, webhook secrets, OIDC secrets, JWT secrets, or encrypted configuration keys into tickets or chat.
• A warning that status-update publishing failed means core CRUD may work while public live invalidation is degraded.
• A scheduler warning that RESULT_INGEST_ENABLED=false means workers can run checks but results will not enter PostgreSQL.
• A plaintext/NoOp encryption warning requires immediate review before storing or updating sensitive configurations.

docker compose logs --since=15m api scheduler worker alerter status-page
docker compose logs -f --tail=200 scheduler worker

Database migrations and administrator bootstrap

make migrate

# Direct local migration command
POSTGRES_URL='postgres://probara:probara@localhost:5432/probara?sslmode=disable' \
MIGRATIONS_PATH='./shared/db/migrations' \
go run ./cmd/migrate

The migration CLI requires POSTGRES_URL, defaults MIGRATIONS_PATH to ./migrations, and retries the initial database connection up to 30 times at one-second intervals. API startup also runs the shared migration set. The Helm migration Job is a post-install/post-upgrade hook.

POSTGRES_URL='postgres://probara:probara@localhost:5432/probara?sslmode=disable' \
ADMIN_USERNAME='admin' \
ADMIN_PASSWORD='use-a-password-manager-generated-value' \
ADMIN_BCRYPT_COST='12' \
go run ./cmd/admin

The command always leaves a platform superadministrator behind, with no tenant memberships (a superadministrator reaches every tenant), matching the first-user form. Unlike that form it also works on a populated install: run against an existing username, it resets the password, re-enables a disabled account and promotes it to superadministrator, which makes it the break-glass recovery path for an operator with database access.

Backup and restore

make db-backup       # writes ./backup.sql

# Inspect and copy the backup away from the workstation before maintenance.
ls -lh backup.sql

# Restore into the configured Compose database.
make db-restore

1. Quiesce writes or capture a database-consistent snapshot appropriate to your PostgreSQL topology.
2. Back up PostgreSQL, including schema migration state, tenants, monitor configuration, alert state, users, API keys, audit data, and encrypted ciphertext.
3. Back up the complete encryption keyring separately. A database backup without the historical keys may be unrecoverable.
4. Protect JetStream state or accept that queued checks/results/notifications may be replayed or lost after recovery.
5. Preserve browser artifacts separately if they are part of your incident evidence policy.
6. Restore into an isolated environment, run readiness and data-integrity checks, then perform a documented cutover.

The bundled PostgreSQL and NATS StatefulSets use PVCs but provide no scheduled backup controller, point-in-time recovery, replication, or cross-zone failover. Production recovery objectives require managed services, an operator, or external backup automation.

Retention, rollups, and deletion

Tenant telemetry retention

Raw check results are kept 30 days for every tenant, including tenants set to 0; ranges of 7 days and longer, status-page history, and latency-anomaly baselines read the hourly and daily rollups, which are kept 400 days. Cleanup removes whole hours only, starting one hour below the 30-day horizon, and keeps any hour still waiting for its rollup rebuild until the rebuild runs; a result that arrives for an hour already past the horizon is stored but not folded into rollups. Each tenant’s data_retention_days controls mesh results and can tighten (never extend) raw agent metric-sample retention below METRIC_RAW_RETENTION_DAYS. A value of 0 preserves mesh results; the public validator otherwise accepts 30–3650 days. It is configured in tenant settings.

State history and SLA reports

Monitor state intervals (the timeline behind time-based availability and SLA reports) are not pruned by time; they are removed only with a monitor’s history or when a deleted monitor is purged. Issued SLA reports are kept until their SLA is deleted and survive monitor purges.

Alert triage

Each alert_triage row lives as long as its alert: it is deleted with the alert (monitor purge or history deletion) or with its workspace, and is not pruned by time or by data_retention_days. Labels on a triage outlive it, like other AI labels.

AI training labels

Operator labels on AI analyses, alert triages and dependency suggestions are not pruned by time or by data_retention_days, and survive monitor purges and incident deletion; they are removed only with their workspace.

Audit retention

AUDIT_RETENTION_DAYS is a separate compliance control, defaults to 365 days, and uses 0 to retain forever.

Monitor purge

Soft-deleted monitors are later purged with dependent rows in bounded batches.

Rollups

Result ingestion marks each affected hour in a dirty-bucket ledger within the same transaction, and scheduler maintenance rebuilds marked buckets wholesale from raw results — late or redelivered rows can never be permanently skipped. A failed batch leaves its marks for the next run.

Scheduler checks once per minute whether the configured UTC cleanup hour is due, runs at most once per UTC date, uses a PostgreSQL advisory lock, and applies a 30-minute cleanup timeout. Batch size and maximum rows per run bound database pressure.

NATS and JetStream operations

nats --server "$NATS_URL" server check connection
nats --server "$NATS_URL" stream ls
nats --server "$NATS_URL" stream info CHECK_JOBS
nats --server "$NATS_URL" consumer ls CHECK_JOBS
nats --server "$NATS_URL" stream info CHECK_RESULTS
nats --server "$NATS_URL" consumer info CHECK_RESULTS result-ingest

• Compare the actual stream subjects with every publisher and consumer environment before deleting or recreating streams.
• Watch pending, redelivered, and unacknowledged counts; queue depth without scheduler/worker progress indicates a contract or connectivity failure.
• CHECK_JOB_LEGACY_CONSUMERS causes scheduler startup to delete named filterless consumers. Review the list before renaming durable consumers.
• JetStream PVC persistence does not replace a broker backup/replication strategy.
• Core NATS status updates are ephemeral by design; status cache TTL is the fallback.

Synthetic-browser artifacts

• Workers write failure screenshots beneath SYNTHETIC_BROWSER_ARTIFACTS_DIR; API reads them for authenticated retrieval. Trace and HAR requests currently emit warnings and do not create artifacts.
• Compose mounts a shared named volume into API and worker.
• Local-process services share the same host directory.
• The current Helm workloads do not mount shared artifact storage, so cross-pod and cross-node retrieval is unreliable.
• Set retention and access controls appropriate for screenshots because they can capture page content, tokens, personal data, or internal application state.

Verification and tests

make test
make lint

# make fmt is a check, not a formatter.
gofmt -w path/to/changed.go
make fmt
make vet

(cd web && npm run lint && npm run build)
(cd website && npm run typecheck && npm run build)

helm lint ./helm/monitoring-platform \
  --set secrets.adminJwtSecret='replace-with-32-plus-char-secret-value' \
  --set api.publicBaseURL='https://probara.example.com'

• make test runs the Go module with race detection and writes coverage.out; it excludes script packages. There is no separate agent module or test suite — the collector is a prebuilt OpenTelemetry distribution (make build-collector-static).
• Database integration tests use Testcontainers and require a working Docker daemon capable of starting PostgreSQL 16.
• Run make test-cover to open the HTML coverage report.
• Frontend changes require both lint and a production build; documentation site changes should at least run npm run typecheck and npm run build from website/.
• Helm lint requires the chart’s mandatory JWT and public base URL values.
• For a release candidate, verify scheduled checks, on-demand checks, all touched monitor types, result ingest, alerts, notification delivery, status pages, OIDC, imports, private locations, and backup restore.

Troubleshooting guide

Incident response checklist

1. Identify the affected tenant, monitor/location, time window, and whether the failure is control plane, check execution, result persistence, alerting, or presentation.
2. Check liveness and readiness on the real service listeners; preserve logs and relevant metrics before restarting.
3. Inspect PostgreSQL availability and NATS stream/consumer state without deleting consumers or messages.
4. Compare deployed queue variables and encryption-key versions across all replicas.
5. Reduce blast radius with reversible actions: pause a broken monitor/policy, scale a healthy worker fleet, or disable optional async paths.
6. Restore service, verify end-to-end from schedule through status/alert delivery, then reconcile delayed or duplicate jobs.
7. Record the operator action in incident notes and retain the audit/log/artifact evidence according to policy.