Operations

Service dependency map

Start, stop, inspect, and restart

make start-all-local
make restart-all-local
make stop-all-local

tail -f /tmp/probara-*.log
tail -f /tmp/probara-ui.log

make start-all
make ps
make logs
make logs-api
make logs-scheduler
make logs-worker
make logs-status
make restart-all
make stop-all

kubectl -n probara get pods,deployments,statefulsets,jobs
kubectl -n probara get events --sort-by=.lastTimestamp
kubectl -n probara logs deployment/probara-api --all-containers --tail=200
kubectl -n probara logs deployment/probara-scheduler --all-containers --tail=200
kubectl -n probara rollout status deployment/probara-api
kubectl -n probara rollout status deployment/probara-worker

Health, readiness, and listener truth

/healthz answers liveness, /readyz checks required dependencies, and /metrics exposes Prometheus data. The port carrying those routes differs by service.

curl -fsS http://localhost:8080/healthz
curl -fsS http://localhost:8080/readyz
curl -fsS http://localhost:9091/readyz
curl -fsS http://localhost:9092/readyz
curl -fsS http://localhost:9093/readyz
curl -fsS http://localhost:9094/readyz

make healthcheck

Worker also serves /mesh/echo on the metrics listener. When its HTTP_PORT differs, it starts a second listener containing only mesh echo and /healthz, allowing mesh exposure without publishing /metrics.

Prometheus metrics

Custom metrics use namespace probara and a sanitized service subsystem, for example probara_scheduler_loops_total. Go runtime and process collectors are also registered.

curl -fsS http://localhost:8080/metrics | grep '^probara_'
curl -fsS http://localhost:9091/metrics | grep '^probara_scheduler_'
curl -fsS http://localhost:9092/metrics | grep '^probara_worker_'
curl -fsS http://localhost:9093/metrics | grep '^probara_status_page_'

Logs and diagnostic context

• Use LOG_LEVEL=debug temporarily for connection, scheduling, and readiness diagnosis; restore info after the incident.
• Correlate monitor ID, tenant ID, location ID, alert ID, NATS stream/subject, and timestamps across services.
• Never paste location-authenticated NATS URLs, API keys, SMTP passwords, webhook secrets, OIDC secrets, JWT secrets, or encrypted configuration keys into tickets or chat.
• A warning that status-update publishing failed means core CRUD may work while public live invalidation is degraded.
• A scheduler warning that RESULT_INGEST_ENABLED=false means workers can run checks but results will not enter PostgreSQL.
• A plaintext/NoOp encryption warning requires immediate review before storing or updating sensitive configurations.

docker compose logs --since=15m api scheduler worker alerter status-page
docker compose logs -f --tail=200 scheduler worker

Database migrations and administrator bootstrap

make migrate

# Direct local migration command
POSTGRES_URL='postgres://probara:probara@localhost:5432/probara?sslmode=disable' \
MIGRATIONS_PATH='./shared/db/migrations' \
go run ./cmd/migrate

The migration CLI requires POSTGRES_URL, defaults MIGRATIONS_PATH to ./migrations, and retries the initial database connection up to 30 times at one-second intervals. API startup also runs the shared migration set. The Helm migration Job is a post-install/post-upgrade hook.

POSTGRES_URL='postgres://probara:probara@localhost:5432/probara?sslmode=disable' \
ADMIN_USERNAME='admin' \
ADMIN_PASSWORD='use-a-password-manager-generated-value' \
ADMIN_BCRYPT_COST='12' \
go run ./cmd/admin

Backup and restore

make db-backup       # writes ./backup.sql

# Inspect and copy the backup away from the workstation before maintenance.
ls -lh backup.sql

# Restore into the configured Compose database.
make db-restore

1. Quiesce writes or capture a database-consistent snapshot appropriate to your PostgreSQL topology.
2. Back up PostgreSQL, including schema migration state, tenants, monitor configuration, alert state, users, API keys, audit data, and encrypted ciphertext.
3. Back up the complete encryption keyring separately. A database backup without the historical keys may be unrecoverable.
4. Protect JetStream state or accept that queued checks/results/notifications may be replayed or lost after recovery.
5. Preserve browser artifacts separately if they are part of your incident evidence policy.
6. Restore into an isolated environment, run readiness and data-integrity checks, then perform a documented cutover.

The bundled PostgreSQL and NATS StatefulSets use PVCs but provide no scheduled backup controller, point-in-time recovery, replication, or cross-zone failover. Production recovery objectives require managed services, an operator, or external backup automation.

Retention, rollups, and deletion

Tenant telemetry retention

Each tenant’s data_retention_days controls raw check results and mesh results, and can tighten (never extend) raw agent metric-sample retention below METRIC_RAW_RETENTION_DAYS. A value of 0 preserves check/mesh results; the public validator otherwise accepts 30–3650 days. It is configured in tenant settings.

Audit retention

AUDIT_RETENTION_DAYS is a separate compliance control, defaults to 365 days, and uses 0 to retain forever.

Monitor purge

Soft-deleted monitors are later purged with dependent rows in bounded batches.

Rollups

Result ingestion marks each affected hour in a dirty-bucket ledger within the same transaction, and scheduler maintenance rebuilds marked buckets wholesale from raw results — late or redelivered rows can never be permanently skipped. A failed batch leaves its marks for the next run.

Scheduler checks once per minute whether the configured UTC cleanup hour is due, runs at most once per UTC date, uses a PostgreSQL advisory lock, and applies a 30-minute cleanup timeout. Batch size and maximum rows per run bound database pressure.

NATS and JetStream operations

nats --server "$NATS_URL" server check connection
nats --server "$NATS_URL" stream ls
nats --server "$NATS_URL" stream info CHECK_JOBS
nats --server "$NATS_URL" consumer ls CHECK_JOBS
nats --server "$NATS_URL" stream info CHECK_RESULTS
nats --server "$NATS_URL" consumer info CHECK_RESULTS result-ingest

• Compare the actual stream subjects with every publisher and consumer environment before deleting or recreating streams.
• Watch pending, redelivered, and unacknowledged counts; queue depth without scheduler/worker progress indicates a contract or connectivity failure.
• CHECK_JOB_LEGACY_CONSUMERS causes scheduler startup to delete named filterless consumers. Review the list before renaming durable consumers.
• JetStream PVC persistence does not replace a broker backup/replication strategy.
• Core NATS status updates are ephemeral by design; status cache TTL is the fallback.

Synthetic-browser artifacts

• Workers write failure screenshots beneath SYNTHETIC_BROWSER_ARTIFACTS_DIR; API reads them for authenticated retrieval. Trace and HAR requests currently emit warnings and do not create artifacts.
• Compose mounts a shared named volume into API and worker.
• Local-process services share the same host directory.
• The current Helm workloads do not mount shared artifact storage, so cross-pod and cross-node retrieval is unreliable.
• Set retention and access controls appropriate for screenshots because they can capture page content, tokens, personal data, or internal application state.

Verification and tests

make test
make lint

# make fmt is a check, not a formatter.
gofmt -w path/to/changed.go
make fmt
make vet

(cd web && npm run lint && npm run build)
(cd website && npm run typecheck && npm run build)

helm lint ./helm/monitoring-platform \
  --set secrets.adminJwtSecret='replace-with-32-plus-char-secret-value' \
  --set api.publicBaseURL='https://probara.example.com'

• make test runs the Go module with race detection and writes coverage.out; it excludes script packages. There is no separate agent module or test suite — the collector is a prebuilt OpenTelemetry distribution (make build-collector-static).
• Database integration tests use Testcontainers and require a working Docker daemon capable of starting PostgreSQL 16.
• Run make test-cover to open the HTML coverage report.
• Frontend changes require both lint and a production build; documentation site changes should at least run npm run typecheck and npm run build from website/.
• Helm lint requires the chart’s mandatory JWT and public base URL values.
• For a release candidate, verify scheduled checks, on-demand checks, all touched monitor types, result ingest, alerts, notification delivery, status pages, OIDC, imports, private locations, and backup restore.

Troubleshooting guide

Incident response checklist

1. Identify the affected tenant, monitor/location, time window, and whether the failure is control plane, check execution, result persistence, alerting, or presentation.
2. Check liveness and readiness on the real service listeners; preserve logs and relevant metrics before restarting.
3. Inspect PostgreSQL availability and NATS stream/consumer state without deleting consumers or messages.
4. Compare deployed queue variables and encryption-key versions across all replicas.
5. Reduce blast radius with reversible actions: pause a broken monitor/policy, scale a healthy worker fleet, or disable optional async paths.
6. Restore service, verify end-to-end from schedule through status/alert delivery, then reconcile delayed or duplicate jobs.
7. Record the operator action in incident notes and retain the audit/log/artifact evidence according to policy.