Observability and logging#

What the services record, where it goes, what can be monitored from outside, and what is not available. Written for operators building monitoring and for security officers asking about audit trails.

1. Logs#

Aspect	Behaviour
Library	pino (packages/logger), JSON lines with ISO timestamps and a level label; one logger per service (core-api, asset-processor, gateway)
Level	LOG_LEVEL (default info), switchable at run time without a restart through the logging toggle (Settings, and attachLoggingToggle in each service)
Destinations	standard output (pretty-printed in development, JSON in production, so Docker's log driver collects it), and a rotating file under LOG_DIR (daily or at 10 MB, pino-roll)
Central copy	on rotation the file is uploaded to the asset store under private/logs/<app>/<yyyy-mm>/ (needs DAM_SERVICE_TOKEN); the dashboard's System logs viewer reads the fresh local files and the uploaded ones
Retention	none enforced: rotated files stay in the bucket; local files stay in the container's log directory (lost with the container unless LOG_DIR is mounted)
Redaction	req.headers.authorization, req.headers.cookie, headers.* equivalents, config.apiKey, config.apiSecret, config.password, config.token
Correlation id	every request has one id, in the X-Request-Id header and in the requestId field of every log line written while serving it (section 5)
Reverse proxy	nginx access and error logs on the host are separate and not managed by Whoobe
Dashboard	System logs (read, filter, download) with dates on every line; Settings > Mail > Log (every outgoing mail with status and body)

Recommended: ship container stdout to your central log platform (the Docker json-file/journald driver, a log agent), set a size limit on the driver, and mount LOG_DIR if you rely on the local files.

2. What is recorded (security-relevant events)#

Event	Where	Level
Failed login: unknown user, inactive user, wrong password (user id, not the password)	core-api log	warn
Failed sign-ins, lockouts, wrong second-factor codes, refused permissions, refused API clients	the audit log (auth.login_failed, auth.mfa_failed, access.permission_refused, api_client.refused; at most 10 rows a minute per address and action)	-
Permission denied, missing or invalid session	console.warn from the auth guard (reaches Docker's stdout, not the rotated file or its DAM copy) and 401/403 in the access log	warn
Form upload refused (upload id, website, reason)	asset-processor log	warn
Form notification skipped (not an authored block, recipient limit)	core-api log	warn
Outbound request refused by the SSRF guard	the calling service	warn
Workflow transitions of data objects	table workflow_transition_log	-
Page and snippet changes	table content_versions (who saved when)	-
Every outgoing mail	table mail_log	-
Sessions: created, last seen, IP, user agent	table sessions (id stored as a hash; rows deleted a week after expiry or revocation)	-

Administrative actions have an audit log (table audit_log, read with the permission audit:read): users and roles, websites, settings, publication, migration sources and object event subscriptions, with the actor, the address and the fields before and after, and no secrets, and the failed attempts listed in the row above. The editing of content (version history), a missing session and the administration of the Gateway are not in it. Evidence for those still comes from the nginx access log (method, path, status, address, but not the session user) and the application log.

3. Health and readiness#

Endpoint	Meaning
GET /health on core-api (3003)	readiness: 200 {status:'ok', database:'ok', uptimeSeconds, ...}, or 503 {status:'unavailable', database:'down'} when the database does not answer select 1 within 2 s (the answer is kept for 2 s, so an anonymous caller cannot make a query per request)
GET /health/live on core-api	liveness: the process is up, nothing else is asked
GET /health on asset-processor (3002) and gateway (3001)	liveness only, as before: they do not test the database or the object store
Docker HEALTHCHECK (30 s interval, 5 s timeout, 20 s start period, 3 retries)	calls /health/live on core-api and /health on the other two (a database outage does not make the core-api container look dead); Compose depends_on: service_healthy makes asset-processor and gateway wait for core-api (which runs the migrations)
the two SPAs	nginx on 8080; no health route (probe /)
PostgreSQL	pg_isready health check in the compose file
Garage	its own health check

Point a load balancer or an uptime probe at core-api's GET /health (it takes the instance out when the database is gone) and the Docker health check at /health/live. For the asset store and the gateway, probe a public read route end to end (for example GET /public/pages/…).

4. Metrics, tracing, alerting#

GET /metrics on core-api (Prometheus text format). It is off unless METRICS_TOKEN is set (404 without it); with it, a scrape sends Authorization: Bearer <token>. The nginx template does not route /metrics, so it is reachable on the service network and on the published port of core-api, not from the internet. Prometheus: authorization: { credentials: <token> } in the scrape job.

Metric	Meaning
whoobe_http_requests_total{method,route,status}	requests by method, matched route pattern (/auth/users/:id, never the URL; a request with no route is unmatched) and status class (2xx...)
whoobe_http_request_duration_seconds (histogram, route)	time to answer
whoobe_database_up	1 / 0, the same check as /health
whoobe_publish_jobs{status}	publish jobs by status (queued is the queue)
whoobe_object_event_deliveries{status}	event deliveries by status (pending and retrying are the queue, dead needs a look)
whoobe_audit_rows_written, whoobe_audit_rows_lost	audit rows since the process started; an alert on lost > 0
whoobe_build_info{service,version}, whoobe_process_uptime_seconds, whoobe_process_resident_memory_bytes, whoobe_process_heap_used_bytes	the build and the process

At most 500 routes are labelled, the rest are other. Counters are per process and restart at zero (a Prometheus rate() handles that). Asset-processor and gateway have no /metrics yet.

Not provided: OpenTelemetry instrumentation, request tracing, built-in alert rules, no uptime page. Use what the platform gives you:

Signal	How
Availability	external HTTP probes of the public site, the dashboard (/), /health through the proxy, and a database-backed route
Errors	rate of 5xx in the nginx access log; core-api/asset-processor/gateway error lines
Abuse	rate of 429 (per-address limits) and of 401/403; lines "Login failed", "Form upload refused"
Capacity	container CPU, memory and restarts (docker stats, cAdvisor); pgdata and garagedata disk use; PostgreSQL connections (pg_stat_activity), table size of sessions, mail_log, form_uploads
Certificates	days to expiry of the TLS certificate; the age of the last renew-certs.sh run
Backups	age of the newest encrypted dump and of the bucket sync (backup section 7)
Mail	the mail log's failed rate; the provider's own dashboard

5. Debugging a request#

The access log of the proxy shows the route and status; the service log at debug shows more (switch it on, reproduce, switch it off: debug logs may contain form values and addresses). Error responses in production carry a generic message; the real cause is in the log line written by the handler (logger.error({ err }, '<operation> error')).

Follow one request with its id. nginx passes on the client's X-Request-Id when it is 8 to 100 characters of A-Z a-z 0-9 . _ : -, otherwise sets its own ($request_id, 32 hex). core-api, the gateway and the asset processor do the same check (a request that reaches them without nginx gets a UUID), send the id back in the X-Request-Id response header, add it as requestId to every log line written while serving the request, write it in the audit row (request_id) and pass it on when they call each other: core-api to the asset processor, the gateway to core-api, to the asset processor and to the endpoints an administrator configured. So the id a user reads in the browser's network tab (or that a client sends) finds the lines of all the services and the audit row:

docker compose logs core-api gateway asset-processor | grep '"requestId":"<id>"'

The publish runner is not called by core-api: it pulls jobs. Its trail is the job id, which core-api's lines about the job (with the requestId of the request that queued it) and the job log share. Lines written outside a request (startup, the retention job, the object-event worker) have no requestId. To have the id in nginx's own access log, add $request_id (or $http_x_request_id) to the log_format of the host's http {} block.

6. Gaps#

The audit log covers successful administrative actions and failed sign-ins, codes, permissions and API clients (no Gateway administration, no tamper evidence beyond the database rule); no tracing, no metrics for asset-processor and gateway, readiness only for core-api (and only the database, not the object store), no log retention policy, no tamper protection of logs. Each is an item for the roadmap of a regulated deployment; the workarounds above cover the common questions.