Troubleshoot a Helm deployment

Norn pods wait for the current revision's migration Job. When API, web, and worker all wait in init containers, diagnose that Job rather than treating each pod as a separate failure.

Migration failures

Find the newest migration Job:

kubectl get jobs --namespace norn \
  -l app.kubernetes.io/instance=norn,app.kubernetes.io/component=migration \
  --sort-by=.metadata.creationTimestamp

Its containers identify the failing dependency:

kubectl logs job/<job> --namespace norn --container wait-for-postgresql
kubectl logs job/<job> --namespace norn --container wait-for-valkey
kubectl logs job/<job> --namespace norn --container migrate
kubectl logs job/<job> --namespace norn --container seed

Container	Check
wait-for-postgresql	DSN, authentication, TLS, network reachability, and database availability
wait-for-valkey	address, ACL username/password, network reachability, and Valkey availability
migrate	database permissions, PostgreSQL compatibility, connection limits, and migration error output
seed	database access and authorisation policy reconciliation

After fixing the dependency, repeat the Helm upgrade. The new revision creates a new Job. Do not run Goose manually; migrations are embedded in the Norn image.

Pods remain in migration init containers

wait-for-migration-job verifies that the revision's Job exists. wait-for-migrations waits for it to complete. If the Job is healthy but these containers report forbidden, check the release ServiceAccount, Role, and RoleBinding. It needs get, list, and watch access to Jobs in the Norn namespace.

Norn opens but sign-in or forms return 403

The browser URL must exactly match Norn's configured public origin. With the chart Ingress, that origin comes from ingress.host and ingress.tls.enabled; otherwise it is norn.baseUrl.

Confirm that proxies preserve the original host and scheme. If client-address forwarding is enabled, NORN_HTTP_CLIENT_IP_HEADER and NORN_HTTP_TRUSTED_PROXIES must be set together. The trusted CIDRs must contain the actual proxies and no untrusted source.

Realtime updates disconnect

/v1/workspaces/<workspace>/events is a Server-Sent Events stream. Every proxy in the path must:

• disable buffering and compression for that route
• avoid a finite response read timeout
• stream the response instead of waiting for it to finish

Reconnects at a consistent interval usually identify a proxy timeout.

Attachments fail

Attachment creation starts through the API, but the browser transfers bytes directly to object storage. Check the failing request in the browser network panel.

The storage endpoint must be browser-reachable with a valid certificate. Its CORS policy must allow the exact Norn origin to use GET, HEAD, and PUT, allow content-type, and expose ETag. Credentials need access to the configured bucket, and the endpoint, region, bucket, and path-style setting must match the provider.

For bundled Garage, confirm that the revision's CORS Job completed:

kubectl get jobs --namespace norn \
  -l app.kubernetes.io/component=garage-cors
kubectl logs job/<garage-cors-job> --namespace norn

Changing the Norn hostname also changes the required CORS origin.

Email is not delivered

Check API and worker logs for SMTP errors. Verify the provider's host, port, sender verification, authentication type, and TLS policy. Norn supports authentication types none, plain, login, and cram-md5; TLS policies are none, opportunistic, and mandatory.

Test invitations and password recovery before disabling password authentication or opening the instance to users.

An upgrade times out

Check the migration Job first, then API, web, and worker readiness:

helm status norn --namespace norn
kubectl get pods,jobs --namespace norn

If a legitimate migration needs more time, increase both migration.activeDeadlineSeconds and the Helm --timeout. Before rollback, determine whether migrations completed; Helm does not reverse the database.

The Helm test fails

helm test norn --namespace norn --logs

The test reaches the API health endpoint and web service inside the cluster. It does not test public DNS, TLS, Ingress, email, or object uploads.

Share diagnostics safely

Include the chart, Kubernetes, Ingress controller, and storage driver versions; enabled bundled or external services; relevant events; and redacted logs. Never share Secret data, DSNs, credentials, cookies, licence keys, private hostnames, or rendered Secret manifests.