Skip to main content

Operate a Helm deployment

Norn's recoverable state spans PostgreSQL, object storage, and the encryption key. Operate them as one system.

Backups​

A complete backup contains:

  • a consistent PostgreSQL backup
  • every object in the configured bucket
  • NORN_SECURITY_ENCRYPTION_KEY and service credentials
  • the chart version and non-secret values file
  • Valkey data when retaining sessions and queued work is part of the recovery objective

PostgreSQL contains attachment metadata while object storage contains the bytes. Align their backup times and test restoring both together. Use the managed service's backup facilities for external services.

For bundled PostgreSQL, use PostgreSQL backup tooling rather than copying a live volume:

kubectl exec --namespace norn norn-norn-postgresql-0 -- \
  pg_dump --username norn --dbname norn --format=custom > norn.dump

Back up bundled Garage through its S3 API or with a snapshot procedure supported by the storage provider. A completed snapshot is not a restore test.

The retained Kubernetes Secret is not a backup. Export it only to an encrypted secret store with restricted access.

Upgrade​

Read the Norn release notes and take a backup first:

helm upgrade norn oci://ghcr.io/usenorn/charts/norn \
  --version <target-version> \
  --namespace norn \
  --values values.yaml \
  --wait \
  --timeout 15m

Each revision creates a migration Job. It waits for PostgreSQL and Valkey, applies embedded database migrations, and reconciles the authorisation policy. API, web, and worker pods wait for that Job. The worker uses a Recreate strategy so old and new workers never overlap.

Verify the result:

helm status norn --namespace norn
helm test norn --namespace norn --logs

Then test sign-in, an issue mutation, an attachment upload and download, and an operation handled by the worker.

Add --atomic with Helm 3 or --rollback-on-failure with Helm 4 if Kubernetes resources should roll back automatically. Database migrations are not reversed by Helm.

warning

Changing postgresql.image.tag across PostgreSQL major versions does not upgrade its data directory. Use PostgreSQL's supported major-upgrade procedure or migrate to a new external database.

Rollback​

Before selecting a previous Helm revision, determine whether the failed revision completed its migration Job. The previous application must be compatible with the current database schema.

helm history norn --namespace norn
helm rollback norn <revision> \
  --namespace norn \
  --wait \
  --timeout 15m

If schema compatibility is unknown, restore the pre-upgrade backup into an isolated environment and validate there instead of repeatedly changing production.

Restore​

Restore to a separate hostname and namespace first:

  1. recreate the Secret with the original encryption key and service credentials
  2. restore PostgreSQL and the object bucket
  3. restore Valkey if it is part of the recovery plan
  4. install the same chart version and service topology with norn.existingSecret
  5. allow the migration and seed Job to finish
  6. run the Helm test and application checks
  7. move traffic only after validation

Use isolated copies of data services. This prevents recovery tests from sending production email, webhooks, or object-storage requests.

Uninstall​

helm uninstall norn --namespace norn

The chart-managed Secret and StatefulSet PVCs are retained. This protects data from accidental uninstall and means uninstall is not data deletion. Remove retained resources only after a final, verified backup and an explicit data-destruction decision.