Backups

A backup only counts if you’ve restored it. Every layer here has a written restore procedure, and the database restore checks its own result.

Overview

What	How	Where	When
Kubernetes volumes	Longhorn backup	NAS over NFS	Daily
PostgreSQL (physical)	CNPG barmanObjectStore, WAL + base backups	Garage S3	Continuous + daily
PostgreSQL (logical)	pg_dump CronJob	NAS over SSH	Daily
Garage S3	rclone sync	NAS	Daily
Gitea, Jenkins, Grafana, OpenBao	Ansible-managed backup scripts	NAS	Daily

Garage runs on Proxmox VMs and the NAS is a separate box. The two database layers deliberately use different storage systems, so no single failure takes out both.

PostgreSQL in three layers

The CloudNative-PG cluster has three instances on Longhorn volumes. Each layer covers a different failure:

1. Longhorn volume backup. Restores a whole volume at block level.
2. Barman to S3. Continuous WAL archiving gives point-in-time recovery, for when the cluster itself is lost.
3. Logical dumps. One portable .sql.gz per database plus the global roles. Use these to restore a single database, or to move it somewhere else entirely.

The dumps use --no-owner --no-acl to stay portable. The catch is that after a restore everything is owned by the superuser.

Restoring a single database

The restore Job is deliberately not listed in kustomization.yaml. It’s destructive and should only run when I ask for it, never because ArgoCD synced. It:

1. Pulls the chosen dump from the NAS
2. Drops and recreates the database, owned by the app role
3. Restores with ON_ERROR_STOP=1
4. Hands ownership of all tables, sequences and views back to the app role
5. Checks that the database has tables, that nothing is still owned by the superuser and that the database owner is right, then prints RESULT: PASS

Infrastructure recovery

Ansible has standalone recovery playbooks for Gitea, Jenkins, Grafana and OpenBao that restore from the NAS. They run without CI on purpose: if Gitea is gone, there is no CI to run them. See Lessons learned.