Secrets

Last updated November 3, 2026

Secrets live in OpenBao, the community fork of HashiCorp Vault. I migrated from a single Vault node to an OpenBao cluster in 2026, full write-up here.

Cluster

  • Three VMs, each on a different Proxmox node, so losing one hypervisor doesn’t break quorum.
  • Integrated Raft storage, no Consul or external database.
  • The Ansible role renders a retry_join block for every other node in the inventory group. Deploys run with serial: 1, so a rolling update never drops quorum.
  • Traefik spreads the UI and API over all three nodes with a retry middleware. Followers forward writes to the leader, so any node can take a request.

Unsealing

A small script runs from cron on every node and unseals it when needed. That way a node that restarts after a power cut or a reboot rejoins without anyone touching it.

The trade-off is that the unseal keys sit on the nodes. For a homelab I prefer a cluster that heals itself over one that waits for me to type keys. In a company setup this would be a cloud KMS or a transit seal instead.

Kubernetes integration

The Vault Secrets Operator syncs KV secrets into native Kubernetes Secret objects:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  name: vss-n8n
spec:
  vaultAuthRef: vault-auth
  mount: homelab-kube
  type: kv-v2
  path: n8n
  refreshAfter: 10s
  destination:
    create: true
    name: prod-n8n-secret

Authentication uses the Kubernetes auth method with a service account per namespace. Every app that needs a secret has one vaultsecret.yaml in its prod overlay, and Reloader restarts the pods when the value changes.

Outside Kubernetes

Ansible variables are encrypted with Ansible Vault. CI gets the vault password from a Gitea Actions secret.

Recovery

One node takes a daily Raft snapshot to the NAS. A recovery playbook rebuilds the cluster from scratch: it does a throwaway init, restores the snapshot on one node, lets Raft replicate it to the others and puts the unseal key back on all three. It runs without CI, because CI itself depends on secrets.