Migrating Vault to a 3-node OpenBao cluster

Back in 2024 I set up a Vault server on a single VM and hooked it into my k3s cluster with the Vault Secrets Operator. It worked fine for two years, but it had two “problems” I kept ignoring.

The first is that it was a single VM. Every secret in my homelab lived on one machine, and half my automation stops working the moment that machine is gone.

The second is the license change. HashiCorp moved Vault to the BUSL license, and OpenBao is the fork that came out of that, now maintained under the Linux Foundation.

So I moved to OpenBao and rebuilt it as a 3-node cluster while I was at it.

The migration itself

This is the good news. OpenBao is API compatible with Vault, so almost nothing on the consumer side had to change.

• The CLI is bao instead of vault
• The HTTP API is the same, so the Vault Secrets Operator kept working untouched
• My secret engine mount (homelab-kube), policies and the vso auth method all carried over

Everything from my old post still applies. The VaultStaticSecret resources in my app overlays did not need a single edit.

The new setup

Three nodes, one on each Proxmox host, so losing a hypervisor does not take the cluster down:

The VMs are provisioned by Terraform (just three more entries in my node_info map) and configured by an Ansible openbao role. Storage is integrated Raft, so there is no external Consul or database to run alongside it.

Quorum needs 2 of the 3 nodes up and unsealed. Losing one node is fine. Losing two stops writes.

Building the Raft config from the inventory

The config template loops over the openbao inventory group and adds a retry_join block for every node that is not itself:

storage "raft" {
  path    = "/opt/openbao/data"
  node_id = "{{ openbao_raft_id }}"

{% for host in groups['openbao'] if host != inventory_hostname %}
  retry_join {
    leader_api_addr     = "https://{{ hostvars[host].openbao_listen_ip }}:8200"
    leader_ca_cert_file = "{{ openbao_tls_cert }}"
  }
{% endfor %}
}

The two variables come from inventory/group_vars/openbao.yaml:

openbao_raft_id: "{{ inventory_hostname }}"
openbao_listen_ip: "{{ ansible_host }}"

All three nodes share the same self-signed TLS certificate, which is also reused as the leader_ca_cert_file. That way a node can validate the leader it is joining without me having to run a proper internal CA for three machines.

Adding a fourth node would mean adding it to the group and rerunning the playbook. Nothing else.

Rolling deploys without losing quorum

Since a deploy restarts the service, doing all three at once would drop the cluster. The play in site.yaml runs with serial: 1:

- name: Deploy openbao
  hosts: openbao
  serial: 1
  roles:
    - openbao

One node restarts, rejoins, and only then does the next one go down. Quorum is never lower than 2 out of 3.

Auto-unseal

OpenBao starts sealed after every restart, which is correct behaviour and also very annoying when a node reboots at 3am.

I handle it with a small script that Ansible installs as /usr/local/bin/openbao-unseal. It reads the unseal keys from /etc/openbao-unseal.key and posts them to the local unseal endpoint. Cron runs it every 5 minutes on each node:

*/5 * * * * /usr/local/bin/openbao-unseal

To be clear about the trade-off: this puts the unseal keys on the same machine that they unseal. Not perfect, but it works. For a homelab where the alternative is manually unsealing three nodes after every reboot, I am fine with it.

The one manual step

This is the bit that will catch you out. The key file is not Ansible-managed when you bootstrap a brand new cluster.

After bao operator init, you get one set of Shamir keys that all three nodes share. You have to copy that key file to the other two nodes by hand, once. A node that is missing it will come up sealed after every single restart and stay that way.

My backup and recovery scripts bundle the key file and restore it automatically, so disaster recovery never needs this step repeated.

Backups and recovery

Only openbao-01 runs the backup timer. Raft keeps all three nodes identical, so backing up all of them would just be three copies of the same thing.

Recovery is a playbook that pulls the backup from my NAS, does a throwaway init and unseal, restores the snapshot against one node, and puts the unseal key file back on all three. Raft replicates the restored data to the others once they rejoin.

The manual version of that, if you ever need to do it by hand:

bao operator init -tls-skip-verify
export BAO_TOKEN=<root token>
bao operator unseal -tls-skip-verify
bao operator raft snapshot restore -tls-skip-verify -force <snapshot>

Use the CLI for this, not the Web UI. Opening the UI on a fresh instance also initialises it, and you will never see the initial keys.

Lessons learned

1. Templating cluster config from the inventory beats maintaining a peer list
2. serial: 1 should be the default for anything with a quorum (ansible)
3. Write the recovery playbook before you need it + test it

The migration was the easy half. Going from one VM to three nodes with proper quorum was the needed part.