Lessons learned

Every homelab breaks. These are the things that went wrong in mine and what I changed afterwards.

The chicken and the egg

All automation runs through Gitea Actions, but Gitea is part of the homelab too. If Gitea dies, nothing can deploy, including a fix for Gitea.

Fix: treat Gitea as a bootstrap component. It has a standalone Ansible recovery playbook that restores it from the NAS without any CI. Once Gitea and its runners are back, everything else rebuilds itself from there. The same applies to OpenBao, because CI needs secrets.

A power cut

In May 2025 the whole street lost power, and I hadn’t shut anything down. Several VMs came back with kernel panics.

Fix: the k3s nodes were already managed by OpenTofu, so recovery meant deleting the broken VMs, rerunning the pipeline and rejoining them to the cluster. That was the point at which provisioning everything from code paid for itself.

396 days of uptime

kubectl suddenly returned 401s. k3s client certificates last one year, and k3s only renews them when the server starts. The cluster had been up for 396 days, so it never restarted and never renewed them.

Fix: restart k3s on one control plane node at a time, then refresh the kubeconfig. The next rotation is planned inside the 90-day renewal window. Something to remember: a 401 means the API is fine and your certificate is dead, while a timeout means the API is down. Full write-up.

Removing a role doesn’t uninstall it

When I moved to fluent-bit I removed promtail from the playbook, and promtail kept running on four hosts for another eight months. Ansible only does what’s in the playbook. Taking a role out of the playbook doesn’t uninstall anything.

Fix: every role removal now comes with a removal playbook. The same thing happens in ArgoCD: core component Applications have no finalizer, so deleting one leaves its resources running.

Host-specific changes don’t deploy

The Ansible workflows trigger on role and group variable changes, but not on host_vars. Those are rare, so for now I run them by hand. Building a dynamic job matrix for them cost more time than the problem was worth.

Changed SSH host keys

After fixing the certificates I got a host key warning for the same node. Running ssh-keygen -R and moving on is almost always the right call, and that’s exactly why I don’t do it blindly. I checked the fingerprint from the Proxmox console first. It turned out to be a package upgrade that regenerated the key.