Expired k3s certificates after 396 days of uptime

My k3s cluster had been running for 396 days without a single restart. I was pretty happy with that number until I noticed kubectl stopped working completely.

You would think that something like this would be easily noticed or picked up. A combination of busy life stuff and not having any robust alerting running for my homelab (as I don’t want to handle this as my second job) made me not notice.

All my workloads kept running fine, my dashboards kept updating. The only thing that was dead was my ability to talk to the API.

The issue

Every command gave me the same thing:

error: You must be logged in to the server (the server has asked for the client to provide credentials)

This was a 401, it told me exactly where to look:

• A timeout means the node or the VIP is down. It’s a network or availability problem.
• A 401 means the API server is perfectly healthy and answering you. It just doesn’t accept your certificate anymore.

So the cluster was fine. My client certificate was the problem. The cert had expired on 01 Jul 2026 and I only noticed it on 02 Aug 2026.

Why this happened

k3s creates its own CA at install time and signs everything from there.

The important bit is the lifetime:

• The CA is valid for 10 years.
• The client certificates are valid for 1 year.

The admin certificate inside /etc/rancher/k3s/k3s.yaml is a client cert, and that is the one kubectl uses.

k3s does rotate these certificates automatically, but it only does it on server startup, and only when the certificate is already within 90 days of expiring.

So the rotation logic is basically:

1. k3s server starts
2. It checks the certs
3. If a cert expires within 90 days, it gets renewed

If your cluster never restarts, step 1 never happens. There is no background timer that renews things while the process keeps running. My cluster was stable for 396 days, which means it comfortably ran past the 90 day window, past the expiry date, and just kept going.

Uptime is normally the thing you want. Here it was the exact reason my cluster broke.

The fix

The fix itself is actually very easy:

On any master:

sudo systemctl restart k3s

That’s it.

On startup k3s sees the expired cert, is now well inside the 90 day window, and mints a new one.

A few things worth knowing before you do this:

• Your workloads are not affected
• You only need to restart one master
• Do not restart all three masters at once

Refreshing the local kubeconfig

With a fresh cert on the node, I still had to get it onto my workstation. My ~/.kube/config also holds work contexts, so I merge instead of overwriting.

Rewrite the server address

k3s ships its kubeconfig pointing at https://127.0.0.1:6443, which is correct on the node and useless from your laptop.

The obvious fix is to replace it with the IP of the master you copied it from, but that’s not what you want either. Then your kubeconfig depends on that one specific master being alive. Point it at the kube-vip VIP instead (192.168.129.230 in my case), so it survives a master going down.

Catching it next time

Rebooting nodes for kernel updates like I do for my other VM’s would be the easy fix. I will have to workout a schedule on when to do this as I don’t want to restart nodes too fast sequentially.

Lessons learned

1. Uptime is not holy
2. Read the error
3. Silent failures need monitoring too

The actual fix was one systemctl restart. Everything else was figuring out why a cluster that had done nothing wrong for over a year suddenly locked me out.