TL;DR

On a single server, the one with the most recent data:

1
2
3
4
5

systemctl stop k3s
k3s server --cluster-reset
# ... ends with:
# Managed etcd cluster membership has been reset, restart without --cluster-reset flag now
systemctl start k3s

On each of the other servers, with their /etc/rancher/k3s/config.yaml pointing at the one you reset (server: https://node1:6443):

1
2
3

systemctl stop k3s
rm -rf /var/lib/rancher/k3s/server/db
systemctl start k3s

Agents need nothing; they reconnect on their own.

When it is needed

etcd needs a majority: with three servers, two alive; with five, three. If you lose the majority (two dead nodes, a network partition, an IP change on several at once, an attempt to remove nodes by hand), the remaining one cannot write and the API freezes:

1
2

kubectl get nodes
Error from server: etcdserver: request timed out

In journalctl -u k3s you will see etcd cluster join failed, failed to get raft status, or waiting for etcd quorum. --cluster-reset is the way out k3s documents for exactly this.

What --cluster-reset does

It rewrites the etcd member list so the current node is the only one, keeping all of its data (the Kubernetes objects, secrets, everything). It does not touch pods or agents. The only thing lost is writes that did not get replicated to this node before the outage, which in practice are few or none if you pick the most up-to-date one.

With --cluster-reset-restore-path it does the same but replaces the data with a snapshot’s; everything after the snapshot disappears.

Step by step

  1. Pick the node. If several servers are still alive but without quorum, keep the one with the most recent snapshot or db: ls -la /var/lib/rancher/k3s/server/db/ and k3s etcd-snapshot ls.

  2. Keep a copy just in case, token included:

    1
2

    cp -a /var/lib/rancher/k3s/server/db /root/k3s-db-$(date +%F)
cp /var/lib/rancher/k3s/server/token /root/k3s-token

  3. Reset and start (the TL;DR block). The command takes a few seconds; do not interrupt it.

  4. Check that the API answers and the node is the only etcd member:

    1
2

    kubectl get nodes -o wide
kubectl get nodes -l node-role.kubernetes.io/etcd=true

    The other servers show NotReady until they join; agents go back to Ready on their own.

  5. Review the other servers’ configuration before starting them. The cluster’s first node usually has cluster-init: true and no server:; if you reset a different node, that first one would start a new cluster of its own. Make sure every node about to join has:

    1
2
3

    # /etc/rancher/k3s/config.yaml
server: https://node1:6443     # the reset node (or the VIP, if it points there)
token: <the same token>

  6. Join them one at a time: stop, delete server/db, start. Each one comes in as a learner and takes a minute or two to become a full member; wait for Ready before the next.

  7. Fresh snapshot once everything is green: k3s etcd-snapshot save --name post-reset.

Restoring from a snapshot

If the survivor’s db is corrupt or you want to go back to an earlier point:

1
2
3
4

k3s etcd-snapshot ls
systemctl stop k3s
k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-node1-1758760000
systemctl start k3s

Then the same rejoin of the other servers. By default k3s takes a snapshot every 12 hours (0 */12 * * *) and keeps 5, under /var/lib/rancher/k3s/server/db/snapshots; --etcd-s3 ships them to S3.

Traps

  • Never on two nodes. --cluster-reset on two servers gives two clusters with the same name and agents confused between them. One resets, the rest join.
  • The token rules. The snapshot carries the certificates and encryption keys protected with the cluster token. Restoring on a new node without the same token (--token) does not start. Keep /var/lib/rancher/k3s/server/token outside the cluster.
  • SQLite does not have this. A single server without etcd neither needs nor accepts --cluster-reset; its backup is copying /var/lib/rancher/k3s/server/db/.
  • An orphaned cluster-init on a node about to rejoin is the easiest way to end up with two clusters. Check step 5.
  • A load balancer or VIP in front of the servers: if it points at a dead node, the agents will not come back either; check it before blaming etcd.

Reference: etcd-snapshot and Backup and Restore in the k3s documentation.