Backup

Consistent Prometheus backups: snapshot through the admin API (enableAdminAPI on the operator), copy it out, restore it and delete series

kubectl patch prometheus … enableAdminAPI true; POST http://localhost:9090/api/v1/admin/tsdb/snapshot; kubectl exec … tar cf - -C /prometheus/snapshots NAME | tar xf -. Restore = copy the blocks into the root of the new instance’s data dir. The API has no authentication: port-forward and turn it off when done.

rsync: resume huge transfers, copy only the directory tree, and decide what counts as 'changed'

rsync -av –partial –append-verify –inplace source/ host:/dest/ picks up where it stopped. -f’+ */’ -f’- *’ copies only folders. By default rsync compares size and date; –checksum reads everything; –ignore-times copies everything.

ZFS: resume an interrupted zfs send | zfs recv (receive_resume_token)

Always receive with zfs recv -s. If it drops: zfs get receive_resume_token target on the receiver and zfs send -t on the sender. To give up, zfs recv -A target deletes the partial state.

k3s: recover an HA cluster with embedded etcd that lost quorum (--cluster-reset)

On the surviving server: systemctl stop k3s; k3s server –cluster-reset; systemctl start k3s. On the others: stop, rm -rf /var/lib/rancher/k3s/server/db, start pointing at the reset node. Never on two nodes.

Send ZFS snapshots to another host (and on to S3)

A zfs send stream is not a tarball. Another pool first, S3 object if you need it.