Posts

Runbooks newest first. By topic: Topics.

rsync: resume huge transfers, copy only the directory tree, and decide what counts as 'changed'

rsync -av –partial –append-verify –inplace source/ host:/dest/ picks up where it stopped. -f’+ */’ -f’- *’ copies only folders. By default rsync compares size and date; –checksum reads everything; –ignore-times copies everything.

Vault Agent Injector on a private GKE cluster: no sidecar and pods take 30 s (the control plane cannot reach the webhook on 8080)

gcloud compute firewall-rules create … –source-ranges= –target-tags= –rules=tcp:8080. Validate by deleting a pod and grepping the injector for ‘handler: Request received: Method=POST URL=/mutate’. Applies to any webhook not listening on 443 or 10250.

ZFS: resume an interrupted zfs send | zfs recv (receive_resume_token)

Always receive with zfs recv -s. If it drops: zfs get receive_resume_token target on the receiver and zfs send -t on the sender. To give up, zfs recv -A target deletes the partial state.

2048-bit DKIM: the key does not fit in a 255-character TXT

The 255 limit is per string, not per record. fold -w 255 + quotes, all in the same TXT; the verifier concatenates them. Route 53 and Cloud DNS require it; Cloudflare does it for you.

Ceph on Proxmox: clearing the 'daemons have recently crashed' HEALTH_WARN and deleting orphaned VM disks in RBD

ceph crash info to learn what fell over, ceph crash archive-all to clear the warning. For orphaned disks: compare rbd ls with the .conf files of every node and delete with pvesm free or rbd rm.

Ceph on Proxmox: device classes and CRUSH rules to keep SAS, SSD and NVMe apart

rm-device-class + set-device-class on the OSD, crush rule create-replicated default host , and the pool onto that rule. No ‘class create’: the class is born with the first OSD that uses it.

cert-manager: the HTTP-01 challenge returns 308 (missing ingressClassName in the solver) and other self-check failures

ingressClassName: nginx inside solvers.http01.ingress and delete the stuck Challenge. If the self-check cannot reach the public IP from inside the cluster, it is NAT reflection or split DNS, not cert-manager.

FreeBSD: replace a disk in the ZFS boot mirror (zroot) with gpart

gpart backup ada0 | gpart restore -F ada3, BIOS bootcode on the freebsd-boot partition, loader.efi on the ESP, and zpool replace zroot ada3p4. Never the whole disk: you would lose the boot partitions.

iLO 2/3 and ERR_SSL_VERSION_OR_CIPHER_MISMATCH: opening the web UI from a modern browser

iLO 2 is stuck on TLS 1.0 and iLO 3 on TLS 1.1 forever. Firefox with deprecated TLS (separate profile) or Edge in IE mode opens the console; over SSH you can enforce AES and regenerate the certificate.

k3s: recover an HA cluster with embedded etcd that lost quorum (--cluster-reset)

On the surviving server: systemctl stop k3s; k3s server –cluster-reset; systemctl start k3s. On the others: stop, rm -rf /var/lib/rancher/k3s/server/db, start pointing at the reset node. Never on two nodes.