Fix first, why after. Lab and production runbooks: Kubernetes, multi-cloud (AWS, GCP, Azure), Linux, networking, bare metal, and virtualization (KVM, Proxmox, Hyper-V, OpenStack).
A LUKS server that hangs and never comes back: watchdog, SSH unlock with dropbear, and netconsole
Three pieces so a hang costs a minute instead of hours: the watchdog reboots, dropbear in the initramfs lets you type the LUKS passphrase over SSH, and netconsole keeps the kernel’s last messages on another box.
OPNsense: port-forward unreachable from the same WAN subnet (reply-to)
OPNsense adds reply-to to WAN rules and sends replies to the gateway even when the client is on the same segment. Disable it and the firewall goes back to its routing table.
Windows 11 without a Microsoft account: oobe\bypassnro and its replacements
The classic oobe\bypassnro trick, how to revive it via registry after Microsoft removed it, and the modern start ms-cxh:localonly alternative.
acme-dns: DNS-01 without handing over the key to the whole zone (CNAME delegation of _acme-challenge) with certbot and cert-manager
NS auth.example.com → acme-dns; POST /register → username, password, fulldomain; CNAME _acme-challenge.example.com → . certbot certonly –manual –manual-auth-hook /etc/letsencrypt/acme-dns-auth.py –preferred-challenges dns -d ‘*.example.com’ -d example.com. On Kubernetes, the dns01.acmeDNS solver with accountSecretRef. acme-dns 2.0.2 (February 2025) is still active.
badblocks on large disks: 'Value too large for defined data type (must be 32-bit value)'
badblocks -b 4096 -sv /dev/sdX. The block counter is 32-bit and with 1 KiB blocks it stops at 4 TiB; with 4 KiB it covers 16 TiB. -w wipes the disk; for a quick check, smartctl -t long.
Consistent Prometheus backups: snapshot through the admin API (enableAdminAPI on the operator), copy it out, restore it and delete series
kubectl patch prometheus … enableAdminAPI true; POST http://localhost:9090/api/v1/admin/tsdb/snapshot; kubectl exec … tar cf - -C /prometheus/snapshots NAME | tar xf -. Restore = copy the blocks into the root of the new instance’s data dir. The API has no authentication: port-forward and turn it off when done.
DHCP option 121: classless static routes (RFC 3442), how to encode them in hex and set them on ISC dhcpd, Kea, pfSense and MikroTik
{prefix}{significant destination octets}{router} per route, in hex; 0.0.0.0/0 has zero destination octets. Always include the default route in 121, because the client discards the option 3 router when it receives 121. Windows asks for it as 249 besides 121; Kea 2.4+ writes it as ‘192.0.2.0/24 - 192.168.1.254’; MikroTik as 0x… with code=121.
GCP Cloud VPN will not come up: reading the IKE logs (peer not responding, traffic selectors narrowed) and BGP with pfSense
Logs Explorer with resource.type=vpn_gateway and the IKE status text. ‘peer not responding’ = IKE version, PSK, IP or UDP 500/4500; ’traffic selectors narrowed’ = phase 2 subnets that do not match. With pfSense and BGP: permit route maps in/out or no routes are accepted.
GitHub Actions → Google Cloud without keys: Workload Identity Federation with gcloud and Terraform
Pool + OIDC provider with issuer token.actions.githubusercontent.com and an attribute-condition on repository_owner; roles/iam.workloadIdentityUser to the repo’s principalSet; auth@v3 with workload_identity_provider and service_account. No secrets to rotate.
Google-managed certificate that never gets issued: CAA (pki.goog, letsencrypt.org), FAILED_NOT_VISIBLE and the right order
CAA with issue pki.goog and issue letsencrypt.org (or no CAA at all), A/AAAA pointing only to the load balancer IP, certificate attached to the target proxy, and up to 60 minutes. gcloud compute ssl-certificates describe shows the domainStatus that tells which of the three is failing.