Kubernetes

helmfile in production: three-way diff, 'already set in map', OCI 403, duplicate dependencies and adopting resources that were already deployed

Three variables (HELM_DIFF_THREE_WAY_MERGE, HELM_DIFF_NORMALIZE_MANIFESTS, HELMFILE_GOCCY_GOYAML), two flags (–skip-diff-on-install, -l name=) and one rule: pin helm, helmfile and helm-diff in CI. Adopting existing resources = the managed-by=Helm label plus the meta.helm.sh/release-name and release-namespace annotations.

Removing Rancher from a cluster: cattle-system stuck Terminating, clusters that will not delete, the webhook that blocks everything, and getting the admin back

To wipe everything: the rancher-cleanup job and its verify job. For surgery: remove metadata finalizers (patch) and do not force the namespace spec.finalizers without first checking what resources remain; patch finalizers on clusters.provisioning and clusters.management; export and delete the rancher.cattle.io webhooks until the pod is back; reset-password inside the Rancher pod.

Vault Agent Injector on a private GKE cluster: no sidecar and pods take 30 s (the control plane cannot reach the webhook on 8080)

gcloud compute firewall-rules create … –source-ranges= –target-tags= –rules=tcp:8080. Validate by deleting a pod and grepping the injector for ‘handler: Request received: Method=POST URL=/mutate’. Applies to any webhook not listening on 443 or 10250.

cert-manager: the HTTP-01 challenge returns 308 (missing ingressClassName in the solver) and other self-check failures

ingressClassName: nginx inside solvers.http01.ingress and delete the stuck Challenge. If the self-check cannot reach the public IP from inside the cluster, it is NAT reflection or split DNS, not cert-manager.

k3s: recover an HA cluster with embedded etcd that lost quorum (--cluster-reset)

On the surviving server: systemctl stop k3s; k3s server –cluster-reset; systemctl start k3s. On the others: stop, rm -rf /var/lib/rancher/k3s/server/db, start pointing at the reset node. Never on two nodes.

k3s: use images without a registry (docker save → k3s ctr images import) and clean up the leftovers

docker save -o app.tar app:v1; k3s ctr images import app.tar on every node; imagePullPolicy: Never. For several nodes, /var/lib/rancher/k3s/agent/images/ or –embedded-registry. Cleanup: k3s crictl rmi –prune.

Move k3s data to another disk

Root disk is full. Stop k3s, move the directories, leave a symlink, start again.