Kubernetes

helmfile in production: three-way diff, 'already set in map', OCI 403, duplicate dependencies and adopting resources that were already deployed

Three variables (HELM_DIFF_THREE_WAY_MERGE, HELM_DIFF_NORMALIZE_MANIFESTS, HELMFILE_GOCCY_GOYAML), two flags (–skip-diff-on-install, -l name=) and one rule: pin helm, helmfile and helm-diff in CI. Adopting existing resources = the managed-by=Helm label plus the meta.helm.sh/release-name and release-namespace annotations.

Removing Rancher from a cluster: cattle-system stuck Terminating, clusters that will not delete, the webhook that blocks everything, and getting the admin back

To wipe everything: the rancher-cleanup job and its verify job. For surgery: remove metadata finalizers (patch) and do not force the namespace spec.finalizers without first checking what resources remain; patch finalizers on clusters.provisioning and clusters.management; export and delete the rancher.cattle.io webhooks until the pod is back; reset-password inside the Rancher pod.

Vault Agent Injector on a private GKE cluster: no sidecar and pods take 30 s (the control plane cannot reach the webhook on 8080)

gcloud compute firewall-rules create … –source-ranges= –target-tags= –rules=tcp:8080. Validate by deleting a pod and grepping the injector for ‘handler: Request received: Method=POST URL=/mutate’. Applies to any webhook not listening on 443 or 10250.

cert-manager: the HTTP-01 challenge returns 308 (missing ingressClassName in the solver) and other self-check failures

ingressClassName: nginx inside solvers.http01.ingress and delete the stuck Challenge. If the self-check cannot reach the public IP from inside the cluster, it is NAT reflection or split DNS, not cert-manager.