1
2
3
4
5
6

export HELM_DIFF_THREE_WAY_MERGE=true        # diff against what is in the cluster, not against the last release
export HELM_DIFF_NORMALIZE_MANIFESTS=true    # strip formatting noise from the diff
export HELMFILE_GOCCY_GOYAML=true            # helmfile 0.x: avoids "key already set in map"

helmfile -e prod -l name=rabbitmq diff        # a single release
helmfile -e prod apply --skip-diff-on-install # giant first install without a diff

And in CI, pinned versions of all three pieces (helm, helmfile, helm-diff): half of helmfile’s odd errors are combinations nobody tested together.

The errors and what fixes them

Adopting resources that already exist

The typical case: someone installed ArgoCD (or whatever) with kubectl apply -f, and now you want a helmfile release to manage it. Helm refuses to install over resources that are not its own (rendered manifests contain a resource that already exists … invalid ownership metadata). Since Helm 3.2 it is enough to mark them as Helm’s before installing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

NS=argocd; REL=argocd
for kind in deploy sts svc cm secret sa ingress role rolebinding networkpolicy; do
  kubectl -n $NS get $kind -o name 2>/dev/null | while read -r r; do
    kubectl -n $NS label "$r" app.kubernetes.io/managed-by=Helm --overwrite
    kubectl -n $NS annotate "$r" meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=$NS --overwrite
  done
done
# cluster-scoped resources: same labels, no -n
for r in $(kubectl get crd,clusterrole,clusterrolebinding -o name | grep -i argo); do
  kubectl label "$r" app.kubernetes.io/managed-by=Helm --overwrite
  kubectl annotate "$r" meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=$NS --overwrite
done
helmfile -e prod -l name=$REL diff     # should be nearly empty
helmfile -e prod -l name=$REL apply

Only mark what the chart is going to render: a labeled resource the chart does not include is harmless, but one the chart renders under another name ends up duplicated.

helmfile 1.x and Helm 4

Coming from 0.x, this is what changes when you upgrade:

• A helmfile.yaml with Go templates must be called helmfile.yaml.gotmpl; a plain .yaml is no longer rendered.
• environments: and releases: go in separate parts split by --- within the same file.
• helmfile delete is gone (it is destroy), so is helmfile charts, and --args is removed.
• HELMFILE_SKIP_INSECURE_TEMPLATE_FUNCTIONS becomes HELMFILE_DISABLE_INSECURE_FEATURES.
• helmfile 1.x works with Helm 3 and Helm 4. With Helm 4, install helm-diff with helm plugin install https://github.com/databus23/helm-diff --verify=false (Helm 4 verifies plugin provenance and helm-diff does not publish it).

Traps

• apply with no prior diff in CI: helmfile apply already diffs and only applies when there are changes; sync always applies. In pipelines, diff on the PR and apply on main.
• installed: false uninstalls the release on the next apply; handy to retire things through Git, dangerous if copied from another environment.
• Secrets in values: helmfile leaves them in the release manifest (Helm’s Secret in the namespace); use helm-secrets with SOPS or external references.
• --skip-diff-on-install is not --skip-diff: it only skips the diff for releases that do not exist yet. Upgrades still show their diff.
• Three binaries, three versions: helm version, helmfile version and helm plugin list as the first pipeline step, pinned by tag in the CI image.

References: helm-diff, helmfile and the towards 1.0 proposal.

See also

• k3s: import images without a registry — when the release depends on an image that lives in no registry.

To remove Rancher completely from a cluster where it was installed (irreversible, backup taken):

1
2
3
4

kubectl create -f https://raw.githubusercontent.com/rancher/rancher-cleanup/main/deploy/rancher-cleanup.yaml
kubectl -n kube-system logs -l job-name=cleanup-job -f
kubectl create -f https://raw.githubusercontent.com/rancher/rancher-cleanup/main/deploy/verify.yaml
kubectl -n kube-system logs -l job-name=verify-job -f      # should print nothing

To fix a single thing without razing everything, read on.

Namespace stuck in Terminating

cattle-system, cattle-fleet-system, cattle-impersonation-system, local, the cluster-fleet-* ones, or any namespace Rancher created, stay in Terminating forever:

1

kubectl get ns cattle-system -o jsonpath='{.metadata.finalizers}{"\n"}{.spec.finalizers}{"\n"}{.status.conditions}{"\n"}'

Two different cases:

1. metadata.finalizers contains controller.cattle.io/namespace-auth (or any *.cattle.io). Rancher put it there and only Rancher removes it; if Rancher is gone, nobody will. Remove it yourself:

   1	kubectl patch ns cattle-system --type=merge -p '{"metadata":{"finalizers":[]}}'

2. spec.finalizers contains kubernetes and status.conditions says SomeResourcesRemain or DiscoveryFailed. The API server will not delete the namespace because resources remain inside, almost always CRs of an API group whose service disappeared (a Rancher apiservice). Before forcing, look at what it is:

   1 2 3

   kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n1 kubectl -n cattle-system get --ignore-not-found --show-kind 2>/dev/null kubectl get apiservice | grep -v True # dead APIs that break discovery

   Delete the dead apiservice or the CRs with finalizers, and the namespace goes away on its own. Forcing spec.finalizers to empty is the last resort and leaves orphans behind:

   1 2	kubectl get ns cattle-system -o json | jq '.spec.finalizers=[]' \ | kubectl replace --raw /api/v1/namespaces/cattle-system/finalize -f -

Cluster that cannot be deleted

An imported cluster that no longer exists stays in Removing in the UI, or kubectl delete never finishes. There are two objects with finalizers, one per API:

1
2
3
4
5

kubectl -n fleet-default get clusters.provisioning.cattle.io
kubectl -n fleet-default patch clusters.provisioning.cattle.io my-cluster --type=merge -p '{"metadata":{"finalizers":[]}}'

kubectl get clusters.management.cattle.io          # c-m-xxxxxxxx (v2.6+) or c-xxxxx
kubectl patch clusters.management.cattle.io c-m-xxxxxxxx --type=merge -p '{"metadata":{"finalizers":[]}}'

Then clean up the cluster-fleet-default-<name>-<hash> namespace and whatever clusters.fleet.cattle.io remain in fleet-default.

The webhook that rejects everything

rancher-webhook validates and mutates *.cattle.io resources (and some core ones, such as namespaces and ServiceAccounts on managed clusters). If the pod is gone or broken, every change fails with:

1

Error from server (InternalError): Internal error occurred: failed calling webhook "rancher.cattle.io": … connect: connection refused

Export the configurations, delete them, let the pod come back healthy, and restore them:

1
2
3
4
5
6

kubectl get validatingwebhookconfiguration rancher.cattle.io -o yaml > vwc-rancher.yaml
kubectl get mutatingwebhookconfiguration rancher.cattle.io -o yaml > mwc-rancher.yaml
kubectl delete validatingwebhookconfiguration rancher.cattle.io
kubectl delete mutatingwebhookconfiguration rancher.cattle.io
# … fix rancher-webhook (kubectl -n cattle-system rollout restart deploy/rancher-webhook) …
kubectl apply -f vwc-rancher.yaml -f mwc-rancher.yaml

If you are uninstalling, do not restore them: they are the first thing to remove so the rest of the deletion can proceed.

Getting access back

Lost admin password, or the admin user disabled by mistake:

1
2
3

kubectl -n cattle-system exec -it deploy/rancher -- reset-password
# New password for default administrator (user-xxxxx): …
kubectl -n cattle-system exec -it deploy/rancher -- ensure-default-admin

ensure-default-admin re-enables the user and gives the administrator role back; reset-password only changes the password.

Traps

• rancher-cleanup deletes the downstream cluster objects (the clusters keep running, but are no longer registered) and everything Rancher installed: monitoring, logging, Gatekeeper. Longhorn stays. Do not run it on the Rancher cluster you want to keep.
• Backup first: kubectl get -A -o yaml of the *.cattle.io resources you care about, or Rancher’s backup-operator. Finalizers go away in a second; what was behind them does not come back.
• kubectl delete ns --force --grace-period=0 does nothing for a namespace in Terminating: the block is in the finalizers, not in the grace period.
• Rancher puts the finalizers back while it is alive: if you are cleaning a cluster still connected to a Rancher, disconnect it first (delete the cattle-cluster-agent) or your patches undo themselves.
• The *.cattle.io CRDs survive a Helm uninstall; rancher-cleanup removes them, a manual uninstall does not.

See also

• k3s: cluster reset after losing quorum — the other cluster surgery worth keeping at hand.
• rancher-cleanup and the Rancher technical FAQ.

1
2
3
4
5
6
7
8
9

CLUSTER=my-cluster; REGION=us-central1; NETWORK=my-vpc
MASTER_CIDR=$(gcloud container clusters describe $CLUSTER --region $REGION \
  --format='value(privateClusterConfig.masterIpv4CidrBlock)')
NODE_TAG=$(gcloud compute firewall-rules list --filter="name~^gke-$CLUSTER" \
  --format='value(targetTags.list())' | head -1)

gcloud compute firewall-rules create gke-$CLUSTER-master-to-vault-injector \
  --network=$NETWORK --direction=INGRESS --action=ALLOW \
  --source-ranges=$MASTER_CIDR --target-tags=$NODE_TAG --rules=tcp:8080

Delete an annotated pod and check that the injector gets the call:

1
2
3
4
5

kubectl -n app delete pod -l app=api
kubectl -n vault logs deploy/vault-agent-injector | grep mutate
# [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
kubectl -n app get pod -l app=api -o jsonpath='{.items[0].spec.initContainers[*].name} {.items[0].spec.containers[*].name}{"\n"}'
# vault-agent-init api vault-agent

The symptom

• The pod carries vault.hashicorp.com/agent-inject: "true" and the rest of the annotations, but starts with the application container only: no vault-agent-init, no vault-agent.
• Creating the pod takes about 30 seconds longer than usual.
• kubectl get events says nothing about the webhook. The injector logs do not have a single Request received line.
• On other clusters (public zonal, kind, k3s) the same chart and the same annotations work.

Why

On a private GKE cluster the control plane lives in a Google-owned VPC and reaches your nodes through the firewall rule GKE creates by itself (gke-<cluster>-<hash>-master). That rule only allows TCP 443 and 10250 from the control plane range to the nodes. Everything else is closed.

An admission webhook does not go through the Service like a normal client would: the API server resolves the Service to its endpoints and connects straight to the pod IP on the targetPort. The injector’s Service (vault-agent-injector-svc) exposes 443, but the pod listens on 8080 (injector.port in the chart), so the actual connection goes to 8080 and the firewall drops it.

What happens next is explained by two chart defaults:

Hence the 30-second delay and the pod with no sidecar and no message. With failurePolicy: Fail you would have seen failed calling webhook "vault.hashicorp.com": … context deadline exceeded in the ReplicaSet events, which at least points at the right place.

Not just Vault

Any admission webhook, CRD conversion webhook or aggregated API listening on a port other than 443 or 10250 fails the same way on private GKE: the OpenTelemetry operator (9443), Kyverno (9443), Gatekeeper (8443)… cert-manager listens on 10250 precisely to avoid needing this rule. The GKE documentation lists it as a requirement for “admission webhooks, aggregated API servers, webhook conversion”: one rule per port, from masterIpv4CidrBlock to the node tags.

If the cluster lives in Terraform, keep the rule next to the cluster so it survives a rebuild:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

resource "google_compute_firewall" "master_to_vault_injector" {
  name          = "gke-${google_container_cluster.main.name}-master-to-vault-injector"
  network       = google_compute_network.main.name
  direction     = "INGRESS"
  source_ranges = [google_container_cluster.main.private_cluster_config[0].master_ipv4_cidr_block]
  target_tags   = ["gke-${google_container_cluster.main.name}"]   # the tag your node pools use
  allow {
    protocol = "tcp"
    ports    = ["8080"]
  }
}

Traps

• The wrong tag: nodes carry gke-<cluster>-<hash>-node; copy it from the existing gke-<cluster>-* rules instead of typing it. With several node pools, check that all of them have the tag.
• The control plane range changes if you recreate the cluster; a hand-written rule with the old CIDR silently stops working.
• failurePolicy: Fail makes the failure visible, but while the webhook is down no pod gets created in the namespaces matching the namespaceSelector; the chart excludes the injector itself, but not Vault if they share namespace and labels. Use it with replicas: 2.
• The injector does receive the request and there is still no sidecar: then it is no longer the network. Read the vault-agent-init logs: permission denied is the Kubernetes auth role (bound_service_account_names, bound_service_account_namespaces) or the policy; connection refused is vault.hashicorp.com/service or the Vault address.
• One port per rule does not scale: as webhooks pile up, one rule with --rules=tcp:8080,8443,9443 from the control plane is easier to maintain than five.

References: vault-k8s #46, vault-k8s #32 and firewall rules on private clusters in the GKE documentation.

See also

• cert-manager: the HTTP-01 challenge returns 308 — another component that “does nothing” until you look in the right place.
• GitHub Actions → Google Cloud without keys — to deploy to that GKE without JSON keys.

1

Waiting for HTTP-01 challenge propagation: wrong status code '308', expected '200'

cert-manager creates a temporary Ingress (cm-acme-http-solver-…) to answer the challenge, but without a class your ingress controller does not adopt it; the request to /.well-known/acme-challenge/… gets answered by the application’s Ingress, which redirects to HTTPS with 308. Put the class in the solver:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          ingressClassName: nginx        # cert-manager >= 1.12; before: class: nginx

Apply and delete the stuck challenge so it retries:

1

kubectl delete challenge -n <ns> --all

Diagnosis

1
2
3
4

kubectl get certificate,certificaterequest,order,challenge -A
kubectl describe challenge -n <ns> <name>              # Status → Reason is the line you want
kubectl get ingress -n <ns>                            # does cm-acme-http-solver-xxxx show CLASS <none>?
curl -i http://app.example.com/.well-known/acme-challenge/test   # from outside and from a pod

Before asking Let’s Encrypt to validate, cert-manager runs a self-check: it tries the challenge URL itself. The Challenge’s Reason is the result of that self-check and almost always points at the cause.

Why ingressClassName

Since Kubernetes 1.18 an Ingress’s class goes in spec.ingressClassName; the kubernetes.io/ingress.class annotation is deprecated and modern controllers (ingress-nginx 1.x, Traefik) may ignore it. In cert-manager, the solver’s class field still writes the annotation; ingressClassName (since 1.12) writes the field. Use one of the two, and in 2026 the right one is ingressClassName.

Without a class, the temporary Ingress exists but nobody serves it. The controller routes the request by host, finds the app’s Ingress, and applies its rules: ssl-redirect → 308. That is why the error looks like the app’s and not cert-manager’s.

Alternative: DNS-01 with Cloudflare

When the cluster is not reachable from the Internet on port 80, or you want a wildcard, the challenge goes through DNS. Token in Cloudflare with Zone → DNS → Edit and Zone → Zone → Read on the zone:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token
  namespace: cert-manager
type: Opaque
stringData:
  api-token: <token>
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-dns-account-key
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token
            key: api-token

With DNS-01 there is no temporary Ingress, no HTTP self-check, no hairpin; the self-check queries the TXT record, and if the cluster’s DNS resolves against an internal server with the zone cached, the Reason will be DNS record for … not yet propagated: patience, or --dns01-recursive-nameservers=1.1.1.1:53 on cert-manager.

Traps

• Staging first. https://acme-staging-v02.api.letsencrypt.org/directory has no issuance limits; production blocks 5 issuances per name per week, and every failed Challenge that reaches Let’s Encrypt counts as an attempt.
• Delete the Challenge, not the Certificate. Deleting the Order or the Challenge forces a retry with the new configuration; deleting the Certificate also deletes the secret and leaves the app without TLS meanwhile.
• pathType: Exact on the solver Ingress (recent cert-manager) clashes with some old controllers; if you see 503 on OpenShift, that is it.
• Issuer per namespace vs ClusterIssuer: the Ingress must annotate the one that exists (cert-manager.io/cluster-issuer or cert-manager.io/issuer); annotating the one that does not leaves the Certificate in Issuing forever with no Challenge to describe.

References: HTTP01 solver, Cloudflare DNS01, and Troubleshooting ACME in the cert-manager documentation.