TL;DR

To remove Rancher completely from a cluster where it was installed (irreversible, backup taken):

1
2
3
4

kubectl create -f https://raw.githubusercontent.com/rancher/rancher-cleanup/main/deploy/rancher-cleanup.yaml
kubectl -n kube-system logs -l job-name=cleanup-job -f
kubectl create -f https://raw.githubusercontent.com/rancher/rancher-cleanup/main/deploy/verify.yaml
kubectl -n kube-system logs -l job-name=verify-job -f      # should print nothing

To fix a single thing without razing everything, read on.

Namespace stuck in Terminating

cattle-system, cattle-fleet-system, cattle-impersonation-system, local, the cluster-fleet-* ones, or any namespace Rancher created, stay in Terminating forever:

1

kubectl get ns cattle-system -o jsonpath='{.metadata.finalizers}{"\n"}{.spec.finalizers}{"\n"}{.status.conditions}{"\n"}'

Two different cases:

  1. metadata.finalizers contains controller.cattle.io/namespace-auth (or any *.cattle.io). Rancher put it there and only Rancher removes it; if Rancher is gone, nobody will. Remove it yourself:

    1

    kubectl patch ns cattle-system --type=merge -p '{"metadata":{"finalizers":[]}}'

  2. spec.finalizers contains kubernetes and status.conditions says SomeResourcesRemain or DiscoveryFailed. The API server will not delete the namespace because resources remain inside, almost always CRs of an API group whose service disappeared (a Rancher apiservice). Before forcing, look at what it is:

    1
2
3

    kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n1 kubectl -n cattle-system get --ignore-not-found --show-kind 2>/dev/null
kubectl get apiservice | grep -v True        # dead APIs that break discovery

    Delete the dead apiservice or the CRs with finalizers, and the namespace goes away on its own. Forcing spec.finalizers to empty is the last resort and leaves orphans behind:

    1
2

    kubectl get ns cattle-system -o json | jq '.spec.finalizers=[]' \
  | kubectl replace --raw /api/v1/namespaces/cattle-system/finalize -f -

Cluster that cannot be deleted

An imported cluster that no longer exists stays in Removing in the UI, or kubectl delete never finishes. There are two objects with finalizers, one per API:

1
2
3
4
5

kubectl -n fleet-default get clusters.provisioning.cattle.io
kubectl -n fleet-default patch clusters.provisioning.cattle.io my-cluster --type=merge -p '{"metadata":{"finalizers":[]}}'

kubectl get clusters.management.cattle.io          # c-m-xxxxxxxx (v2.6+) or c-xxxxx
kubectl patch clusters.management.cattle.io c-m-xxxxxxxx --type=merge -p '{"metadata":{"finalizers":[]}}'

Then clean up the cluster-fleet-default-<name>-<hash> namespace and whatever clusters.fleet.cattle.io remain in fleet-default.

The webhook that rejects everything

rancher-webhook validates and mutates *.cattle.io resources (and some core ones, such as namespaces and ServiceAccounts on managed clusters). If the pod is gone or broken, every change fails with:

1

Error from server (InternalError): Internal error occurred: failed calling webhook "rancher.cattle.io": … connect: connection refused

Export the configurations, delete them, let the pod come back healthy, and restore them:

1
2
3
4
5
6

kubectl get validatingwebhookconfiguration rancher.cattle.io -o yaml > vwc-rancher.yaml
kubectl get mutatingwebhookconfiguration rancher.cattle.io -o yaml > mwc-rancher.yaml
kubectl delete validatingwebhookconfiguration rancher.cattle.io
kubectl delete mutatingwebhookconfiguration rancher.cattle.io
# … fix rancher-webhook (kubectl -n cattle-system rollout restart deploy/rancher-webhook) …
kubectl apply -f vwc-rancher.yaml -f mwc-rancher.yaml

If you are uninstalling, do not restore them: they are the first thing to remove so the rest of the deletion can proceed.

Getting access back

Lost admin password, or the admin user disabled by mistake:

1
2
3

kubectl -n cattle-system exec -it deploy/rancher -- reset-password
# New password for default administrator (user-xxxxx): …
kubectl -n cattle-system exec -it deploy/rancher -- ensure-default-admin

ensure-default-admin re-enables the user and gives the administrator role back; reset-password only changes the password.

Traps

  • rancher-cleanup deletes the downstream cluster objects (the clusters keep running, but are no longer registered) and everything Rancher installed: monitoring, logging, Gatekeeper. Longhorn stays. Do not run it on the Rancher cluster you want to keep.
  • Backup first: kubectl get -A -o yaml of the *.cattle.io resources you care about, or Rancher’s backup-operator. Finalizers go away in a second; what was behind them does not come back.
  • kubectl delete ns --force --grace-period=0 does nothing for a namespace in Terminating: the block is in the finalizers, not in the grace period.
  • Rancher puts the finalizers back while it is alive: if you are cleaning a cluster still connected to a Rancher, disconnect it first (delete the cattle-cluster-agent) or your patches undo themselves.
  • The *.cattle.io CRDs survive a Helm uninstall; rancher-cleanup removes them, a manual uninstall does not.

See also