1
2
3
4
5
6

export HELM_DIFF_THREE_WAY_MERGE=true        # diff against what is in the cluster, not against the last release
export HELM_DIFF_NORMALIZE_MANIFESTS=true    # strip formatting noise from the diff
export HELMFILE_GOCCY_GOYAML=true            # helmfile 0.x: avoids "key already set in map"

helmfile -e prod -l name=rabbitmq diff        # a single release
helmfile -e prod apply --skip-diff-on-install # giant first install without a diff

And in CI, pinned versions of all three pieces (helm, helmfile, helm-diff): half of helmfile’s odd errors are combinations nobody tested together.

The errors and what fixes them

Adopting resources that already exist

The typical case: someone installed ArgoCD (or whatever) with kubectl apply -f, and now you want a helmfile release to manage it. Helm refuses to install over resources that are not its own (rendered manifests contain a resource that already exists … invalid ownership metadata). Since Helm 3.2 it is enough to mark them as Helm’s before installing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

NS=argocd; REL=argocd
for kind in deploy sts svc cm secret sa ingress role rolebinding networkpolicy; do
  kubectl -n $NS get $kind -o name 2>/dev/null | while read -r r; do
    kubectl -n $NS label "$r" app.kubernetes.io/managed-by=Helm --overwrite
    kubectl -n $NS annotate "$r" meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=$NS --overwrite
  done
done
# cluster-scoped resources: same labels, no -n
for r in $(kubectl get crd,clusterrole,clusterrolebinding -o name | grep -i argo); do
  kubectl label "$r" app.kubernetes.io/managed-by=Helm --overwrite
  kubectl annotate "$r" meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=$NS --overwrite
done
helmfile -e prod -l name=$REL diff     # should be nearly empty
helmfile -e prod -l name=$REL apply

Only mark what the chart is going to render: a labeled resource the chart does not include is harmless, but one the chart renders under another name ends up duplicated.

helmfile 1.x and Helm 4

Coming from 0.x, this is what changes when you upgrade:

• A helmfile.yaml with Go templates must be called helmfile.yaml.gotmpl; a plain .yaml is no longer rendered.
• environments: and releases: go in separate parts split by --- within the same file.
• helmfile delete is gone (it is destroy), so is helmfile charts, and --args is removed.
• HELMFILE_SKIP_INSECURE_TEMPLATE_FUNCTIONS becomes HELMFILE_DISABLE_INSECURE_FEATURES.
• helmfile 1.x works with Helm 3 and Helm 4. With Helm 4, install helm-diff with helm plugin install https://github.com/databus23/helm-diff --verify=false (Helm 4 verifies plugin provenance and helm-diff does not publish it).

Traps

• apply with no prior diff in CI: helmfile apply already diffs and only applies when there are changes; sync always applies. In pipelines, diff on the PR and apply on main.
• installed: false uninstalls the release on the next apply; handy to retire things through Git, dangerous if copied from another environment.
• Secrets in values: helmfile leaves them in the release manifest (Helm’s Secret in the namespace); use helm-secrets with SOPS or external references.
• --skip-diff-on-install is not --skip-diff: it only skips the diff for releases that do not exist yet. Upgrades still show their diff.
• Three binaries, three versions: helm version, helmfile version and helm plugin list as the first pipeline step, pinned by tag in the CI image.

References: helm-diff, helmfile and the towards 1.0 proposal.

See also

• k3s: import images without a registry — when the release depends on an image that lives in no registry.