TL;DR

1
2
3
4
5
6

export HELM_DIFF_THREE_WAY_MERGE=true        # diff against what is in the cluster, not against the last release
export HELM_DIFF_NORMALIZE_MANIFESTS=true    # strip formatting noise from the diff
export HELMFILE_GOCCY_GOYAML=true            # helmfile 0.x: avoids "key already set in map"

helmfile -e prod -l name=rabbitmq diff        # a single release
helmfile -e prod apply --skip-diff-on-install # giant first install without a diff

And in CI, pinned versions of all three pieces (helm, helmfile, helm-diff): half of helmfile’s odd errors are combinations nobody tested together.

The errors and what fixes them

MessageCauseFix
helmfile diff says nothing changed, but the cluster does not match the charthelm-diff compares the last release’s manifest with the new one, not the live state; a manual kubectl edit never shows upHELM_DIFF_THREE_WAY_MERGE=true (or --three-way-merge in helm-diff)
The diff is full of quoting, key order and indentation changesDifferent Helm or chart versions serialize differentlyHELM_DIFF_NORMALIZE_MANIFESTS=true
key "image" already set in map while rendering valuesThe old YAML library rejects duplicate keys and some anchors in valuesHELMFILE_GOCCY_GOYAML=true on helmfile 0.x; helmfile 1.x already uses that library
Body is too long (maximum is 65536 characters) or the job takes forever on the first installA fresh install’s diff is the whole chart, and the PR comment or the log cannot hold it--skip-diff-on-install
Error: failed to authorize: … 403 Forbidden pulling a chart from oci://ghcr.io/…Helm older than 3.8 with experimental OCI support, or no helm registry loginUpgrade helm; helm registry login ghcr.io if the registry is private
Error: validation: more than one dependency with name or aliashelm and helmfile out of step (helm 3.15 needed helmfile ≥ 0.164; helm 3.13 went with 0.158)Upgrade helmfile alongside helm; on 1.x it no longer happens
The diff has no color in CIhelm-diff detects there is no TTYHELM_DIFF_COLOR=true

Adopting resources that already exist

The typical case: someone installed ArgoCD (or whatever) with kubectl apply -f, and now you want a helmfile release to manage it. Helm refuses to install over resources that are not its own (rendered manifests contain a resource that already exists … invalid ownership metadata). Since Helm 3.2 it is enough to mark them as Helm’s before installing:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

NS=argocd; REL=argocd
for kind in deploy sts svc cm secret sa ingress role rolebinding networkpolicy; do
  kubectl -n $NS get $kind -o name 2>/dev/null | while read -r r; do
    kubectl -n $NS label "$r" app.kubernetes.io/managed-by=Helm --overwrite
    kubectl -n $NS annotate "$r" meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=$NS --overwrite
  done
done
# cluster-scoped resources: same labels, no -n
for r in $(kubectl get crd,clusterrole,clusterrolebinding -o name | grep -i argo); do
  kubectl label "$r" app.kubernetes.io/managed-by=Helm --overwrite
  kubectl annotate "$r" meta.helm.sh/release-name=$REL meta.helm.sh/release-namespace=$NS --overwrite
done
helmfile -e prod -l name=$REL diff     # should be nearly empty
helmfile -e prod -l name=$REL apply

Only mark what the chart is going to render: a labeled resource the chart does not include is harmless, but one the chart renders under another name ends up duplicated.

helmfile 1.x and Helm 4

Coming from 0.x, this is what changes when you upgrade:

  • A helmfile.yaml with Go templates must be called helmfile.yaml.gotmpl; a plain .yaml is no longer rendered.
  • environments: and releases: go in separate parts split by --- within the same file.
  • helmfile delete is gone (it is destroy), so is helmfile charts, and --args is removed.
  • HELMFILE_SKIP_INSECURE_TEMPLATE_FUNCTIONS becomes HELMFILE_DISABLE_INSECURE_FEATURES.
  • helmfile 1.x works with Helm 3 and Helm 4. With Helm 4, install helm-diff with helm plugin install https://github.com/databus23/helm-diff --verify=false (Helm 4 verifies plugin provenance and helm-diff does not publish it).

Traps

  • apply with no prior diff in CI: helmfile apply already diffs and only applies when there are changes; sync always applies. In pipelines, diff on the PR and apply on main.
  • installed: false uninstalls the release on the next apply; handy to retire things through Git, dangerous if copied from another environment.
  • Secrets in values: helmfile leaves them in the release manifest (Helm’s Secret in the namespace); use helm-secrets with SOPS or external references.
  • --skip-diff-on-install is not --skip-diff: it only skips the diff for releases that do not exist yet. Upgrades still show their diff.
  • Three binaries, three versions: helm version, helmfile version and helm plugin list as the first pipeline step, pinned by tag in the CI image.

References: helm-diff, helmfile and the towards 1.0 proposal.

See also