TL;DR

1
2
3
4
5
6
7
8
9

CLUSTER=my-cluster; REGION=us-central1; NETWORK=my-vpc
MASTER_CIDR=$(gcloud container clusters describe $CLUSTER --region $REGION \
  --format='value(privateClusterConfig.masterIpv4CidrBlock)')
NODE_TAG=$(gcloud compute firewall-rules list --filter="name~^gke-$CLUSTER" \
  --format='value(targetTags.list())' | head -1)

gcloud compute firewall-rules create gke-$CLUSTER-master-to-vault-injector \
  --network=$NETWORK --direction=INGRESS --action=ALLOW \
  --source-ranges=$MASTER_CIDR --target-tags=$NODE_TAG --rules=tcp:8080

Delete an annotated pod and check that the injector gets the call:

1
2
3
4
5

kubectl -n app delete pod -l app=api
kubectl -n vault logs deploy/vault-agent-injector | grep mutate
# [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
kubectl -n app get pod -l app=api -o jsonpath='{.items[0].spec.initContainers[*].name} {.items[0].spec.containers[*].name}{"\n"}'
# vault-agent-init api vault-agent

The symptom

  • The pod carries vault.hashicorp.com/agent-inject: "true" and the rest of the annotations, but starts with the application container only: no vault-agent-init, no vault-agent.
  • Creating the pod takes about 30 seconds longer than usual.
  • kubectl get events says nothing about the webhook. The injector logs do not have a single Request received line.
  • On other clusters (public zonal, kind, k3s) the same chart and the same annotations work.

Why

On a private GKE cluster the control plane lives in a Google-owned VPC and reaches your nodes through the firewall rule GKE creates by itself (gke-<cluster>-<hash>-master). That rule only allows TCP 443 and 10250 from the control plane range to the nodes. Everything else is closed.

An admission webhook does not go through the Service like a normal client would: the API server resolves the Service to its endpoints and connects straight to the pod IP on the targetPort. The injector’s Service (vault-agent-injector-svc) exposes 443, but the pod listens on 8080 (injector.port in the chart), so the actual connection goes to 8080 and the firewall drops it.

What happens next is explained by two chart defaults:

ValueDefaultEffect
injector.webhook.timeoutSeconds30The API server waits 30 s for the webhook’s answer
injector.webhook.failurePolicyIgnoreIf the webhook does not answer, the pod is created unmutated, with no error

Hence the 30-second delay and the pod with no sidecar and no message. With failurePolicy: Fail you would have seen failed calling webhook "vault.hashicorp.com": … context deadline exceeded in the ReplicaSet events, which at least points at the right place.

Not just Vault

Any admission webhook, CRD conversion webhook or aggregated API listening on a port other than 443 or 10250 fails the same way on private GKE: the OpenTelemetry operator (9443), Kyverno (9443), Gatekeeper (8443)… cert-manager listens on 10250 precisely to avoid needing this rule. The GKE documentation lists it as a requirement for “admission webhooks, aggregated API servers, webhook conversion”: one rule per port, from masterIpv4CidrBlock to the node tags.

If the cluster lives in Terraform, keep the rule next to the cluster so it survives a rebuild:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

resource "google_compute_firewall" "master_to_vault_injector" {
  name          = "gke-${google_container_cluster.main.name}-master-to-vault-injector"
  network       = google_compute_network.main.name
  direction     = "INGRESS"
  source_ranges = [google_container_cluster.main.private_cluster_config[0].master_ipv4_cidr_block]
  target_tags   = ["gke-${google_container_cluster.main.name}"]   # the tag your node pools use
  allow {
    protocol = "tcp"
    ports    = ["8080"]
  }
}

Traps

  • The wrong tag: nodes carry gke-<cluster>-<hash>-node; copy it from the existing gke-<cluster>-* rules instead of typing it. With several node pools, check that all of them have the tag.
  • The control plane range changes if you recreate the cluster; a hand-written rule with the old CIDR silently stops working.
  • failurePolicy: Fail makes the failure visible, but while the webhook is down no pod gets created in the namespaces matching the namespaceSelector; the chart excludes the injector itself, but not Vault if they share namespace and labels. Use it with replicas: 2.
  • The injector does receive the request and there is still no sidecar: then it is no longer the network. Read the vault-agent-init logs: permission denied is the Kubernetes auth role (bound_service_account_names, bound_service_account_namespaces) or the policy; connection refused is vault.hashicorp.com/service or the Vault address.
  • One port per rule does not scale: as webhooks pile up, one rule with --rules=tcp:8080,8443,9443 from the control plane is easier to maintain than five.

References: vault-k8s #46, vault-k8s #32 and firewall rules on private clusters in the GKE documentation.

See also