In Logs Explorer, every tunnel phase has its query:

Cloud VPN runs strongSwan underneath, so the messages are strongSwan’s: if you know charon, you already know how to read them.

What each message means

establishing IKE_SA failed, peer not responding. Cloud VPN sends IKE_SA_INIT and nobody answers. In order of likelihood: the peer is on IKEv1 and the tunnel on IKEv2 (or the reverse), the configured peer IP is not the real public one (NAT in front of the firewall), UDP 500 and 4500 do not reach the peer, or the peer does not even have the tunnel enabled. The PSK does not produce this message: with a wrong PSK, phase 1 starts and fails at IKE_AUTH.

IKE_SA … established between and then nothing. Phase 1 fine, phase 2 not. Look for CHILD_SA and traffic selectors.

Remote traffic selectors narrowed / Local traffic selectors narrowed. The phase 2 networks do not match on both sides. On HA VPN (and route-based Classic VPN) Cloud VPN proposes 0.0.0.0/0 on both sides and expects the same; a policy-based peer with specific subnets makes strongSwan narrow the selectors and pass traffic only between what matched, or nothing. On the peer: phase 2 with 0.0.0.0/0 local and remote (on pfSense, Routed mode, which creates an ipsecN tunnel interface), and routing handled by BGP or static routes.

received DELETE for ESP CHILD_SA in a loop. The peer tears phase 2 down every few minutes: mismatched phase 2 lifetimes with badly negotiated rekeys, or aggressive DPD. Match lifetimes (Cloud VPN uses 3 hours for phase 2 and 10 for phase 1) and let only one side initiate rekeys.

Comes up, passes traffic for a while, dies with NAT-T. A peer behind NAT on UDP 4500 and a middlebox that “helps” (IPsec ALG on the router) drops encapsulated ESP packets. Disable the ALG or move the firewall onto a direct public IP.

HA VPN with BGP against pfSense (FRR)

The tunnel comes up, bgp establishes, and no route comes in or goes out. FRR on pfSense starts with a policy that rejects everything not explicitly permitted:

1. Services → FRR → BGP → Route Maps: create ALLOW-ALL, Action Permit, Sequence 100, no criteria.
2. BGP → Neighbors → the Cloud Router peer: Remote AS the Cloud Router’s ASN, and under Peer Filtering assign ALLOW-ALL as route map in and out.
3. BGP → Network Distribution: list the local networks you want to advertise (or Default Originate if everything goes through the tunnel).
4. On GCP, the Cloud Router with --advertisement-mode custom if you want to control which prefixes go out; by default it advertises the VPC subnets.

Without step 2, vtysh -c "show ip bgp summary" shows the session Established with 0 prefixes in both directions, which is exactly the symptom.

Traps

• IKEv2 unless the peer cannot: IKEv1 is still supported, but without IPv6 and with fewer ciphers. Classic VPN still exists (99.9%), but part of its functionality is retired; new work goes on HA VPN (99.99%).
• Two tunnels per gateway on HA VPN for the 99.99%: if you only bring one up, the gateway status stays in warning even though it works.
• MTU: 1460 by default on the GCP VPC; with IPsec on top, let TCP do MSS clamping on the peer (pfSense does it on the tunnel’s ipsecN interface) or you will see transfers hanging after the handshake.
• Logs lag: a minute or two between the event and Logs Explorer. Refresh before changing the configuration again.
• A PSK with special characters copied from a password manager: check on the peer that nothing got eaten; the symptom is IKE_AUTH failing, not peer not responding.

See also

• pfSense: NAT reflection and split DNS — the other classic of “the tunnel is fine and I still cannot get there”.
• Cloud VPN troubleshooting in the Google Cloud documentation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

export PROJECT_ID=my-project
export PROJECT_NUMBER=$(gcloud projects describe $PROJECT_ID --format='value(projectNumber)')
export GITHUB_ORG=my-org
export REPO=my-org/my-repo

gcloud iam workload-identity-pools create github \
  --project=$PROJECT_ID --location=global --display-name="GitHub Actions"

gcloud iam workload-identity-pools providers create-oidc github-oidc \
  --project=$PROJECT_ID --location=global --workload-identity-pool=github \
  --display-name="GitHub OIDC" \
  --issuer-uri="https://token.actions.githubusercontent.com" \
  --attribute-mapping="google.subject=assertion.sub,attribute.actor=assertion.actor,attribute.repository=assertion.repository,attribute.repository_owner=assertion.repository_owner" \
  --attribute-condition="assertion.repository_owner == '$GITHUB_ORG'"

gcloud iam service-accounts create github-actions --project=$PROJECT_ID --display-name="GitHub Actions"
gcloud iam service-accounts add-iam-policy-binding github-actions@$PROJECT_ID.iam.gserviceaccount.com \
  --project=$PROJECT_ID --role=roles/iam.workloadIdentityUser \
  --member="principalSet://iam.googleapis.com/projects/$PROJECT_NUMBER/locations/global/workloadIdentityPools/github/attribute.repository/$REPO"

And in the workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      id-token: write            # without this GitHub does not issue the OIDC token
    steps:
      - uses: actions/checkout@v4
      - id: auth
        uses: google-github-actions/auth@v3
        with:
          workload_identity_provider: projects/123456789012/locations/global/workloadIdentityPools/github/providers/github-oidc
          service_account: github-actions@my-project.iam.gserviceaccount.com
      - uses: google-github-actions/setup-gcloud@v3
      - run: gcloud storage ls

Whatever roles the pipeline needs (Artifact Registry, Cloud Run, GKE…) go to github-actions@… like to any service account.

Why

The classic alternative is exporting a service account JSON key and storing it as a GitHub secret. That key never expires, anyone with access to the repo or the logs can copy it, and rotating it is manual work nobody does. With Workload Identity Federation GitHub issues a short-lived OIDC token for each job, Google verifies it against GitHub’s issuer, checks your attribute condition, and hands out temporary service account credentials. There is no secret to store or rotate, and you can restrict who gets in down to the repo or branch.

The pieces

To restrict to a branch, map the claim and use it in both the condition and the binding:

1
2

--attribute-mapping="…,attribute.ref=assertion.ref"
--attribute-condition="assertion.repository_owner == 'my-org' && assertion.ref == 'refs/heads/main'"

With Terraform

The official module creates pool, provider, and bindings in one block:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

module "github_actions_sa" {
  source     = "terraform-google-modules/service-accounts/google"
  version    = "~> 4.0"
  project_id = var.project_id
  names      = ["github-actions"]
  project_roles = ["${var.project_id}=>roles/artifactregistry.writer"]
}

module "gh_oidc" {
  source              = "terraform-google-modules/github-actions-runners/google//modules/gh-oidc"
  project_id          = var.project_id
  pool_id             = "github"
  provider_id         = "github-oidc"
  attribute_condition = "assertion.repository_owner == 'my-org'"
  sa_mapping = {
    "github-actions" = {
      sa_name   = "projects/${var.project_id}/serviceAccounts/${module.github_actions_sa.email}"
      attribute = "attribute.repository/my-org/my-repo"
    }
  }
}

output "workload_identity_provider" { value = module.gh_oidc.provider_name }

The output is exactly the value that goes into the workflow’s workload_identity_provider.

Without a service account

auth@v3 also supports direct federation: the principalSet gets IAM roles without going through a service account (workload_identity_provider and project_id, no service_account). Fewer pieces, but not every service accepts federated principals; Artifact Registry, GCS, and most APIs do, and it is the path Google recommends for new setups.

Traps

• id-token: write in the job’s permissions. Without it, auth fails with “Unable to get ACTIONS_ID_TOKEN_REQUEST_URL”.
• Project number, not ID, in the provider path and the principalSet: projects/123456789012/….
• Permission 'iam.serviceAccounts.getAccessToken' denied: the roles/iam.workloadIdentityUser binding is missing or attribute.repository does not match (case and exact repo name).
• The condition must reference mapped claims: if you use assertion.ref in the condition without mapping it to an attribute, the provider gets created but never matches.
• Forks and external PRs: tokens from a fork’s PR carry the fork’s repository; a condition on repository_owner keeps them out, which is usually what you want.
• Rotation: nothing to rotate, but something to review. gcloud iam workload-identity-pools providers describe and the STS audit logs tell you who is coming in.

Reference: google-github-actions/auth and Workload Identity Federation in the Google Cloud documentation.

1
2

example.com.   CAA 0 issue "pki.goog"
example.com.   CAA 0 issue "letsencrypt.org"

1
2
3
4
5

gcloud compute ssl-certificates describe web-cert --global --format='yaml(managed)'
# managed:
#   domainStatus:
#     www.example.com: PROVISIONING | ACTIVE | FAILED_NOT_VISIBLE | FAILED_CAA_FORBIDDEN | FAILED_CAA_CHECKING | FAILED_RATE_LIMITED
#   status: PROVISIONING | ACTIVE

To reach ACTIVE three things must hold at once: the domain’s A/AAAA points to the load balancer IP (and nothing else), the certificate is attached to a target proxy with its forwarding rule, and no CAA in the zone forbids Google’s CAs. Then, up to 60 minutes.

Why

Google issues managed certificates through Google Trust Services (pki.goog) or Let’s Encrypt, and picks one or the other on each issuance or renewal. To validate the domain it runs an HTTP check against the IP the name resolves to, so until DNS points at the load balancer there is nothing to validate. And like any CA, it honors CAA records: if your zone says only digicert.com may issue (a typical corporate policy), Google cannot.

The statuses tell the story:

The right CAA

With no CAA record at all, any CA may issue and Google works. If you have CAA (or your organization requires it), add both CAs:

1
2

example.com.   CAA 0 issue "pki.goog"
example.com.   CAA 0 issue "letsencrypt.org"

Listing only one forces Google to always use that one; the documentation advises against it because a renewal may come out through the other. CAA records inherit downwards: a CAA on example.com covers www.example.com if the subdomain has none of its own.

1

dig +short CAA example.com

Certificate Manager: issue before switching DNS

With classic certificates (compute ssl-certificates) the domain must already point at the load balancer, which during a migration means a few minutes of broken HTTPS. Certificate Manager solves that with DNS authorization: you publish a validation CNAME and Google issues the certificate while traffic still flows to the old site.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

gcloud certificate-manager dns-authorizations create www-auth --domain=www.example.com
gcloud certificate-manager dns-authorizations describe www-auth --format='yaml(dnsResourceRecord)'
# dnsResourceRecord: name: _acme-challenge.www.example.com. type: CNAME data: xxxx.authorize.certificatemanager.goog.

# publish that CNAME in your zone and then:
gcloud certificate-manager certificates create www-cert \
  --domains=www.example.com --dns-authorizations=www-auth
gcloud certificate-manager maps create web-map
gcloud certificate-manager maps entries create www --map=web-map --certificate=www-cert --hostname=www.example.com
gcloud compute target-https-proxies update web-proxy --certificate-map=web-map

Once the certificate is ACTIVE, you flip the A/AAAA and the cutover is zero. CAA applies the same way.

Traps

• Cloudflare or another proxy in front of the load balancer: the A record points at the proxy, not at Google, and validation fails with FAILED_NOT_VISIBLE. Grey cloud during issuance, or Certificate Manager with DNS authorization.
• A managed certificate attached to nothing never gets issued. Creating it and waiting does nothing; attach it to the target proxy first.
• Changing the domain list creates a new certificate: attach the new one, wait for ACTIVE, and only then remove the old one from the proxy, or you get a gap.
• Wildcards only with Certificate Manager and DNS authorization; classic managed certificates do not support them.
• Badly signed DNSSEC breaks CAA lookups: FAILED_CAA_CHECKING even when you have no CAA at all.

Reference: Google-managed SSL certificates and Certificate Manager in the Google Cloud documentation.

MySQL, on the source server:

1
2
3
4

mysqldump --databases app -h db.example -u dumper -p \
  --hex-blob --single-transaction --set-gtid-purged=OFF \
  --default-character-set=utf8mb4 --routines --triggers --events \
  | sed -E 's/DEFINER=`[^`]+`@`[^`]+`//g' | gzip > app.sql.gz

PostgreSQL, on the source server:

1
2

pg_dump app -h db.example -U dumper --format=plain --no-owner --no-acl \
  | sed -E 's/(DROP|CREATE|COMMENT ON) EXTENSION/-- \1 EXTENSION/g' | gzip > app.sql.gz

And in both cases:

1
2
3
4
5
6
7

SA=$(gcloud sql instances describe my-instance --format='value(serviceAccountEmailAddress)')
gcloud storage buckets add-iam-policy-binding gs://my-dumps \
  --member="serviceAccount:$SA" --role=roles/storage.objectAdmin
gcloud storage cp app.sql.gz gs://my-dumps/

gcloud sql import sql my-instance gs://my-dumps/app.sql.gz --database=app                 # MySQL
gcloud sql import sql my-instance gs://my-dumps/app.sql.gz --database=app --user=postgres  # PostgreSQL

gcloud sql import sql takes plain SQL or .gz: the custom format of pg_dump goes through pg_restore (below).

The errors and where they come from

On Cloud SQL nobody has SUPER (MySQL) or is a superuser (PostgreSQL); the admin is root@% or postgres, a member of cloudsqlsuperuser. Everything a normal dump takes for granted on your own server breaks right there:

MySQL: what to do with DEFINER clauses

First see how many there are on the source, and whose:

1
2
3
4

SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.VIEWS;
SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.TRIGGERS;
SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.ROUTINES;
SELECT DISTINCT DEFINER FROM INFORMATION_SCHEMA.EVENTS;

Three ways out, from most to least faithful to the original:

1. Create the users on the instance before importing, with the same user@host: gcloud sql users create app --host=% --instance=my-instance --password=…. This is what the documentation recommends, and it keeps views and procedures exactly as they were. root@localhost is the exception: it cannot be created on Cloud SQL, so a DEFINER with that value must be changed no matter what.
2. Strip the DEFINER with the sed above: objects get created with the importing user as definer. Enough when the source users will never exist on Cloud SQL.
3. Switch to SQL SECURITY INVOKER (sed -E 's/SQL SECURITY DEFINER/SQL SECURITY INVOKER/g' on top of the previous one): the view or procedure runs with the caller’s privileges and the definer stops mattering. It changes the permission model; check that the application users can read the underlying tables.

The sed is deliberately broad because DEFINER= also shows up inside conditional comments (/*!50013 DEFINER=… SQL SECURITY DEFINER */); deleting just the DEFINER= leaves the rest intact and valid.

MySQL: the flags, one by one

If the user you dump with lacks PROCESS, mysqldump 8.0.21+ fails right away with “Access denied; you need (at least one of) the PROCESS privilege(s)” while querying tablespaces: add --no-tablespaces.

PostgreSQL: plain or custom

--no-owner --no-acl make everything imported belong to --user, with no GRANT to phantom roles. If the application connects as another role, create the role first and use --user=app, or reassign afterwards:

1

REASSIGN OWNED BY postgres TO app;   -- connected as postgres, in the imported database

Create the extensions the database needs (pg_trgm, postgis, uuid-ossp…) before importing, as postgres, in the target database: CREATE EXTENSION IF NOT EXISTS pg_trgm;. Only the ones on Cloud SQL’s supported extensions list work; plpgsql is already there. With that in place, commenting out the dump’s CREATE EXTENSION lines removes nothing.

If you prefer the custom format (-Fc), the extension filter goes into the pg_restore table of contents, exactly as Google documents it, and you restore through the Cloud SQL Auth Proxy:

1
2
3

pg_dump app -h db.example -U dumper --format=custom --no-owner --no-acl > app.dmp
pg_restore --list app.dmp | sed -E 's/(.*EXTENSION )/; \1/g' > app.toc
pg_restore --use-list=app.toc --no-owner --no-acl -j 4 -h 127.0.0.1 -U postgres -d app app.dmp

-j 4 restores in parallel, which the import from the bucket does not do.

Traps

• The bucket role goes to the instance’s service account, not yours: the docs ask for storage.objectAdmin (for import alone storage.objects.get and list, that is objectViewer, are enough). Without it, “The service account does not have the required permissions for the bucket” even though you can read the object yourself.
• One operation at a time per instance: with a backup or export running, the import fails with “operation already in progress”. gcloud sql operations list --instance=my-instance shows it.
• Large dumps: the import is serial and slow; launch it with --async and watch it with gcloud sql operations wait OPERATION_ID --timeout=unlimited, or gcloud dies on you along the way.
• pg_dump version: use the same major version as the target or lower. pg_dump 17 writes SET transaction_timeout = 0;, which a PostgreSQL 16 rejects with “unrecognized configuration parameter” and aborts the import.
• utf8mb4 on the instance: --default-character-set=utf8mb4 in the dump does not fix an instance created with latin1; check character_set_server in the instance flags before importing.
• The DEFINER sed also touches data: if some row literally contains DEFINER=`x`@`y` (rare, but a CMS that stores SQL can), skip it with a pattern anchored to CREATE, or review the dump diff.

See also

• PostgreSQL: admin cheat sheet — roles, pg_dump, REASSIGN OWNED and friends.
• Import and export using SQL dump files (MySQL), DEFINER clause and pg_dump / pg_restore (PostgreSQL) in the Cloud SQL documentation.

1
2
3
4
5
6
7
8
9

CLUSTER=my-cluster; REGION=us-central1; NETWORK=my-vpc
MASTER_CIDR=$(gcloud container clusters describe $CLUSTER --region $REGION \
  --format='value(privateClusterConfig.masterIpv4CidrBlock)')
NODE_TAG=$(gcloud compute firewall-rules list --filter="name~^gke-$CLUSTER" \
  --format='value(targetTags.list())' | head -1)

gcloud compute firewall-rules create gke-$CLUSTER-master-to-vault-injector \
  --network=$NETWORK --direction=INGRESS --action=ALLOW \
  --source-ranges=$MASTER_CIDR --target-tags=$NODE_TAG --rules=tcp:8080

Delete an annotated pod and check that the injector gets the call:

1
2
3
4
5

kubectl -n app delete pod -l app=api
kubectl -n vault logs deploy/vault-agent-injector | grep mutate
# [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
kubectl -n app get pod -l app=api -o jsonpath='{.items[0].spec.initContainers[*].name} {.items[0].spec.containers[*].name}{"\n"}'
# vault-agent-init api vault-agent

The symptom

• The pod carries vault.hashicorp.com/agent-inject: "true" and the rest of the annotations, but starts with the application container only: no vault-agent-init, no vault-agent.
• Creating the pod takes about 30 seconds longer than usual.
• kubectl get events says nothing about the webhook. The injector logs do not have a single Request received line.
• On other clusters (public zonal, kind, k3s) the same chart and the same annotations work.

Why

On a private GKE cluster the control plane lives in a Google-owned VPC and reaches your nodes through the firewall rule GKE creates by itself (gke-<cluster>-<hash>-master). That rule only allows TCP 443 and 10250 from the control plane range to the nodes. Everything else is closed.

An admission webhook does not go through the Service like a normal client would: the API server resolves the Service to its endpoints and connects straight to the pod IP on the targetPort. The injector’s Service (vault-agent-injector-svc) exposes 443, but the pod listens on 8080 (injector.port in the chart), so the actual connection goes to 8080 and the firewall drops it.

What happens next is explained by two chart defaults:

Hence the 30-second delay and the pod with no sidecar and no message. With failurePolicy: Fail you would have seen failed calling webhook "vault.hashicorp.com": … context deadline exceeded in the ReplicaSet events, which at least points at the right place.

Not just Vault

Any admission webhook, CRD conversion webhook or aggregated API listening on a port other than 443 or 10250 fails the same way on private GKE: the OpenTelemetry operator (9443), Kyverno (9443), Gatekeeper (8443)… cert-manager listens on 10250 precisely to avoid needing this rule. The GKE documentation lists it as a requirement for “admission webhooks, aggregated API servers, webhook conversion”: one rule per port, from masterIpv4CidrBlock to the node tags.

If the cluster lives in Terraform, keep the rule next to the cluster so it survives a rebuild:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

resource "google_compute_firewall" "master_to_vault_injector" {
  name          = "gke-${google_container_cluster.main.name}-master-to-vault-injector"
  network       = google_compute_network.main.name
  direction     = "INGRESS"
  source_ranges = [google_container_cluster.main.private_cluster_config[0].master_ipv4_cidr_block]
  target_tags   = ["gke-${google_container_cluster.main.name}"]   # the tag your node pools use
  allow {
    protocol = "tcp"
    ports    = ["8080"]
  }
}

Traps

• The wrong tag: nodes carry gke-<cluster>-<hash>-node; copy it from the existing gke-<cluster>-* rules instead of typing it. With several node pools, check that all of them have the tag.
• The control plane range changes if you recreate the cluster; a hand-written rule with the old CIDR silently stops working.
• failurePolicy: Fail makes the failure visible, but while the webhook is down no pod gets created in the namespaces matching the namespaceSelector; the chart excludes the injector itself, but not Vault if they share namespace and labels. Use it with replicas: 2.
• The injector does receive the request and there is still no sidecar: then it is no longer the network. Read the vault-agent-init logs: permission denied is the Kubernetes auth role (bound_service_account_names, bound_service_account_namespaces) or the policy; connection refused is vault.hashicorp.com/service or the Vault address.
• One port per rule does not scale: as webhooks pile up, one rule with --rules=tcp:8080,8443,9443 from the control plane is easier to maintain than five.

References: vault-k8s #46, vault-k8s #32 and firewall rules on private clusters in the GKE documentation.

See also

• cert-manager: the HTTP-01 challenge returns 308 — another component that “does nothing” until you look in the right place.
• GitHub Actions → Google Cloud without keys — to deploy to that GKE without JSON keys.