Firewall → Settings → Advanced → Disable reply-to on WAN rules → check → Save → Apply.

Symptom: a port-forward or WAN rule answers from any internet IP, but from a host that lives in the same subnet as the firewall’s WAN everything times out (SSH, HTTPS, even ping), even though the rule says from any.

With a single WAN this is safe: the firewall stops forcing replies through the gateway and goes back to its routing table.

The setup

1
2
3
4
5
6
7
8

                203.0.113.0/24  (same L2 segment)
   ┌────────────────────┬────────────────────┬────────────────────┐
   │                    │                    │                    │
 gateway             fw01 (OPNsense)      neighbor             internet
 203.0.113.1         WAN 203.0.113.40     203.0.113.50         (via gateway)
                        │ rdr :22 → 192.168.1.10
                        ▼
                     LAN server

• From the internet: ssh -p 22 203.0.113.40 reaches the LAN server. Fine.
• From neighbor (203.0.113.50): every port times out.

What reply-to does

OPNsense generates the rules on WAN-type interfaces (those with a gateway) with reply-to pointing at that gateway:

1

pfctl -sr | grep reply-to

1

pass in quick on vtnet0 reply-to (vtnet0 203.0.113.1) inet proto tcp from any to 192.168.1.10 port = ssh flags S/SA keep state

reply-to is a pf option. When a packet comes in and creates a state, pf records in that state that every reply must leave through vtnet0 toward 203.0.113.1. Those replies skip the routing table: pf picks the next hop on its own.

It exists for multi-WAN. If you have two providers and a client comes in on WAN2, the reply must leave through WAN2. If it leaves through the default route (WAN1), it carries a source IP that provider 1 does not recognize and gets lost. reply-to guarantees a symmetric return path without thinking about routes.

Why it breaks neighbors

For an internet client, “reply to the gateway” is exactly what the routing table would do anyway, so nothing looks wrong.

For the neighbor it is different. Its IP (203.0.113.50) is on a connected network of the firewall. Without reply-to, the firewall would ARP for its MAC and answer directly. With reply-to, the SYN-ACK leaves with the gateway’s MAC as destination:

1
2

# on the OPNsense box (csh shell: wrap in sh -c)
sh -c "tcpdump -lnei vtnet0 host 203.0.113.50 and tcp port 22"

1
2

<neighbor-mac> > <fw01-mac>          203.0.113.50.49612 > 203.0.113.40.22: Flags [S]    ← arrives directly
<fw01-mac>     > 00:00:5e:00:01:<vrid> 203.0.113.40.22 > 203.0.113.50.49612: Flags [S.]   ← leaves to the gateway

(00:00:5e:00:01:xx is the VRRP virtual MAC: the gateway was a redundant pair.)

From there it depends on the gateway. To it, the packet arrives on the same interface it would have to leave through, and if it runs a stateful filter it sees a SYN-ACK for a connection whose SYN it never saw. In the lab, the gateway simply did not send it back onto the segment. On the neighbor you only see retried SYNs and no [S.].

The fingerprint on the firewall is a half-open state:

1

pfctl -ss | grep 203.0.113.50

1

all tcp 192.168.1.10:22 (203.0.113.40:22) <- 203.0.113.50:57792   SYN_SENT:ESTABLISHED

The server already answered (ESTABLISHED) and the client is still waiting (SYN_SENT): the reply is lost on the way back, not on the way in. The filter is not blocking anything.

What the fix changes

With Disable reply-to on WAN rules checked, OPNsense regenerates the rules without reply-to:

1

pass in quick on vtnet0 inet proto tcp from any to 192.168.1.10 port = ssh flags S/SA keep state

The state no longer pins the next hop, so replies go through the normal routing table:

The last row is the cost. With multi-WAN, do not disable it globally. Instead, drop reply-to only on the affected port-forward’s rule: in the rule, Advanced → reply-to. Check which options your version offers.

Verify

1
2
3

# on the OPNsense box
sh -c "pfctl -sr | grep -c reply-to; grep -n disablereplyto /conf/config.xml"
# expected: 0  and  <disablereplyto>yes</disablereplyto>

1
2

# from the neighbor
nc -z -w5 203.0.113.40 22 && echo open

Traps

• “Disable force gateway” is not this. It sits on the same page with a similar name, but it only affects traffic originated by the firewall itself (DNS, NTP, updates). It does not touch replies to inbound traffic.
• A static route on the gateway does not help. The gateway already has the subnet as connected; the problem is that it will not send the packet back onto the same segment, not that it does not know the way.
• csh shell. Over SSH, OPNsense drops you into csh, and things like 2>/dev/null fail with Ambiguous output redirect. Use sh -c "..." or ssh root@fw01 sh <<'EOF'.
• Test every port, not one. If only one port fails it is a different problem (rule, NAT, service). If they all fail, ping included, and only from the same subnet, suspect reply-to.

See also

• OPNsense docs, firewall settings: https://docs.opnsense.org/manual/firewall_settings.html
• pf.conf(5), route-to / reply-to options: https://man.freebsd.org/cgi/man.cgi?query=pf.conf

1
2
3
4
5
6
7
8
9

CLUSTER=my-cluster; REGION=us-central1; NETWORK=my-vpc
MASTER_CIDR=$(gcloud container clusters describe $CLUSTER --region $REGION \
  --format='value(privateClusterConfig.masterIpv4CidrBlock)')
NODE_TAG=$(gcloud compute firewall-rules list --filter="name~^gke-$CLUSTER" \
  --format='value(targetTags.list())' | head -1)

gcloud compute firewall-rules create gke-$CLUSTER-master-to-vault-injector \
  --network=$NETWORK --direction=INGRESS --action=ALLOW \
  --source-ranges=$MASTER_CIDR --target-tags=$NODE_TAG --rules=tcp:8080

Delete an annotated pod and check that the injector gets the call:

1
2
3
4
5

kubectl -n app delete pod -l app=api
kubectl -n vault logs deploy/vault-agent-injector | grep mutate
# [INFO]  handler: Request received: Method=POST URL=/mutate?timeout=30s
kubectl -n app get pod -l app=api -o jsonpath='{.items[0].spec.initContainers[*].name} {.items[0].spec.containers[*].name}{"\n"}'
# vault-agent-init api vault-agent

The symptom

• The pod carries vault.hashicorp.com/agent-inject: "true" and the rest of the annotations, but starts with the application container only: no vault-agent-init, no vault-agent.
• Creating the pod takes about 30 seconds longer than usual.
• kubectl get events says nothing about the webhook. The injector logs do not have a single Request received line.
• On other clusters (public zonal, kind, k3s) the same chart and the same annotations work.

Why

On a private GKE cluster the control plane lives in a Google-owned VPC and reaches your nodes through the firewall rule GKE creates by itself (gke-<cluster>-<hash>-master). That rule only allows TCP 443 and 10250 from the control plane range to the nodes. Everything else is closed.

An admission webhook does not go through the Service like a normal client would: the API server resolves the Service to its endpoints and connects straight to the pod IP on the targetPort. The injector’s Service (vault-agent-injector-svc) exposes 443, but the pod listens on 8080 (injector.port in the chart), so the actual connection goes to 8080 and the firewall drops it.

What happens next is explained by two chart defaults:

Hence the 30-second delay and the pod with no sidecar and no message. With failurePolicy: Fail you would have seen failed calling webhook "vault.hashicorp.com": … context deadline exceeded in the ReplicaSet events, which at least points at the right place.

Not just Vault

Any admission webhook, CRD conversion webhook or aggregated API listening on a port other than 443 or 10250 fails the same way on private GKE: the OpenTelemetry operator (9443), Kyverno (9443), Gatekeeper (8443)… cert-manager listens on 10250 precisely to avoid needing this rule. The GKE documentation lists it as a requirement for “admission webhooks, aggregated API servers, webhook conversion”: one rule per port, from masterIpv4CidrBlock to the node tags.

If the cluster lives in Terraform, keep the rule next to the cluster so it survives a rebuild:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

resource "google_compute_firewall" "master_to_vault_injector" {
  name          = "gke-${google_container_cluster.main.name}-master-to-vault-injector"
  network       = google_compute_network.main.name
  direction     = "INGRESS"
  source_ranges = [google_container_cluster.main.private_cluster_config[0].master_ipv4_cidr_block]
  target_tags   = ["gke-${google_container_cluster.main.name}"]   # the tag your node pools use
  allow {
    protocol = "tcp"
    ports    = ["8080"]
  }
}

Traps

• The wrong tag: nodes carry gke-<cluster>-<hash>-node; copy it from the existing gke-<cluster>-* rules instead of typing it. With several node pools, check that all of them have the tag.
• The control plane range changes if you recreate the cluster; a hand-written rule with the old CIDR silently stops working.
• failurePolicy: Fail makes the failure visible, but while the webhook is down no pod gets created in the namespaces matching the namespaceSelector; the chart excludes the injector itself, but not Vault if they share namespace and labels. Use it with replicas: 2.
• The injector does receive the request and there is still no sidecar: then it is no longer the network. Read the vault-agent-init logs: permission denied is the Kubernetes auth role (bound_service_account_names, bound_service_account_namespaces) or the policy; connection refused is vault.hashicorp.com/service or the Vault address.
• One port per rule does not scale: as webhooks pile up, one rule with --rules=tcp:8080,8443,9443 from the control plane is easier to maintain than five.

References: vault-k8s #46, vault-k8s #32 and firewall rules on private clusters in the GKE documentation.

See also

• cert-manager: the HTTP-01 challenge returns 308 — another component that “does nothing” until you look in the right place.
• GitHub Actions → Google Cloud without keys — to deploy to that GKE without JSON keys.