Google-managed certificate that never gets issued: CAA (pki.goog, letsencrypt.org), FAILED_NOT_VISIBLE and the right order

TL;DR

1
2

example.com.   CAA 0 issue "pki.goog"
example.com.   CAA 0 issue "letsencrypt.org"

1
2
3
4
5

gcloud compute ssl-certificates describe web-cert --global --format='yaml(managed)'
# managed:
#   domainStatus:
#     www.example.com: PROVISIONING | ACTIVE | FAILED_NOT_VISIBLE | FAILED_CAA_FORBIDDEN | FAILED_CAA_CHECKING | FAILED_RATE_LIMITED
#   status: PROVISIONING | ACTIVE

To reach ACTIVE three things must hold at once: the domain’s A/AAAA points to the load balancer IP (and nothing else), the certificate is attached to a target proxy with its forwarding rule, and no CAA in the zone forbids Google’s CAs. Then, up to 60 minutes.

Why

Google issues managed certificates through Google Trust Services (pki.goog) or Let’s Encrypt, and picks one or the other on each issuance or renewal. To validate the domain it runs an HTTP check against the IP the name resolves to, so until DNS points at the load balancer there is nothing to validate. And like any CA, it honors CAA records: if your zone says only digicert.com may issue (a typical corporate policy), Google cannot.

The statuses tell the story:

The right CAA

With no CAA record at all, any CA may issue and Google works. If you have CAA (or your organization requires it), add both CAs:

1
2

example.com.   CAA 0 issue "pki.goog"
example.com.   CAA 0 issue "letsencrypt.org"

Listing only one forces Google to always use that one; the documentation advises against it because a renewal may come out through the other. CAA records inherit downwards: a CAA on example.com covers www.example.com if the subdomain has none of its own.

1

dig +short CAA example.com

Certificate Manager: issue before switching DNS

With classic certificates (compute ssl-certificates) the domain must already point at the load balancer, which during a migration means a few minutes of broken HTTPS. Certificate Manager solves that with DNS authorization: you publish a validation CNAME and Google issues the certificate while traffic still flows to the old site.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

gcloud certificate-manager dns-authorizations create www-auth --domain=www.example.com
gcloud certificate-manager dns-authorizations describe www-auth --format='yaml(dnsResourceRecord)'
# dnsResourceRecord: name: _acme-challenge.www.example.com. type: CNAME data: xxxx.authorize.certificatemanager.goog.

# publish that CNAME in your zone and then:
gcloud certificate-manager certificates create www-cert \
  --domains=www.example.com --dns-authorizations=www-auth
gcloud certificate-manager maps create web-map
gcloud certificate-manager maps entries create www --map=web-map --certificate=www-cert --hostname=www.example.com
gcloud compute target-https-proxies update web-proxy --certificate-map=web-map

Once the certificate is ACTIVE, you flip the A/AAAA and the cutover is zero. CAA applies the same way.

Traps

• Cloudflare or another proxy in front of the load balancer: the A record points at the proxy, not at Google, and validation fails with FAILED_NOT_VISIBLE. Grey cloud during issuance, or Certificate Manager with DNS authorization.
• A managed certificate attached to nothing never gets issued. Creating it and waiting does nothing; attach it to the target proxy first.
• Changing the domain list creates a new certificate: attach the new one, wait for ACTIVE, and only then remove the old one from the proxy, or you get a gap.
• Wildcards only with Certificate Manager and DNS authorization; classic managed certificates do not support them.
• Badly signed DNSSEC breaks CAA lookups: FAILED_CAA_CHECKING even when you have no CAA at all.

Reference: Google-managed SSL certificates and Certificate Manager in the Google Cloud documentation.