1. Delegate a small zone to the machine that will run acme-dns (records in your real DNS):

   1 2	auth.example.com. NS acme.example.com. acme.example.com. A 203.0.113.10

2. acme-dns on that machine (/etc/acme-dns/config.cfg), DNS on 53 and the API on 443:

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

   [general] listen = "0.0.0.0:53" protocol = "both" domain = "auth.example.com" nsname = "acme.example.com" nsadmin = "hostmaster.example.com" records = [ "auth.example.com. A 203.0.113.10", "auth.example.com. NS acme.example.com.", ] [api] ip = "0.0.0.0" port = "443" tls = "letsencrypt" disable_registration = false

3. Register a client and create the CNAME:

   1 2 3

   curl -s -X POST https://acme.example.com/register -H 'Content-Type: application/json' \ -d '{"allowfrom": ["203.0.113.0/24"]}' # {"username":"eabcdb41-…","password":"pbAX…","fulldomain":"d420c923-….auth.example.com","subdomain":"d420c923-…","allowfrom":["203.0.113.0/24"]}

   1	_acme-challenge.example.com. CNAME d420c923-….auth.example.com.

4. certbot with the hook:

   1 2 3 4 5

   curl -fsSL -o /etc/letsencrypt/acme-dns-auth.py https://raw.githubusercontent.com/joohoi/acme-dns-certbot-joohoi/master/acme-dns-auth.py chmod +x /etc/letsencrypt/acme-dns-auth.py sed -i 's|^#!/usr/bin/env python$|#!/usr/bin/env python3|; s|^ACMEDNS_URL = .*|ACMEDNS_URL = "https://acme.example.com"|' /etc/letsencrypt/acme-dns-auth.py certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py \ --preferred-challenges dns --debug-challenges -d '*.example.com' -d example.com

   The first time, the hook registers itself, stores the credentials in /etc/letsencrypt/acmedns.json and pauses so you can create the CNAME it prints; Enter, and it validates. Renewals (certbot renew) never ask again.

Why

DNS-01 is the only challenge that gives wildcards and the only one that works for machines with no public port 80. The usual price is an API token from your DNS provider on every server that requests certificates; that token can typically change any record in the zone, and it sits in plain text under /etc/letsencrypt.

acme-dns turns that around: it is an authoritative DNS server for a zone of its own (auth.example.com) that only serves TXT records under random subdomains, with an API to update them. Since _acme-challenge.example.com is a CNAME to one of those subdomains, Let’s Encrypt follows the chain and validates the TXT there. Each client’s credentials only work for its subdomain; if stolen, the worst case is someone requesting a certificate for that name, not rewriting your zone.

cert-manager

The same scheme, with the credentials in a Secret:

1

kubectl -n cert-manager create secret generic acme-dns --from-file=acmedns.json=./acmedns.json

1
2
3
4
5
6
7
8
9

{
  "example.com": {
    "username": "eabcdb41-d89f-4580-826f-3e62e9755ef2",
    "password": "<password from the registration>",
    "fulldomain": "d420c923-bbd7-4056-ab64-c3ca54c9b3cf.auth.example.com",
    "subdomain": "d420c923-bbd7-4056-ab64-c3ca54c9b3cf",
    "allowfrom": ["203.0.113.0/24"]
  }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-dns
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: admin@example.com
    privateKeySecretRef:
      name: letsencrypt-dns-account-key
    solvers:
    - dns01:
        acmeDNS:
          host: https://acme.example.com
          accountSecretRef:
            name: acme-dns
            key: acmedns.json

The JSON key is the certificate’s domain; *.example.com and example.com share the same entry (the _acme-challenge.example.com CNAME covers both). Every distinct domain needs its own registration and CNAME.

Traps

• allowfrom with the IP acme-dns sees: the public egress IP of the server or the cluster, not the pod’s. If /update fails with 401, it is this or the password.
• CNAME TTL: keep it low (60–300 s) the first time; with a one-hour TTL, a mistake in the CNAME takes an hour to correct in Let’s Encrypt’s eyes. The TXT is served by acme-dns with TTL 1.
• A real port 53: acme-dns must be reachable over UDP and TCP from the Internet, and be the NS for auth.example.com in your parent zone. Without the NS record there is no delegation and validation returns NXDOMAIN.
• One registration per domain, not per server: if three servers request certificates for example.com, they share the same registration and the same acmedns.json, or each registers its own and the CNAME can only point to one. Share the credentials.
• --manual renews fine because the hook is stored in /etc/letsencrypt/renewal/<domain>.conf; what you cannot do is change the hook without running certonly again.
• Version: acme-dns 2.0.2 is from February 2025 and the project is still active; the certbot hook has not changed in years and works with Python 3 after fixing the shebang.

References: acme-dns, acme-dns-certbot-joohoi and ACMEDNS in cert-manager.

See also

• cert-manager: the HTTP-01 challenge returns 308 — when HTTP-01 is enough, and when it is not.
• HAProxy on pfSense with ACME and Cloudflare — DNS-01 with the provider’s token, the case acme-dns avoids.
• 2048-bit DKIM in a split TXT — another DNS record that breaks over details.

1
2

example.com.   CAA 0 issue "pki.goog"
example.com.   CAA 0 issue "letsencrypt.org"

1
2
3
4
5

gcloud compute ssl-certificates describe web-cert --global --format='yaml(managed)'
# managed:
#   domainStatus:
#     www.example.com: PROVISIONING | ACTIVE | FAILED_NOT_VISIBLE | FAILED_CAA_FORBIDDEN | FAILED_CAA_CHECKING | FAILED_RATE_LIMITED
#   status: PROVISIONING | ACTIVE

To reach ACTIVE three things must hold at once: the domain’s A/AAAA points to the load balancer IP (and nothing else), the certificate is attached to a target proxy with its forwarding rule, and no CAA in the zone forbids Google’s CAs. Then, up to 60 minutes.

Why

Google issues managed certificates through Google Trust Services (pki.goog) or Let’s Encrypt, and picks one or the other on each issuance or renewal. To validate the domain it runs an HTTP check against the IP the name resolves to, so until DNS points at the load balancer there is nothing to validate. And like any CA, it honors CAA records: if your zone says only digicert.com may issue (a typical corporate policy), Google cannot.

The statuses tell the story:

The right CAA

With no CAA record at all, any CA may issue and Google works. If you have CAA (or your organization requires it), add both CAs:

1
2

example.com.   CAA 0 issue "pki.goog"
example.com.   CAA 0 issue "letsencrypt.org"

Listing only one forces Google to always use that one; the documentation advises against it because a renewal may come out through the other. CAA records inherit downwards: a CAA on example.com covers www.example.com if the subdomain has none of its own.

1

dig +short CAA example.com

Certificate Manager: issue before switching DNS

With classic certificates (compute ssl-certificates) the domain must already point at the load balancer, which during a migration means a few minutes of broken HTTPS. Certificate Manager solves that with DNS authorization: you publish a validation CNAME and Google issues the certificate while traffic still flows to the old site.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

gcloud certificate-manager dns-authorizations create www-auth --domain=www.example.com
gcloud certificate-manager dns-authorizations describe www-auth --format='yaml(dnsResourceRecord)'
# dnsResourceRecord: name: _acme-challenge.www.example.com. type: CNAME data: xxxx.authorize.certificatemanager.goog.

# publish that CNAME in your zone and then:
gcloud certificate-manager certificates create www-cert \
  --domains=www.example.com --dns-authorizations=www-auth
gcloud certificate-manager maps create web-map
gcloud certificate-manager maps entries create www --map=web-map --certificate=www-cert --hostname=www.example.com
gcloud compute target-https-proxies update web-proxy --certificate-map=web-map

Once the certificate is ACTIVE, you flip the A/AAAA and the cutover is zero. CAA applies the same way.

Traps

• Cloudflare or another proxy in front of the load balancer: the A record points at the proxy, not at Google, and validation fails with FAILED_NOT_VISIBLE. Grey cloud during issuance, or Certificate Manager with DNS authorization.
• A managed certificate attached to nothing never gets issued. Creating it and waiting does nothing; attach it to the target proxy first.
• Changing the domain list creates a new certificate: attach the new one, wait for ACTIVE, and only then remove the old one from the proxy, or you get a gap.
• Wildcards only with Certificate Manager and DNS authorization; classic managed certificates do not support them.
• Badly signed DNSSEC breaks CAA lookups: FAILED_CAA_CHECKING even when you have no CAA at all.

Reference: Google-managed SSL certificates and Certificate Manager in the Google Cloud documentation.

1
2

PUB=$(openssl rsa -in dkim.key -pubout -outform DER 2>/dev/null | base64 | tr -d '\n')
echo "v=DKIM1; k=rsa; p=$PUB" | fold -w 255 | sed 's/.*/"&"/' | paste -sd' ' -

That prints the value ready to paste into the DNS provider, two quoted strings separated by a space:

1

"v=DKIM1; k=rsa; p=MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAvs7EROOYh35C/1s7M//pIaRvu1XWXzdexgBR2dO7eq1oDw+X08BNJUVUEte3z7os5CNeP7YfZrAqyDJvA59wMK1qul6oHyy7z5P1aB4zNczvO7SZ20HogWjHjzYJ0qwmSUUtjlKvECQzoTCbS/F0gB9rdUZN02MJopDs8YWcohy8oqNLnrflHdxWm6jAb3sSa" "EQReu6iRKKPiXTGUISSetY+iXLo2j/2c5g5ppiIYkUTq58asamMNiCzB7BeF4WNkjz2P+9IJmmtkkJ0M/U4oThUMvBBtQ/XUV4i2N4PGB359G5d/PIMiCj4go2axA4p8ul7Qafbtv0fyPwOIGSojwIDAQAB"

Both go into the same TXT record. Whoever verifies DKIM concatenates them with no spaces. If Google Workspace or Microsoft 365 handed you the text and you do not have the private key, it is the same without the openssl: echo "<value>" | fold -w 255 | sed 's/.*/"&"/' | paste -sd' ' -.

The problem

A TXT record is made of one or more character-strings, and each one is capped at 255 bytes (RFC 1035). The whole record can be much longer; what it cannot have is a single 410-character string.

Google Workspace and Microsoft 365 generate 2048-bit keys by default, and rightly so: 1024 has been discouraged since RFC 8301. So almost any new DKIM setup today runs into this, and each provider reacts differently:

RFC 6376 (§3.6.2.2) requires verifiers to concatenate the strings before using them, so splitting the key is not a trick: it is the intended format.

Per provider

BIND / zone file

1
2

selector._domainkey  300  IN  TXT  ( "v=DKIM1; k=rsa; p=MIIBIjANBgkq...first part"
                                     "second part...IDAQAB" )

Route 53

In the console, paste both quoted strings into Value separated by a space or a line break. From the CLI the value is a single string with escaped quotes:

1
2
3
4
5

aws route53 change-resource-record-sets --hosted-zone-id Z0123456789EXAMPLE --change-batch '{
  "Changes": [{"Action": "UPSERT", "ResourceRecordSet": {
    "Name": "selector._domainkey.example.com", "Type": "TXT", "TTL": 300,
    "ResourceRecords": [{"Value": "\"v=DKIM1; k=rsa; p=...first part\" \"second part...\""}]
  }}]}'

Google Cloud DNS

In the console, same format in the data field. With gcloud:

1
2
3

gcloud dns record-sets create selector._domainkey.example.com. --zone=example-zone \
  --type=TXT --ttl=300 \
  --rrdatas='"v=DKIM1; k=rsa; p=...first part" "second part..."'

With Terraform (google_dns_record_set), a single rrdatas element with escaped quotes:

1

rrdatas = ["\"v=DKIM1; k=rsa; p=...first part\" \"second part...\""]

Cloudflare

Paste the full value into Content; the dashboard splits it on save. When you query it with dig you will see two strings even though you entered one: that is normal.

Verifying

1

dig +short TXT selector._domainkey.example.com

Expected: "v=DKIM1; k=rsa; p=MIIB..." "...IDAQAB", two strings, one record. Then check that the key decodes to 2048 bits, which confirms no character was lost at the cut:

1
2
3
4

dig +short TXT selector._domainkey.example.com \
  | tr -d '" \n' | sed -E 's/.*p=([A-Za-z0-9+/=]+).*/\1/' \
  | base64 -d | openssl pkey -pubin -inform DER -noout -text | head -1
# Public-Key: (2048 bit)

If base64 complains, a space slipped inside the key or characters are missing. Finish with a test email and Authentication-Results: ... dkim=pass in the headers; for a closer look, dmarcian’s DKIM Inspector.

Traps

• The cut can go anywhere, even mid-base64, but never put spaces inside the key. The space between the two strings separates strings; it is not data.
• Do not double the quotes. If the panel adds them for you and you paste "...", dig returns "\"v=DKIM1...\"" and the verifier fails.
• One record, not two. Two separate TXT records with the same name are two independent values, and neither is a valid key.
• 2048 is the sweet spot. 4096 means three strings and a DNS answer close to the UDP limit; it works, but gains nothing for DKIM.
• Rotation: publish the new key under a new selector, switch the signer, and delete the old selector after the TTL. Never edit the selector in use.
• SPF has the same limit, but an SPF that does not fit in 255 is usually a sign of too many include:; there the fix is shortening it, not splitting it.

References: Google Workspace — troubleshoot DKIM issues, AWS re:Post — CharacterStringTooLong, and the Mailhardener DNS record splitter if you would rather not use fold.

Clean option (split DNS): Services → DNS Resolver → Host Overrides → app.example.com → the server’s internal IP (192.168.1.50). LAN clients must use the firewall as their DNS.

NAT reflection option: System → Advanced → Firewall & NAT:

• NAT Reflection mode for port forwards: Pure NAT
• Enable NAT Reflection for 1:1 NAT: ✔
• Enable automatic outbound NAT for Reflection: ✔

Save and test from a LAN client against the public IP.

The problem

The port forward WAN:443 → 192.168.1.50:443 works from the Internet. From a machine on the same LAN, https://app.example.com (which resolves to the public IP) answers with the pfSense GUI, a certificate error, or Potential DNS Rebind attack detected.

The packet leaves the client towards the public IP, reaches the firewall through the LAN interface, and stops there: the port forward rule listens on the WAN, not the LAN, and the destination IP is one of the firewall’s own addresses. It never “goes out and comes back in”.

Option 1: split DNS

This is what Netgate recommends: the name resolves to the internal IP inside and to the public one outside. Traffic goes direct, the server sees the client’s real IP, and the firewall touches nothing.

1. Services → DNS Resolver → Host Overrides → Add: host app, domain example.com, IP 192.168.1.50. If the service has several names, add Additional Names.
2. Clients must use the firewall as DNS (the norm if DHCP hands it out). A client with 8.8.8.8 hardcoded will keep going to the public IP.
3. nslookup app.example.com from the LAN → 192.168.1.50.

Requirement: the internal and external ports must be the same (if outside you forward 8443 → 443, the internal client will have to use 443 directly).

If you run the DNS Forwarder (dnsmasq) instead of the Resolver, the equivalent setting is under Services → DNS Forwarder → Host Overrides.

Option 2: NAT reflection

When you do not control the clients’ DNS, or there are lots of ports, the firewall can “reflect” the connection: it rewrites the destination to the internal IP and, with automatic outbound NAT, the source too, so the reply comes back through the firewall.

System → Advanced → Firewall & NAT, Network Address Translation section:

It can also be enabled per rule: Firewall → NAT → Port Forward → edit → NAT reflection → Enable (Pure NAT), leaving the global setting on Disable.

Costs: the server sees the firewall’s LAN IP as the source (not the client’s), and all that traffic crosses the firewall even when client and server sit on the same switch.

OPNsense

Same idea, other menus:

• Split DNS: Services → Unbound DNS → Overrides → Host Overrides.
• Reflection: Firewall → Settings → Advanced → Reflection for port forwards, Reflection for 1:1, Automatic outbound NAT for Reflection.

Traps

• DNS rebinding. If instead of a host override you point the Resolver at an external DNS that returns private IPs (an internal DNS at another site, a provider with views), Unbound drops those answers as rebinding protection. Do not untick Disable DNS Rebinding Checks under System → Advanced → Admin Access: add private-domain: "example.com" under Services → DNS Resolver → Custom options and only that domain is exempt.
• A client with hardcoded DNS bypasses split DNS. Either force it through DHCP, catch it with a port 53 NAT rule, or use reflection.
• Pure NAT needs to know the interface and gateway when rules load; with dynamic WANs that change IP, a Filter Reload after the IP change fixes it.
• Different ports inside and outside only work with reflection; split DNS does not translate ports.
• Certificates: with split DNS the browser still sees app.example.com, so the server’s certificate works as is. When reflecting through the public IP, too. The certificate error at the start was the firewall GUI’s, not the server’s.

References: Port Forwards from Local Networks and NAT Reflection in the Netgate documentation.