A LUKS server that hangs and never comes back: watchdog, SSH unlock with dropbear, and netconsole

TL;DR

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# 1. Hardware watchdog: if the kernel freezes, the chipset resets the machine
echo iTCO_wdt > /etc/modules-load.d/iTCO_wdt.conf
mkdir -p /etc/systemd/system.conf.d
printf '[Manager]\nRuntimeWatchdogSec=60s\nRebootWatchdogSec=5min\n' > /etc/systemd/system.conf.d/90-watchdog.conf
printf 'kernel.panic = 10\nkernel.softlockup_panic = 1\nkernel.hardlockup_panic = 1\nkernel.printk = 5 4 1 7\n' > /etc/sysctl.d/90-crash-recovery.conf

# 2. SSH in the initramfs to type the LUKS passphrase
apt install dropbear-initramfs
echo 'DROPBEAR_OPTIONS="-p 2222 -s -j -k -I 300 -c cryptroot-unlock"' > /etc/dropbear/initramfs/dropbear.conf
echo 'IP=192.168.1.20::192.168.1.1:255.255.255.0:server1:eno1:off' > /etc/initramfs-tools/conf.d/90-remote-unlock
# public keys in /etc/dropbear/initramfs/authorized_keys
update-initramfs -u

# 3. After a reboot, from the LAN
ssh -p 2222 root@192.168.1.20      # asks for the passphrase, boot continues

Plus netconsole to another machine to learn why it went down (below).

The problem

A host with a LUKS-encrypted root (passphrase only, no TPM) freezes mid-morning. No power cut, the journal stops mid-line, no errors. Nobody notices until someone walks over and powers it on by hand: six hours down.

Three things stretch the outage and erase the evidence:

Problem	Consequence	Fix
No watchdog	A frozen kernel waits for a human	Hardware watchdog petted by systemd
kdump active (Ubuntu default) with `kernel.panic=0`	The capture kernel needs the passphrase to write `/var/crash`, boots with no USB and no network, and waits forever	Turn kdump off, `kernel.panic=10`
Passphrase-only LUKS root	Any unattended reboot stops at the prompt	dropbear in the initramfs
A hard freeze leaves nothing in the local journal	No way to find the cause	netconsole to another machine

1. Hardware watchdog and reboot on panic

On Intel machines the chipset watchdog is iTCO_wdt. Ubuntu blacklists every watchdog driver, but that only stops automatic loading; loading it by name works:

1
2
3
echo iTCO_wdt > /etc/modules-load.d/iTCO_wdt.conf
modprobe iTCO_wdt
cat /sys/class/watchdog/watchdog0/identity      # iTCO_wdt

systemd pets it. If PID 1 stops petting for 60 s (frozen kernel), the chipset resets the machine. RebootWatchdogSec also bounds shutdowns that hang:

1
2
3
4
# /etc/systemd/system.conf.d/90-watchdog.conf
[Manager]
RuntimeWatchdogSec=60s
RebootWatchdogSec=5min

1
2
3
systemctl daemon-reexec
systemctl show -p WatchdogDevice -p RuntimeWatchdogUSec
# WatchdogDevice=/dev/watchdog0  RuntimeWatchdogUSec=1min

So that panics and lockups reboot instead of hanging:

1
2
3
4
5
# /etc/sysctl.d/90-crash-recovery.conf
kernel.panic = 10
kernel.softlockup_panic = 1
kernel.hardlockup_panic = 1
kernel.printk = 5 4 1 7

kernel.printk = 5 … raises the console level so warnings also go out over netconsole (step 3). hardlockup_panic needs the NMI watchdog, which is on by default.

kdump: with an encrypted root and no automatic unlock, its capture kernel cannot write the dump and sits at the passphrase prompt. Turn it off:

1
2
3
sed -i 's/^USE_KDUMP=.*/USE_KDUMP=0/' /etc/default/kdump-tools
systemctl disable --now kdump-tools
kdump-config unload

And get back the memory it reserves. kdump-tools.cfg adds crashkernel= from /etc/default/grub.d/, and those files are read in alphabetical order: a file that strips it has to sort after it (zz-, not 99-):

1
2
3
4
5
cat > /etc/default/grub.d/zz-no-crashkernel.cfg <<'EOF'
GRUB_CMDLINE_LINUX_DEFAULT="$(printf '%s' "$GRUB_CMDLINE_LINUX_DEFAULT" | sed -E 's/(^| )crashkernel=[^ ]*//g; s/^ +//')"
EOF
update-grub
grep -c crashkernel /boot/grub/grub.cfg     # 0

That way a package update cannot bring it back.

2. Unlock LUKS over SSH with dropbear

dropbear-initramfs puts a minimal SSH server in the initramfs, started before the disk is opened:

1
apt install dropbear-initramfs

Options in /etc/dropbear/initramfs/dropbear.conf (on older Debian/Ubuntu the file is /etc/dropbear-initramfs/config):

1
2
DROPBEAR_OPTIONS="-p 2222 -s -j -k -I 300 -c cryptroot-unlock"
IFDOWN="*"

Option	Why
`-p 2222`	A different port from the real sshd: the host key differs, so `known_hosts` does not clash
`-s`	No passwords, keys only
`-j -k`	No port forwarding
`-I 300`	Drops idle sessions after 5 min
`-c cryptroot-unlock`	Every login runs `cryptroot-unlock` and nothing else
`IFDOWN="*"`	When done, takes the NIC down and flushes it so the real system configures it from scratch (matters if a bridge sits on top)

Restricted public keys, one per line:

1
2
3
# /etc/dropbear/initramfs/authorized_keys  (mode 0600)
no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAA... laptop
no-port-forwarding,no-agent-forwarding,no-X11-forwarding ssh-ed25519 AAAA... phone

Initramfs network with a static IP, without touching GRUB (kernel ip= syntax: client::gateway:netmask:hostname:device:autoconf):

1
2
# /etc/initramfs-tools/conf.d/90-remote-unlock
IP=192.168.1.20::192.168.1.1:255.255.255.0:server1:eno1:off

Use the physical NIC. If the system has a bridge (br0), it does not exist yet in the initramfs: name the interface that is its uplink.

Rebuild and check that everything made it in:

1
2
update-initramfs -u
lsinitramfs /boot/initrd.img-$(uname -r) | grep -E 'sbin/dropbear$|cryptroot-unlock$|90-remote-unlock|authorized_keys'

Unlock

1
2
ssh -p 2222 root@192.168.1.20
# Please unlock disk dm_crypt-0:

Type the passphrase; the session closes by itself and the boot continues. The local keyboard still works in parallel: whichever answers first wins. The first time, compare the host key with the ones in /etc/dropbear/initramfs/dropbear_*_host_key:

1
dropbearkey -y -f /etc/dropbear/initramfs/dropbear_ed25519_host_key | grep -o 'SHA256:.*'

3. netconsole: make the crash log survive

A hard freeze gets nothing to disk. netconsole sends kernel messages over UDP to another machine from inside the kernel (netpoll), so they get out even when userspace is already dead.

On the host, through configfs and after the interface exists. The module option form (netconsole=… on the command line) runs before NetworkManager builds a bridge and is useless here:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
modprobe netconsole
mountpoint -q /sys/kernel/config || mount -t configfs none /sys/kernel/config
T=/sys/kernel/config/netconsole/log-host
mkdir $T
echo br0          > $T/dev_name
echo 192.168.1.20 > $T/local_ip
echo 192.168.1.30 > $T/remote_ip
echo 6666         > $T/remote_port
echo 1            > $T/enabled
echo "<4>netconsole: $(hostname) streaming" > /dev/kmsg

Put it in a oneshot service with After= and BindsTo= on sys-subsystem-net-devices-br0.device so it is set up on every boot. If the bridge is recreated while the system is up, the target is not re-armed by itself: restart the service.

On the other machine (physical, on the same segment: a VM on the same host goes down with it), a socat receiver that only accepts that host and writes to the journal:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
# /etc/systemd/system/netconsole-server1.service
[Unit]
Description=netconsole receiver for server1
After=network-online.target
Wants=network-online.target

[Service]
ExecStart=/usr/bin/socat -u UDP-RECV:6666,bind=192.168.1.30,range=192.168.1.20/32 STDOUT
StandardOutput=journal
SyslogIdentifier=netconsole-server1
Restart=always
DynamicUser=yes

[Install]
WantedBy=multi-user.target

1
journalctl -t netconsole-server1 --since "-12h"

The netconsole: … streaming line marks each boot; what comes before the last marker is the end of the previous boot. Look for Kernel panic, soft lockup, hard LOCKUP, NVRM: Xid, or PCIe/Thunderbolt AER errors.

Test it

1
2
3
4
cat /sys/class/watchdog/watchdog0/{identity,state,timeout}   # iTCO_wdt active 60
cat /sys/kernel/kexec_crash_loaded                           # 0: kdump off
echo "<4>netconsole test $(date +%T)" > /dev/kmsg            # shows up on the receiver
systemctl reboot                                             # then unlock over SSH

The full test, only with someone ready to unlock: echo c > /proc/sysrq-trigger triggers a panic. Its text should reach the receiver, the host reboots after 10 s and waits at the LUKS prompt.

Traps

The previous kernel has no dropbear. update-initramfs -u only rebuilds the running kernel’s initramfs. If you boot the previous kernel from GRUB (the usual way out when a new one fails), its initramfs may predate dropbear: only the keyboard works there. Check with lsinitramfs or rebuild it with update-initramfs -u -k <version>.
LAN only. There is no VPN or Tailscale in the initramfs. To unlock from outside, advertise the host’s IP (/32) from a subnet router that runs on another machine on the LAN.
“Host key changed”. Port 2222 has different keys from the normal sshd; the known_hosts entry is [192.168.1.20]:2222.
The wrong NIC. If the cable is on a USB adapter or a dock and the IP is on the onboard NIC, the initramfs has no network. At the local console, look for IP-Config: eno1 ….
netconsole does not show up in the host’s own tcpdump: netpoll bypasses the capture taps. Look on the receiver.
Only messages below console_loglevel go out: with 5, warnings and worse, never info or debug.
Watchdog false positives: if systemd goes 60 s without petting it, the box reboots and you type the passphrase again. For kernel debugging, remove the .conf and run systemctl daemon-reexec.
TPM2/clevis would unlock by itself, but also for anyone who walks off with the machine and powers it on. Here the passphrase over SSH won.

TL;DR#

The problem#

1. Hardware watchdog and reboot on panic#

2. Unlock LUKS over SSH with dropbear#

Unlock#

3. netconsole: make the crash log survive#

Test it#

Traps#

See also#