Good disk ada0, new disk ada3, the dead one was ada1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

zpool status zroot                                  # ada1p4 FAULTED / UNAVAIL
gpart backup ada0 | gpart restore -F ada3           # clone the partition table
gpart show ada3                                     # p1 efi, p2 freebsd-boot, p3 freebsd-swap, p4 freebsd-zfs

# BIOS bootcode: the index is the freebsd-boot partition's
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada3

# UEFI: populate the ESP
newfs_msdos -F 32 -c 1 /dev/ada3p1
mount -t msdosfs /dev/ada3p1 /mnt
mkdir -p /mnt/EFI/BOOT /mnt/EFI/FREEBSD
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
cp /boot/loader.efi /mnt/EFI/FREEBSD/LOADER.EFI
umount /mnt

# The new disk joins the mirror in the dead one's place
zpool replace zroot ada1p4 ada3p4
zpool status zroot                                  # resilver in progress

When zpool status shows ONLINE on both, test booting with only the new disk.

Before starting

• Identify the disk. zpool status names the dead member (ada1p4, or a long GUID if it is already gone). camcontrol devlist lists the disks with model and serial so you know which bay to touch.
• If the bad disk is still connected, take it out of the pool before pulling the cable: zpool offline zroot ada1p4.
• Layout. A default FreeBSD 13/14 zroot install creates, per disk: p1 efi, p2 freebsd-boot, p3 freebsd-swap, p4 freebsd-zfs. Old or hand-made installs vary: look at gpart show ada0 and adjust the indexes. The bootcode’s -i is freebsd-boot’s.
• The new disk must be the same size or bigger. gpart restore copies exact sizes; if it is bigger, the rest stays free and you can use it later.

Why partitions and not the whole disk

In a boot pool every disk carries the boot partitions in addition to the ZFS one. If you run zpool replace zroot ada1p4 ada3 with the raw disk, the mirror works but that disk cannot boot: the day ada0 dies, you will have the data and no loader. That is why the order is: partition identically, bootcode, and only then zpool replace with the partition.

Step by step

1. Clone the partition table.

1
2

gpart backup ada0 | gpart restore -F ada3
gpart show ada3

-F destroys whatever was on ada3. If the pool uses GPT labels (gpt/zfs0 in zpool status) instead of adaXpY names, give the new one its own: gpart modify -i 4 -l zfs1 ada3, and use gpt/zfs1 in the zpool replace.

2. Bootcode. Install both: that way it boots whether the board is in BIOS/CSM mode or UEFI.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# BIOS: protective MBR + gptzfsboot on the freebsd-boot partition (p2 in the default layout)
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada3

# UEFI: the ESP (p1) is a FAT32 with loader.efi at the standard paths
newfs_msdos -F 32 -c 1 /dev/ada3p1
mount -t msdosfs /dev/ada3p1 /mnt
mkdir -p /mnt/EFI/BOOT /mnt/EFI/FREEBSD
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
cp /boot/loader.efi /mnt/EFI/FREEBSD/LOADER.EFI
umount /mnt

Old guides say gpart bootcode -p /boot/boot1.efifat -i 1; boot1.efi is deprecated on current FreeBSD and the path documented in loader.efi(8) is copying loader.efi onto the ESP. If you want a dedicated firmware entry: efibootmgr -c -l /mnt/EFI/FREEBSD/LOADER.EFI -L FreeBSD-ada3 -a.

3. Replace in the pool.

1

zpool replace zroot ada1p4 ada3p4

If the dead member no longer shows and the mirror is down to one disk, instead of replace you attach to the survivor: zpool attach zroot ada0p4 ada3p4.

4. Resilver.

1

zpool status zroot           # scan: resilver in progress ... % done

Hours, depending on how full the pool is. Optional, so the resilver yields less to normal load:

1

sysctl vfs.zfs.resilver_min_time_ms=5000

5. Swap. If /etc/fstab had /dev/ada1p3 as swap, change it to /dev/ada3p3 (or better, to gpt/swapN labels that do not change with the disk name) and swapon -a.

6. Check the boot. With the resilver done, power off, disconnect ada0 (or change the boot order) and boot from the new one. If it comes up, reconnect and zpool status must still read ONLINE with no complaints. A closing zpool scrub zroot does not hurt.

Traps

• gpart restore -F does not ask. Double-check which disk is the new one; camcontrol devlist and diskinfo -v ada3 (serial) before writing.
• Bootcode of a single kind leaves the disk bootable in only one firmware mode. Install both.
• Different physical sector size (512 vs 4K) can make zpool replace complain about ashift; in a modern mirror with ashift=12 a 4Kn disk goes in fine, one with ashift=9 does not.
• Bigger disk: after the resilver, gpart resize -i 4 ada3 and zpool online -e zroot ada3p4; once both mirror disks are big, the pool grows.
• adaX names shift when disks are removed; GPT labels and GUIDs do not. For the operation use whatever zpool status shows at that moment.

See also

• Replace a disk in a ZFS pool — the Linux/Proxmox version, without boot partitions.
• Import a pool after a power outage.
• FreeBSD: man 8 loader.efi, man 8 gpart, and Babak Farrokhi’s article on the BIOS replacement.

Boot the Proxmox ISO → Advanced Options → Install Proxmox VE (Debug mode); at the first shell press Ctrl+D to reach the second one, which already has LVM:

1
2
3
4

vgscan
vgchange -ay
lvcreate -L 4M pve -n grubtemp   # any change to the LVM metadata will do
reboot

With the node back up, clean up and make the fix permanent:

1
2
3

lvremove pve/grubtemp
[ -d /sys/firmware/efi ] && apt install grub-efi-amd64     # UEFI
# legacy BIOS: grub-install /dev/sdX && update-grub        # the disk, not the partition

The error

1
2
3

error: disk 'lvmid/Xxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxxx/Yyyyyy-yyyy-yyyy-yyyy-yyyy-yyyy-yyyyyy' not found.
Entering rescue mode...
grub rescue>

The disk is fine, the pve VG is fine, and the data is fine. Sometimes the same bug warns you earlier, during an apt full-upgrade:

1

/usr/sbin/grub-probe: error: disk `lvmid/...' not found.

If you see that in update-grub, apply the fix before rebooting.

Why it happens

LVM stores its on-disk metadata in a ring buffer. Every change (creating, deleting, or resizing an LV) writes a new copy further along the buffer, and sooner or later the text wraps around the end. The GRUB 2.06 in Debian 11 (PVE 7, 2.06-3) cannot read that wrap: it does not find the VG, does not find the pve/root LV, and drops to grub rescue.

That is why “fixing” it is as odd as making any LVM change: the metadata gets rewritten at another position, usually without a wrap, and GRUB reads it again. And that is why it comes back weeks or months later, when the buffer wraps again, if the GRUB installed on the disk is still the old one.

In the lab the change that got us out was lvextend -L +1G /dev/pve/root. It works just the same, but undoing it with lvreduce on a mounted root is playing with fire; if you did it that way, better keep the 1G (resize2fs /dev/pve/root) or use the throwaway LV from the TL;DR.

Full procedure

1. Get into the system. Two ways from the Proxmox ISO:

   • Rescue Boot (Advanced Options): the ISO finds the installation and boots its kernel, bypassing the broken GRUB. If it works you are already inside the real system and can jump to step 3.
   • Debug mode: rescue shell. Ctrl+D at the first prompt; the second one has the LVM modules loaded.

2. Trigger a metadata change (only if you came in through Debug mode):

   1 2 3 4

   vgscan vgchange -ay lvcreate -L 4M pve -n grubtemp reboot

3. Back in the normal system, remove the temporary LV and check whether the metadata still wraps:

   1 2	lvremove pve/grubtemp vgscan -vvv 2>&1 | grep 'Reading metadata'

   Output ending in (+0) means no wrap; any other value means the problem will repeat until you reinstall GRUB.

4. Reinstall the GRUB that has the fix. The PVE 8 and 9 package (2.06-13 or newer) already carries it, but the binary sitting in the ESP or the MBR is only rewritten when the package gets (re)installed:

   1 2 3 4 5 6 7 8

   # UEFI: nodes upgraded from PVE 7 often have grub-pc instead of grub-efi-amd64 dpkg -l | grep -E '^ii +grub-' apt install grub-efi-amd64 efibootmgr -v # confirm the "proxmox" entry # legacy BIOS grub-install /dev/sdX update-grub

   On PVE 7 the real fix is upgrading to PVE 8; the workaround only buys time.

Traps

• grub rescue> will not get you out of this. ls (lvm/pve-root) shows nothing because the lvm module is exactly what cannot read the metadata. Do not waste time there.
• Root on ZFS does not go through here. Nodes with rpool boot via proxmox-boot-tool; that is another path with other errors.
• Point at the right devices if you run grub-install by hand: the disk (/dev/sda), not /dev/sda1; the ESP mounted on /boot/efi for UEFI.
• In a cluster, once one node is fixed check the others with vgscan -vvv | grep 'Reading metadata'; same installation, same buffer.

References: Recover From Grub Failure on the Proxmox wiki and the forum thread.