TL;DR

Good disk ada0, new disk ada3, the dead one was ada1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

zpool status zroot                                  # ada1p4 FAULTED / UNAVAIL
gpart backup ada0 | gpart restore -F ada3           # clone the partition table
gpart show ada3                                     # p1 efi, p2 freebsd-boot, p3 freebsd-swap, p4 freebsd-zfs

# BIOS bootcode: the index is the freebsd-boot partition's
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada3

# UEFI: populate the ESP
newfs_msdos -F 32 -c 1 /dev/ada3p1
mount -t msdosfs /dev/ada3p1 /mnt
mkdir -p /mnt/EFI/BOOT /mnt/EFI/FREEBSD
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
cp /boot/loader.efi /mnt/EFI/FREEBSD/LOADER.EFI
umount /mnt

# The new disk joins the mirror in the dead one's place
zpool replace zroot ada1p4 ada3p4
zpool status zroot                                  # resilver in progress

When zpool status shows ONLINE on both, test booting with only the new disk.

Before starting

  • Identify the disk. zpool status names the dead member (ada1p4, or a long GUID if it is already gone). camcontrol devlist lists the disks with model and serial so you know which bay to touch.
  • If the bad disk is still connected, take it out of the pool before pulling the cable: zpool offline zroot ada1p4.
  • Layout. A default FreeBSD 13/14 zroot install creates, per disk: p1 efi, p2 freebsd-boot, p3 freebsd-swap, p4 freebsd-zfs. Old or hand-made installs vary: look at gpart show ada0 and adjust the indexes. The bootcode’s -i is freebsd-boot’s.
  • The new disk must be the same size or bigger. gpart restore copies exact sizes; if it is bigger, the rest stays free and you can use it later.

Why partitions and not the whole disk

In a boot pool every disk carries the boot partitions in addition to the ZFS one. If you run zpool replace zroot ada1p4 ada3 with the raw disk, the mirror works but that disk cannot boot: the day ada0 dies, you will have the data and no loader. That is why the order is: partition identically, bootcode, and only then zpool replace with the partition.

Step by step

1. Clone the partition table.

1
2

gpart backup ada0 | gpart restore -F ada3
gpart show ada3

-F destroys whatever was on ada3. If the pool uses GPT labels (gpt/zfs0 in zpool status) instead of adaXpY names, give the new one its own: gpart modify -i 4 -l zfs1 ada3, and use gpt/zfs1 in the zpool replace.

2. Bootcode. Install both: that way it boots whether the board is in BIOS/CSM mode or UEFI.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# BIOS: protective MBR + gptzfsboot on the freebsd-boot partition (p2 in the default layout)
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada3

# UEFI: the ESP (p1) is a FAT32 with loader.efi at the standard paths
newfs_msdos -F 32 -c 1 /dev/ada3p1
mount -t msdosfs /dev/ada3p1 /mnt
mkdir -p /mnt/EFI/BOOT /mnt/EFI/FREEBSD
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
cp /boot/loader.efi /mnt/EFI/FREEBSD/LOADER.EFI
umount /mnt

Old guides say gpart bootcode -p /boot/boot1.efifat -i 1; boot1.efi is deprecated on current FreeBSD and the path documented in loader.efi(8) is copying loader.efi onto the ESP. If you want a dedicated firmware entry: efibootmgr -c -l /mnt/EFI/FREEBSD/LOADER.EFI -L FreeBSD-ada3 -a.

3. Replace in the pool.

1

zpool replace zroot ada1p4 ada3p4

If the dead member no longer shows and the mirror is down to one disk, instead of replace you attach to the survivor: zpool attach zroot ada0p4 ada3p4.

4. Resilver.

1

zpool status zroot           # scan: resilver in progress ... % done

Hours, depending on how full the pool is. Optional, so the resilver yields less to normal load:

1

sysctl vfs.zfs.resilver_min_time_ms=5000

5. Swap. If /etc/fstab had /dev/ada1p3 as swap, change it to /dev/ada3p3 (or better, to gpt/swapN labels that do not change with the disk name) and swapon -a.

6. Check the boot. With the resilver done, power off, disconnect ada0 (or change the boot order) and boot from the new one. If it comes up, reconnect and zpool status must still read ONLINE with no complaints. A closing zpool scrub zroot does not hurt.

Traps

  • gpart restore -F does not ask. Double-check which disk is the new one; camcontrol devlist and diskinfo -v ada3 (serial) before writing.
  • Bootcode of a single kind leaves the disk bootable in only one firmware mode. Install both.
  • Different physical sector size (512 vs 4K) can make zpool replace complain about ashift; in a modern mirror with ashift=12 a 4Kn disk goes in fine, one with ashift=9 does not.
  • Bigger disk: after the resilver, gpart resize -i 4 ada3 and zpool online -e zroot ada3p4; once both mirror disks are big, the pool grows.
  • adaX names shift when disks are removed; GPT labels and GUIDs do not. For the operation use whatever zpool status shows at that moment.

See also