Good disk ada0, new disk ada3, the dead one was ada1:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

zpool status zroot                                  # ada1p4 FAULTED / UNAVAIL
gpart backup ada0 | gpart restore -F ada3           # clone the partition table
gpart show ada3                                     # p1 efi, p2 freebsd-boot, p3 freebsd-swap, p4 freebsd-zfs

# BIOS bootcode: the index is the freebsd-boot partition's
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada3

# UEFI: populate the ESP
newfs_msdos -F 32 -c 1 /dev/ada3p1
mount -t msdosfs /dev/ada3p1 /mnt
mkdir -p /mnt/EFI/BOOT /mnt/EFI/FREEBSD
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
cp /boot/loader.efi /mnt/EFI/FREEBSD/LOADER.EFI
umount /mnt

# The new disk joins the mirror in the dead one's place
zpool replace zroot ada1p4 ada3p4
zpool status zroot                                  # resilver in progress

When zpool status shows ONLINE on both, test booting with only the new disk.

Before starting

• Identify the disk. zpool status names the dead member (ada1p4, or a long GUID if it is already gone). camcontrol devlist lists the disks with model and serial so you know which bay to touch.
• If the bad disk is still connected, take it out of the pool before pulling the cable: zpool offline zroot ada1p4.
• Layout. A default FreeBSD 13/14 zroot install creates, per disk: p1 efi, p2 freebsd-boot, p3 freebsd-swap, p4 freebsd-zfs. Old or hand-made installs vary: look at gpart show ada0 and adjust the indexes. The bootcode’s -i is freebsd-boot’s.
• The new disk must be the same size or bigger. gpart restore copies exact sizes; if it is bigger, the rest stays free and you can use it later.

Why partitions and not the whole disk

In a boot pool every disk carries the boot partitions in addition to the ZFS one. If you run zpool replace zroot ada1p4 ada3 with the raw disk, the mirror works but that disk cannot boot: the day ada0 dies, you will have the data and no loader. That is why the order is: partition identically, bootcode, and only then zpool replace with the partition.

Step by step

1. Clone the partition table.

1
2

gpart backup ada0 | gpart restore -F ada3
gpart show ada3

-F destroys whatever was on ada3. If the pool uses GPT labels (gpt/zfs0 in zpool status) instead of adaXpY names, give the new one its own: gpart modify -i 4 -l zfs1 ada3, and use gpt/zfs1 in the zpool replace.

2. Bootcode. Install both: that way it boots whether the board is in BIOS/CSM mode or UEFI.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# BIOS: protective MBR + gptzfsboot on the freebsd-boot partition (p2 in the default layout)
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 2 ada3

# UEFI: the ESP (p1) is a FAT32 with loader.efi at the standard paths
newfs_msdos -F 32 -c 1 /dev/ada3p1
mount -t msdosfs /dev/ada3p1 /mnt
mkdir -p /mnt/EFI/BOOT /mnt/EFI/FREEBSD
cp /boot/loader.efi /mnt/EFI/BOOT/BOOTX64.EFI
cp /boot/loader.efi /mnt/EFI/FREEBSD/LOADER.EFI
umount /mnt

Old guides say gpart bootcode -p /boot/boot1.efifat -i 1; boot1.efi is deprecated on current FreeBSD and the path documented in loader.efi(8) is copying loader.efi onto the ESP. If you want a dedicated firmware entry: efibootmgr -c -l /mnt/EFI/FREEBSD/LOADER.EFI -L FreeBSD-ada3 -a.

3. Replace in the pool.

1

zpool replace zroot ada1p4 ada3p4

If the dead member no longer shows and the mirror is down to one disk, instead of replace you attach to the survivor: zpool attach zroot ada0p4 ada3p4.

4. Resilver.

1

zpool status zroot           # scan: resilver in progress ... % done

Hours, depending on how full the pool is. Optional, so the resilver yields less to normal load:

1

sysctl vfs.zfs.resilver_min_time_ms=5000

5. Swap. If /etc/fstab had /dev/ada1p3 as swap, change it to /dev/ada3p3 (or better, to gpt/swapN labels that do not change with the disk name) and swapon -a.

6. Check the boot. With the resilver done, power off, disconnect ada0 (or change the boot order) and boot from the new one. If it comes up, reconnect and zpool status must still read ONLINE with no complaints. A closing zpool scrub zroot does not hurt.

Traps

• gpart restore -F does not ask. Double-check which disk is the new one; camcontrol devlist and diskinfo -v ada3 (serial) before writing.
• Bootcode of a single kind leaves the disk bootable in only one firmware mode. Install both.
• Different physical sector size (512 vs 4K) can make zpool replace complain about ashift; in a modern mirror with ashift=12 a 4Kn disk goes in fine, one with ashift=9 does not.
• Bigger disk: after the resilver, gpart resize -i 4 ada3 and zpool online -e zroot ada3p4; once both mirror disks are big, the pool grows.
• adaX names shift when disks are removed; GPT labels and GUIDs do not. For the operation use whatever zpool status shows at that moment.

See also

• Replace a disk in a ZFS pool — the Linux/Proxmox version, without boot partitions.
• Import a pool after a power outage.
• FreeBSD: man 8 loader.efi, man 8 gpart, and Babak Farrokhi’s article on the BIOS replacement.

1
2
3
4
5
6
7
8
9

pfctl -sT                                    # loaded tables
pfctl -t <table> -T show                     # contents
pfctl -t <table> -T add 192.168.1.10         # add an IP or network (10.20.0.0/16 works too)
pfctl -t <table> -T delete 192.168.1.10
pfctl -t <table> -T test 192.168.1.10        # is it inside? (match / no match)
pfctl -t <table> -T flush                    # empty the table
pfctl -t <table> -T replace -f /path/list    # replace the whole contents from a file

pfctl -t sshguard -T flush                   # pfSense: clear the failed-login lockout

What a table is

A PF table is a set of addresses and networks that rules reference as <name>. It changes live, without reloading the ruleset, and holds thousands of entries with fast lookups; that is why PF uses tables for block lists, bogons, and anything that changes more often than the rules.

In pf.conf:

1
2

table <blocklist> persist file "/etc/pf/blocklist.txt"
block in quick from <blocklist>

persist keeps the table even when no rule uses it at the moment; file loads it from a file (one IP or network per line).

Commands

All in the form pfctl -t <table> -T <command>:

And without -t:

1
2
3
4

pfctl -sT                    # names of all tables
pfctl -vvsT                  # global info per table: entries, evaluations, references
pfctl -T load -f /etc/pf.conf   # reload only the table definitions from pf.conf
pfctl -sr | grep '<name>'    # which rules use a table

Tables on pfSense

• Aliases are tables. A Host, Network, or URL Table alias called Admins exists in PF as <Admins>; pfctl -t Admins -T show shows what pfSense resolved, including the FQDNs it already turned into IPs.
• sshguard: where IPs blocked for failed SSH or webGUI logins land. If you locked yourself out, pfctl -t sshguard -T flush (documented by Netgate under Locked out of the WebGUI).
• virusprot: hosts blocked for exceeding a rule’s Max. src. conn. rate or Max. states limits. pfctl -t virusprot -T show and -T delete <ip> to release one.
• snort2c: Snort/Suricata blocks in legacy mode. -T flush to unblock everything.
• bogons / bogonsv6: the lists that update themselves when you enable Block bogon networks on an interface.
• pfBlockerNG: every list is a pfB_* alias. To know whether an IP is blocked by one: pfctl -t pfB_Top_v4 -T test 203.0.113.7.

On OPNsense the mechanics are the same (pfctl -sT to see the names); the login lockout table is called sshlockout.

What gets lost on reload

Any -T add, -T delete, or -T flush lives until the next filter reload: saving almost anything in the GUI, Filter Reload under Status → Filter Reload, or the alias update cron. At that point pfSense regenerates every table from its alias and your manual changes are gone. To make something permanent, edit the alias; for quick tests and unblocking, pfctl is perfect.

Beyond tables

What gets used most alongside the above:

1
2
3
4
5
6
7

pfctl -sr                    # loaded rules (with -vv, counters and rule numbers)
pfctl -ss                    # state table
pfctl -si                    # global statistics
pfctl -k 192.168.1.10        # kill a host's states
pfctl -F states              # kill all states (cuts every connection)
pfctl -d / pfctl -e          # disable / enable PF (no PF means no NAT: only to get out of a lockout)
pfctl -nf /etc/pf.conf       # syntax check without loading

Full reference: the OpenBSD Handbook’s pfctl cheat sheet and man pfctl on FreeBSD.

From the console (option 8) or Diagnostics → Command Prompt:

1
2
3
4
5
6

certctl rehash
pkg-static clean -ay
pkg-static install -fy pkg
pkg-static install -xfy pfSense-repo pfSense-upgrade
pkg-static update -f
pfSense-upgrade -d -c        # shows which version the configured branch offers

With pkg alive again, finish the pending upgrade through System → Update (or pfSense-upgrade -d), not with pkg upgrade.

The error

1

ld-elf.so.1: Shared object "libssl.so.30" not found, required by "pkg"

You see it on pkg update, when installing a package, or indirectly: System → Package Manager shows up empty and System → Update cannot check versions. Sometimes the binary complaining is pfSense-repoc instead of pkg; it is the same problem.

Why it happens

pfSense is FreeBSD, and pkg is a binary dynamically linked against the system’s libssl. When the configured repository belongs to a pfSense release built on a newer FreeBSD, its packages come linked against a newer OpenSSL (libssl.so.30 = OpenSSL 3). If pkg, or pfSense-repoc, gets updated from that repo before the base system, the new binary asks for a library that does not exist on your install yet.

How you get there:

• A manual pkg upgrade from the shell. On pfSense you never do that.
• Installing or reinstalling a package after switching the Branch in System → Update to a higher version, without upgrading.
• An upgrade interrupted halfway (typical in the 2.7.0 → 2.7.1/2.7.2 jump, when the base moved to OpenSSL 3).
• On pfSense Plus, rolling back to an older boot environment with the repo pointing at the new one (Redmine #15070).

The library name will change across releases; the mechanics and the fix stay the same.

Fix

pkg-static is the statically linked copy of pkg: it needs no libssl, which is why it works when pkg does not.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# 1. Let pkg-static talk TLS to the repo (holds until the next reboot)
certctl rehash

# 2. Drop the cache and reinstall pkg and the repo configuration
pkg-static clean -ay
pkg-static install -fy pkg
pkg-static install -xfy pfSense-repo pfSense-upgrade   # -x: regex, so pfSense-repoc comes along

# 3. Refresh metadata and check
pkg-static update -f
pfSense-upgrade -d -c

If pkg-static update still fails, pkg-static -d update prints the full trace (certificates, DNS, wrong branch).

Before step 3 check the Branch under System → Update: it must be the version you want to reach, not an intermediate one. And finish the job through the GUI or with pfSense-upgrade -d: a stray pkg upgrade leaves you halfway again.

If the base system is so mixed up that even this does not fly, Netgate’s documented last resort is pkg-static upgrade -f (reinstalls everything), followed by a reboot.

Traps

• pkg upgrade does not exist on pfSense. The only supported path is System → Update / pfSense-upgrade. This whole situation is born from skipping it.
• Backup first: Diagnostics → Backup & Restore. On ZFS, also a boot environment (System → Boot Environments) lets you roll back in one reboot.
• Do not copy libssl.so.30 from another FreeBSD. It circulates as a trick; it leaves the system with libraries from two releases and breaks the next upgrade.
• Packages first, no. With an upgrade pending, upgrade the system and then install packages, not the other way around.
• The repo and the upgrade use HTTPS: if certctl rehash is not enough, check the firewall’s system time and DNS.

Reference: Upgrade Troubleshooting in the Netgate documentation.