ZVOL = block device: dd if=/dev/zvol/pool/vol of=vm.raw. Snapshot first if the VM is running.

On Proxmox a VM disk on ZFS is a ZVOL (rpool/data/vm-101-disk-0), not a file. To copy it to another hypervisor or a USB stick, dump a RAW.

Thread: Import/convert/export RAW images to ZFS volume.

Snapshot and dd

If the VM is running, freeze the disk with a snapshot and read the snap (the @snap ZVOL shows up under /dev/zvol/…):

1
2
3

zfs snapshot rpool/data/vm-101-disk-0@export
dd if=/dev/zvol/rpool/data/vm-101-disk-0@export of=/mnt/backup/vm-101.raw bs=1M status=progress
zfs destroy rpool/data/vm-101-disk-0@export

VM off, you can read the live zvol:

1

dd if=/dev/zvol/rpool/data/vm-101-disk-0 of=/mnt/backup/vm-101.raw bs=1M status=progress

bs=1M matters: the 512 B default takes forever.

qemu-img (qcow2 / vmdk)

1

qemu-img convert -p -f raw -O qcow2 /mnt/backup/vm-101.raw /mnt/backup/vm-101.qcow2

Or straight from the zvol:

1

qemu-img convert -p -f raw -O qcow2 /dev/zvol/rpool/data/vm-101-disk-0 vm-101.qcow2

The other way (RAW → ZVOL)

1
2

zfs create -s -V 32G rpool/data/vm-101-disk-0
dd if=file.raw of=/dev/zvol/rpool/data/vm-101-disk-0 bs=1M status=progress

-s = sparse. The RAW must not be larger than the zvol.

See also: basic commands.

Pool will not import after power loss: zpool import -f, then -F if you must rewind txgs, then zpool scrub.

After a power cut the pool is often still on the disks but zpool status is empty: it was left dirty-exported, or the host booted without it. Do not zpool create on those disks.

See what is there

1

zpool import

Lists importable pools by name (VOL1 below) and whether they are FAULTED / UNAVAIL.

Import

1

zpool import -f VOL1

-f means “I know it might still look imported elsewhere”. That is the usual crash case: ZFS thinks the other side still holds it.

If -f is not enough because the last txgs were half-written:

1

zpool import -F VOL1

-F rewinds to an earlier consistent txg. You can lose the last seconds of writes. That is better than a pool that will not mount.

Dry-run:

1

zpool import -F -n VOL1

Afterwards: scrub

1
2

zpool scrub VOL1
zpool status VOL1

Scrub confirms the rewind did not leave checksum errors. Let it finish before you delete “just in case” snapshots.

Do not

• zpool create on the same sdX devices wipes the labels.
• -F is not the first knob: try -f first.
• If the pool is the root (rpool / Proxmox bpool), import from live media, not from the system that will not boot.

See also: replacing disks, basic commands.

zfs create -o encryption=on …. Children inherit. Rotate keys on Linux with zfs change-key (not Solaris zfs key -c).

OpenZFS encrypts datasets, not the whole pool. Pool labels and metaslabs stay visible; tank/secret contents do not. This is the TrueNAS / Proxmox / Linux path.

Background: Ars native encryption, ZFS encrypted backups.

Create

1
2

zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt tank/encrypted
zfs create tank/encrypted/child1

child1 inherits encryption. Do not set encryption=off on a child: OpenZFS will not let you punch a plaintext hole under an encrypted parent.

Keyfile instead of prompt:

1
2

zfs create -o encryption=on -o keyformat=passphrase \
  -o keylocation=file:///root/zfs.key tank/encrypted

The key file does not live on the encrypted dataset. USB, TPM, or a path on another pool.

Load the key at boot

1
2

zfs load-key tank/encrypted
zfs mount tank/encrypted

Without load-key the dataset exists and zfs list shows it, but it will not mount.

Receive plaintext as encrypted

On the first recv you can set encryption properties:

1
2
3

zfs send tank/test@snap1 \
  | zfs recv -o encryption=on -o keyformat=passphrase \
      -o keylocation=file:///path/to/keyfile tank/encrypted

That receive is without -w. Raw -w copies the source wrapping key: only valid if the source was already encrypted. Details: unencrypted send into encrypted.

Rotate the passphrase (OpenZFS)

On Solaris the command was zfs key -c. On OpenZFS / Linux / TrueNAS:

1
2

zfs change-key tank/encrypted
zfs get keyformat,keylocation,keystatus tank/encrypted

New keyfile:

1

zfs change-key -o keylocation=file:///media/stick/key tank/encrypted

Wrapping-key rekey (does not rewrite every block; rotates the key that wraps master keys) is zfs change-key -i depending on version — read your zfs change-key man page before production.

See also: snapshot holds, off-host backups.

zfs hold keep pool/dataset@snap — zfs destroy fails with dataset is busy. Release with zfs release.

Retention scripts (zfs destroy -r …@auto-…) do not ask. A hold is a named lock on a snapshot: destroy fails until you drop the tag.

Use it on the snapshot you are zfs sending, or on the only known-good rollback after an upgrade.

Take the hold

1

zfs hold keep tank/home/cindys@snap1

Recursive on a whole tree (snapshot first, then hold):

1
2

zfs snapshot -r tank/home@now
zfs hold -r keep tank/home@now

The tag (keep) is yours, but unique per snapshot. You can have both keep and offsite on the same @now.

What destroy does

1
2

zfs destroy tank/home/cindys@snap1
# cannot destroy 'tank/home/cindys@snap1': dataset is busy

That is the hold, not a mounted filesystem. zfs destroy -d defers deletion until holds are gone; it does not override them.

List

1
2

zfs holds tank/home@now
zfs holds -r tank/home@now

1
2
3

NAME                  TAG   TIMESTAMP
tank/home@now         keep  Thu Jul 15 11:25:39 2010
tank/home/cindys@now  keep  Thu Jul 15 11:25:39 2010

Release, then destroy

1
2

zfs release -r keep tank/home@now
zfs destroy -r tank/home@now

If a long send is still running, wait: the hold exists so prune cannot eat your incremental cursor.

Oracle’s description (same mechanism on OpenZFS): Holding ZFS snapshots.

See also: send into an encrypted dataset.

OFFLINE disk: zpool replace -f <GUID> /dev/disk/by-id/… — GUID from zpool status, not /dev/sdX.

Proxmox documents the case: ZFS on Linux — change a failed device. The symptom is an OFFLINE / FAULTED vdev and a long number instead of sde.

Status

1

zpool status vol1

The dead member often shows up as a GUID:

1

1894156996840098641  OFFLINE

That happens when the kernel no longer has a /dev/sdX for that disk (you yanked it, it died, or the letter moved).

Replace

1

zpool replace -f vol1 1894156996840098641 /dev/disk/by-id/ata-NEWDISK

-f forces if ZFS still “remembers” the old disk. The replacement must be at least the same size (prefer by-id, not sde).

If the new disk is in the same slot and ZFS sees it:

1

zpool replace vol1 /dev/disk/by-id/ata-OLDDISK /dev/disk/by-id/ata-NEWDISK

or auto-detect a replacement in-place:

1

zpool replace vol1 /dev/disk/by-id/ata-OLDDISK

Wait for resilver

1
2

zpool status vol1
watch -n 5 zpool status vol1

Do not reboot, export, or start another replace on the same vdev until resilvered. On RAIDZ1 you are one disk down: a second failure in that window is data loss.

Afterwards

1
2

zpool detach vol1 1894156996840098641   # only if status still shows it as spare/old
zpool labelclear /dev/sdOLD             # if you reuse the old disk elsewhere

See also: import after a power outage.

zfs send -w -R -i over SSH to zfs recv -s. The stream is not a tarball; S3 is an extra hop if needed.

zfs send emits a replicable stream, not a directory. The natural sink is another pool (zfs recv). S3 is a second hop: store the stream as an object, or use something like z3.

Incremental to another host (the command I actually run)

1
2

zfs send -w -R -v -i vol1/secure/backups@initial vol1/secure/backups@new \
  | ssh root@10.0.0.1 zfs recv -s vol1/secure/backups

Hold @initial and @new while the send runs, or prune will break the chain.

The first full (no -i) is mandatory once:

1
2

zfs send -w -R -v vol1/secure/backups@initial \
  | ssh root@10.0.0.1 zfs recv -s vol1/secure/backups

Land it in S3

Same stream, other side of the pipe:

1
2
3

zfs send -w -R vol1/secure/backups@new \
  | gzip -1 \
  | aws s3 cp - s3://my-bucket/zfs/backups@new.zfs.gz

Restore:

1
2
3

aws s3 cp s3://my-bucket/zfs/backups@new.zfs.gz - \
  | gunzip \
  | zfs recv -s vol1/secure/backups

This is not a file sync. It is an opaque blob: you restore the whole stream (or the incremental on top of the full), or you restore nothing. Name @initial / @new and do not delete the full.

For incrementals in S3, tools like z3 keep a catalog of what already landed. A raw aws s3 cp - does not.

-w into S3 is correct if the dataset is already encrypted: AWS never sees plaintext. If the source is not encrypted, either encrypt on recv (post) or encrypt the object (KMS / client) — a cleartext stream in a bucket is a cleartext backup.

See also: native encryption.

Cheat sheet: zpool create, export/import, lz4, L2ARC, SLOG, labelclear, destroy — the 2 a.m. set.

Short reference. Longer procedures are linked at the bottom.

Pool

1
2
3
4
5
6

zpool create vol1 raidz sda sdb sdc
zpool status vol1
zpool export vol1
zpool import -f vol1
zpool destroy tank          # irreversible
zpool labelclear ada0       # wipe ZFS labels from a spare disk

raidz here is RAIDZ1 (one parity). For two, raidz2. Address disks as /dev/disk/by-id/…, not sdX, which reorder.

Mountpoint

1
2
3

zfs unmount vol1
zfs set mountpoint=/mnt/vol1 vol1
zfs mount vol1

L2ARC (cache) and SLOG (ZIL)

1
2
3
4

zpool add vol1 cache <disk-id>
zpool add vol1 log <disk-id>
zpool add vol1 log mirror <disk-id-a> <disk-id-b>
zpool remove vol1 <device>

Cache = hot reads. Log = sync writes (NFS, VMs). A “fast” USB SLOG is worse than no SLOG: if it dies mid-txg, you feel it. Mirror the log.

Compression

1
2
3

zfs set compression=lz4 newvol
zfs set compression=off newvol
zfs get compressratio newvol

lz4 is the sane default on modern OpenZFS. zstd compresses more and costs CPU. The property inherits to child datasets that do not override it.

See also

• Import after a power outage
• Replace disks
• Native encryption

First recv with -o encryption=on (no -w). Destination dataset name is required.

Same bytes, new key. Source stays plaintext; destination is born encrypted. OpenZFS allows this only on the initial receive (dataset that does not exist yet).

Source thread: r/zfs.

Command

1
2
3
4
5
6

zfs snapshot tank/test@snap1
zfs send tank/test@snap1 \
  | zfs recv -o encryption=on \
      -o keyformat=passphrase \
      -o keylocation=file:///path/to/keyfile \
      tank/encrypted

Things that silently break the send:

1. Do not use -w. Raw send replicates the source encryption state. Plaintext in → plaintext out, and -o encryption=… is ignored or errors.
2. Name the destination dataset (tank/encrypted). A recv with no target does not create it.
3. Passphrase vs file: keylocation=prompt is a bad fit across a non-interactive pipe. Use a keyfile.
4. Later incrementals (-i) inherit encryption. Do not pass -o encryption=on again.

Check

1
2

zfs get encryption,keystatus,keyformat tank/encrypted
zfs load-key tank/encrypted   # if keystatus=unavailable

The other direction

Encrypted → encrypted with the same wrapping key: zfs send -w. Encrypted → new wrapping key: send without -w (ZFS decrypts on send; recv encrypts). That needs load-key on the source and is slower.

See also: native encryption.