TL;DR

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Receiver ready to resume: -s on the recv
zfs send -v -w tank/data@snap1 | ssh backup-host zfs recv -s backup/data

# It dropped. On the receiver, ask for the token:
zfs get -H -o value receive_resume_token backup/data

# On the sender, resume from that point (no dataset or snapshot named: the token carries it all):
zfs send -v -t 1-d4f61d893-c0-789c636064000310a500c4ec50360710e72765a5… | ssh backup-host zfs recv -s backup/data

# Give up and free what was half received:
zfs recv -A backup/data

It works the same for incremental sends (-i / -I) and for raw sends of encrypted datasets (-w).

Why

Without -s, an interrupted zfs recv throws away everything it had received: hours of transfer gone because of a network blip or a reboot. With -s (OpenZFS since 2017; Linux 0.7+, FreeBSD 12+) the receiver keeps the received piece in a hidden dataset (backup/data/%recv) and exposes the receive_resume_token property.

The token is the complete description of what is missing: it identifies the source snapshot, the object and offset where it stopped, and the stream options (raw, compressed, large blocks…). That is why zfs send -t needs no other arguments, and why you do not have to remember which flags the original send carried: they are rebuilt from the token.

Step by step

  1. Always receive with -s. It costs nothing and it is the only thing to remember before the drop.

  2. After the interruption, on the receiver:

    1

    zfs get -H -o value receive_resume_token backup/data

    If it returns -, there is no partial state (the send finished or never started).

  3. On the sender, zfs send -t <token> with the usual pipeline into zfs recv -s. You can cut and resume as many times as needed; the token changes each time.

  4. Check the end: zfs get receive_resume_token goes back to - and the snapshot shows up in zfs list -t snapshot -r backup/data.

For progress, -v on the send (bytes sent) or pv in the middle; to buffer latency on slow links, mbuffer between send and ssh.

Aborting

1

zfs recv -A backup/data

Deletes the %recv and frees the space. Needed when the token no longer works (below) or when you decide to resend from another snapshot.

With syncoid

syncoid uses recv -s and the tokens on its own: if a send drops, the next run resumes it by itself. --no-resume turns it off if it ever gets in the way. Sanoid + syncoid are the comfortable way to do this in production.

Traps

  • Destroying the source snapshot invalidates the token. If your snapshot policy (or a manual zfs destroy) takes @snap1 away mid-send, send -t fails: zfs recv -A and start from a snapshot that exists. A hold on the snapshot you are sending prevents it.
  • Both ends must support it: modern OpenZFS on both. Old Solaris and Illumos do not understand -s.
  • -w is preserved in the token: a resumed raw send stays raw; do not try to mix a send -t with a recv into a target different from the original.
  • While partial state exists, zfs destroy of the target dataset complains: recv -A first.
  • -R streams (packages with several snapshots) resume too, but it is simpler to send snapshot by snapshot with -i, which is what syncoid does.

See also