rsync: resume huge transfers, copy only the directory tree, and decide what counts as 'changed'

TL;DR

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Resume a huge file (VM image, dump) without starting from zero
rsync -av --partial --append-verify --inplace --progress source/ host:/dest/

# Only the directory tree, no files
rsync -a -f'+ */' -f'- *' source/ dest/

# Verify a finished copy without copying anything
rsync -avn --checksum --itemize-changes source/ host:/dest/

# Another SSH port
rsync -av -e 'ssh -p 2222' source/ host:/dest/

How rsync decides what to copy

Before moving a byte, rsync compares each file with its copy at the destination using up to three things: size, modification time, and content checksum. By default it uses the first two (one stat, free); the third forces reading the whole file on both sides.

--checksum only hashes when the size matches, but it still reads every file end to end: over terabytes it is slow, not “safer by default”.

Resuming large transfers

With no options, if the connection drops halfway through a 200 GB file, rsync deletes the piece and the next run starts from zero. Three options change that:

• --partial keeps the half-done file. Better --partial-dir=.rsync-partial: it stores it in a subdirectory and the destination never exposes a truncated file.
• --append-verify resumes from the size the destination already has, trusting what is there, and at the end verifies the checksum of the whole file; if it does not match, it resends it entirely. Meant for files written sequentially (an image being copied, a dump, a video). Plain --append, with no verification, do not use.
• --inplace writes straight into the destination file instead of a temp file plus rename. --append-verify needs it, it saves the double space with huge files, and on destinations with snapshots (ZFS, btrfs) it only rewrites the blocks that changed, so the next snapshot stays small. In exchange, while the copy runs the destination file is inconsistent: do not use it on something being served.

1
2
3

rsync -av --partial-dir=.rsync-partial --append-verify --inplace --progress \
  -e 'ssh -o Compression=no -c aes128-gcm@openssh.com' \
  /var/lib/vz/dump/ backup-host:/backups/vz/

Over SSH the cipher is the bottleneck on a LAN: aes128-gcm@openssh.com with AES-NI, or chacha20-poly1305@openssh.com on CPUs without AES-NI, and Compression=no (compressing already compressed data only burns CPU). --bwlimit=50m if you share the link.

Only the directory tree

1

rsync -a -f'+ */' -f'- *' source/ dest/

Filters are evaluated in order: + */ includes anything that is a directory; - * excludes the rest. With -a it keeps permissions and owners. Useful to prepare a destination, replicate a skeleton on another storage, or see the shape of a dataset without dragging the data.

Verifying at the end

1

rsync -avn --checksum --itemize-changes source/ host:/dest/

-n copies nothing; --itemize-changes prints one line per difference. Empty output means an identical copy. It replaces walking both sides with md5sum by hand and does the comparison for you.

Traps

• The trailing slash changes the result: source/ copies the contents; source creates dest/source/.
• --delete with the wrong path empties the destination. Always -n first.
• --append-verify on files that change in the middle (databases, VM disks in use) verifies, fails, and resends the whole thing every time; there, --partial --inplace without append.
• -z compresses in flight: fine for text, useless for videos, ISOs, .gz dumps, or already compressed VM images.
• macOS as the source: --iconv=utf-8-mac,utf-8 converts names from NFD to NFC in flight; without it, accents arrive “odd” on Linux (the Nextcloud case).
• Hard links are lost with --inplace if the file gets rewritten; use -H without --inplace when they matter.