<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Rsync on inetshell</title><link>https://inet.sh/en/tags/rsync/</link><description>Recent content in Rsync on inetshell</description><generator>Hugo</generator><language>en</language><lastBuildDate>Sat, 26 Sep 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://inet.sh/en/tags/rsync/index.xml" rel="self" type="application/rss+xml"/><item><title>rsync: resume huge transfers, copy only the directory tree, and decide what counts as 'changed'</title><link>https://inet.sh/en/posts/linux/rsync-resume-large-transfers/</link><pubDate>Sat, 26 Sep 2026 00:00:00 +0000</pubDate><guid>https://inet.sh/en/posts/linux/rsync-resume-large-transfers/</guid><description>Three rsync things the typical tutorial skips: how to resume copying a file of hundreds of GB without starting over (--partial --append-verify --inplace), how to replicate only the directory tree without files (filters + */ and - *), and what --size-only, --ignore-times and --checksum really do. With the SSH cipher that performs best and how to verify the copy at the end.</description><content:encoded><![CDATA[<h2 id="tldr">TL;DR</h2>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt"> 1
</span><span class="lnt"> 2
</span><span class="lnt"> 3
</span><span class="lnt"> 4
</span><span class="lnt"> 5
</span><span class="lnt"> 6
</span><span class="lnt"> 7
</span><span class="lnt"> 8
</span><span class="lnt"> 9
</span><span class="lnt">10
</span><span class="lnt">11
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl"><span class="c1"># Resume a huge file (VM image, dump) without starting from zero</span>
</span></span><span class="line"><span class="cl">rsync -av --partial --append-verify --inplace --progress source/ host:/dest/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Only the directory tree, no files</span>
</span></span><span class="line"><span class="cl">rsync -a -f<span class="s1">&#39;+ */&#39;</span> -f<span class="s1">&#39;- *&#39;</span> source/ dest/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Verify a finished copy without copying anything</span>
</span></span><span class="line"><span class="cl">rsync -avn --checksum --itemize-changes source/ host:/dest/
</span></span><span class="line"><span class="cl">
</span></span><span class="line"><span class="cl"><span class="c1"># Another SSH port</span>
</span></span><span class="line"><span class="cl">rsync -av -e <span class="s1">&#39;ssh -p 2222&#39;</span> source/ host:/dest/
</span></span></code></pre></td></tr></table>
</div>
</div><h2 id="how-rsync-decides-what-to-copy">How rsync decides what to copy</h2>
<p>Before moving a byte, rsync compares each file with its copy at the destination using up to three things: <strong>size</strong>, <strong>modification time</strong>, and <strong>content checksum</strong>. By default it uses the first two (one <code>stat</code>, free); the third forces reading the whole file on both sides.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>Compares</th>
          <th>Copies the file if…</th>
          <th>When</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>(default)</td>
          <td>size + mtime</td>
          <td>either differs</td>
          <td>Almost always</td>
      </tr>
      <tr>
          <td><code>--size-only</code></td>
          <td>size</td>
          <td>the size differs</td>
          <td>Dates cannot be trusted (restores, FAT, mass <code>touch</code>) and you accept missing changes that keep the size</td>
      </tr>
      <tr>
          <td><code>--checksum</code> / <code>-c</code></td>
          <td>size + checksum</td>
          <td>size or content differ; ignores the date</td>
          <td>Integrity checks; builds that reset dates to 1970</td>
      </tr>
      <tr>
          <td><code>--ignore-times</code> / <code>-I</code></td>
          <td>nothing</td>
          <td>always</td>
          <td>Force a full recopy</td>
      </tr>
  </tbody>
</table>
<p><code>--checksum</code> only hashes when the size matches, but it still reads every file end to end: over terabytes it is slow, not &ldquo;safer by default&rdquo;.</p>
<h2 id="resuming-large-transfers">Resuming large transfers</h2>
<p>With no options, if the connection drops halfway through a 200 GB file, rsync <strong>deletes the piece</strong> and the next run starts from zero. Three options change that:</p>
<ul>
<li><code>--partial</code> keeps the half-done file. Better <code>--partial-dir=.rsync-partial</code>: it stores it in a subdirectory and the destination never exposes a truncated file.</li>
<li><code>--append-verify</code> resumes from the size the destination already has, trusting what is there, and at the end verifies the checksum of the whole file; if it does not match, it resends it entirely. Meant for files written sequentially (an image being copied, a dump, a video). Plain <code>--append</code>, with no verification, do not use.</li>
<li><code>--inplace</code> writes straight into the destination file instead of a temp file plus <code>rename</code>. <code>--append-verify</code> needs it, it saves the double space with huge files, and on destinations with snapshots (ZFS, btrfs) it only rewrites the blocks that changed, so the next snapshot stays small. In exchange, while the copy runs the destination file is inconsistent: do not use it on something being served.</li>
</ul>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span><span class="lnt">2
</span><span class="lnt">3
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">rsync -av --partial-dir<span class="o">=</span>.rsync-partial --append-verify --inplace --progress <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span>  -e <span class="s1">&#39;ssh -o Compression=no -c aes128-gcm@openssh.com&#39;</span> <span class="se">\
</span></span></span><span class="line"><span class="cl"><span class="se"></span>  /var/lib/vz/dump/ backup-host:/backups/vz/
</span></span></code></pre></td></tr></table>
</div>
</div><p>Over SSH the cipher is the bottleneck on a LAN: <code>aes128-gcm@openssh.com</code> with AES-NI, or <code>chacha20-poly1305@openssh.com</code> on CPUs without AES-NI, and <code>Compression=no</code> (compressing already compressed data only burns CPU). <code>--bwlimit=50m</code> if you share the link.</p>
<h2 id="only-the-directory-tree">Only the directory tree</h2>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">rsync -a -f<span class="s1">&#39;+ */&#39;</span> -f<span class="s1">&#39;- *&#39;</span> source/ dest/
</span></span></code></pre></td></tr></table>
</div>
</div><p>Filters are evaluated in order: <code>+ */</code> includes anything that is a directory; <code>- *</code> excludes the rest. With <code>-a</code> it keeps permissions and owners. Useful to prepare a destination, replicate a skeleton on another storage, or see the shape of a dataset without dragging the data.</p>
<h2 id="verifying-at-the-end">Verifying at the end</h2>
<div class="highlight"><div class="chroma">
<table class="lntable"><tr><td class="lntd">
<pre tabindex="0" class="chroma"><code><span class="lnt">1
</span></code></pre></td>
<td class="lntd">
<pre tabindex="0" class="chroma"><code class="language-shell" data-lang="shell"><span class="line"><span class="cl">rsync -avn --checksum --itemize-changes source/ host:/dest/
</span></span></code></pre></td></tr></table>
</div>
</div><p><code>-n</code> copies nothing; <code>--itemize-changes</code> prints one line per difference. Empty output means an identical copy. It replaces walking both sides with <code>md5sum</code> by hand and does the comparison for you.</p>
<h2 id="traps">Traps</h2>
<ul>
<li><strong>The trailing slash changes the result</strong>: <code>source/</code> copies the contents; <code>source</code> creates <code>dest/source/</code>.</li>
<li><strong><code>--delete</code> with the wrong path empties the destination.</strong> Always <code>-n</code> first.</li>
<li><strong><code>--append-verify</code> on files that change in the middle</strong> (databases, VM disks in use) verifies, fails, and resends the whole thing every time; there, <code>--partial --inplace</code> without append.</li>
<li><strong><code>-z</code></strong> compresses in flight: fine for text, useless for videos, ISOs, <code>.gz</code> dumps, or already compressed VM images.</li>
<li><strong>macOS as the source</strong>: <code>--iconv=utf-8-mac,utf-8</code> converts names from NFD to NFC in flight; without it, accents arrive &ldquo;odd&rdquo; on Linux (<a href="/en/posts/nextcloud/incompatible-encoding-nfc-convmv/">the Nextcloud case</a>).</li>
<li><strong>Hard links</strong> are lost with <code>--inplace</code> if the file gets rewritten; use <code>-H</code> without <code>--inplace</code> when they matter.</li>
</ul>
]]></content:encoded></item></channel></rss>