(Draft — short bio TBD.)

Contact

• Personal email: me [at] inetshell [dot] mx
• Work email: me [at] inet [dot] sh
• GPG/PGP: TBD

Social Networks

Icons below link to the same accounts.

In practice you cannot “just open Internet Explorer.” IE 8 speaks TLS 1.0 (maybe 1.2 with brittle hacks). Google Chrome dropped XP/Vista years ago; current installers will not even start. Mozilla Firefox left XP as well (only ancient ESR builds were usable). Banks, GitHub, cloud consoles, and most SaaS demand TLS 1.2+ and a recent Chromium. On a lab ProLiant still running 2003 you get blank pages, certificate errors, or “cannot display the webpage.”

If you still have to touch a legacy host (firmware, SCADA, a license server, a lab box), you need a modern browser that still builds for old Win32. The option that works in the lab: Supermium.

What Supermium is

A Chromium fork aimed at Windows XP SP3, Server 2003 SP2, Vista, 7, 8.x, and newer. It behaves like a Chrome drop-in (extensions, chrome://flags, sandbox) but still ships 32-bit SSE2 builds for CPUs from that era.

Official requirements (short):

• Windows XP SP3 or Server 2003 SP2 (or later)
• An SSE2-capable CPU (Pentium 4 or newer; dual-core recommended)
• ≥ 768 MB RAM (2 GB+ if you want it usable)

Downloads: win32subsystem.live/supermium — 32-bit and 64-bit setups.

Quick install

1. On a machine with a modern browser, download the 32-bit Setup (most lab XP/2003 is x86) or 64-bit if the OS is x64.
2. Copy the installer to the legacy host (USB, SMB, iLO virtual media).
3. Install as Administrator.
4. Open Supermium and hit https://example.com plus the site you actually need.

If the installer will not run: confirm SP3/SP2, SSE2, and that you are not on Windows 2000 / XP RTM without a service pack.

Extensions and privacy

Unlike current Chrome, Supermium is not killing Manifest V2 on a schedule. You can sideload uBlock Origin from GitHub and keep ads out on an OS that should not be on the public Internet.

It also carries ungoogled-chromium-style flags and classic UI options (Aero, old tabs). Nice; do not treat that as OS hardening.

Warnings (read them)

• The OS is still unpatched. A new browser does not fix SMB1, old RDP, or kernel bugs. Keep the host on an isolated VLAN, VPN, or jump box; do not expose it to the Internet.
• Supermium fixes sites broken by TLS; it does not make XP/2003 a safe daily driver.
• Download only from the official site. Random “Chrome for XP” forum builds are a malware classic.
• For real work the goal is still migration (new VM, container, or move the service to a supported OS). This is a bridge, not a strategy.

When it is worth it

• Opening an iLO / iDRAC / appliance UI you only documented on a 2003 lab box.
• Grabbing a driver or EULA from a portal that rejects IE8.
• Reproducing a compatibility failure without spinning up Windows 10 for one click.

Download and docs: Supermium.

The destination (/datadrive here) must exist and be mounted first. Root or sudo.

1
2
3

sudo systemctl stop k3s
sudo systemctl stop k3s-agent   # only if this node is an agent
sudo /usr/local/bin/k3s-killall.sh

1
2
3

sudo mv /run/k3s/ /datadrive/k3s/
sudo mv /var/lib/kubelet/pods/ /datadrive/k3s-pods/
sudo mv /var/lib/rancher/ /datadrive/k3s-rancher/

1
2
3

sudo ln -s /datadrive/k3s/ /run/k3s
sudo ln -s /datadrive/k3s-pods/ /var/lib/kubelet/pods
sudo ln -s /datadrive/k3s-rancher/ /var/lib/rancher

1
2

sudo systemctl start k3s
sudo systemctl start k3s-agent   # agents only

1
2

kubectl get nodes
kubectl get pods -A

What moves

Working state lives there: containerd, manifests, pods. A half-finished mv or a start against empty paths breaks the cluster.

Traps

/run is tmpfs. After a reboot the /run/k3s symlink is gone. Recreate it from a unit After=local-fs.target, or leave /run/k3s on RAM and do not move it.

k3s-agent does not exist on a server-only node. systemctl stop k3s-agent failing is not a migration failure.

Create the destination parent (mkdir -p /datadrive) and confirm the mount (findmnt /datadrive) before mv. A mv onto an unmounted path leaves the data on the old disk under a new name.

containerd and kubelet sometimes reject a symlink. If the node never goes Ready, switch to a bind mount in /etc/fstab:

1

/datadrive/k3s-rancher  /var/lib/rancher  none  bind  0  0

Cleaner alternative: --data-dir

The official k3s path is --data-dir (default /var/lib/rancher/k3s). On a new node, install with the data-dir already on the large disk. On an existing node, mv + symlink (or bind) of /var/lib/rancher is the shortcut; do not rewrite the unit mid-cluster without a plan.

Write-up of this mv + symlink sequence: How to Move K3s Data to a New Location.

1

nc -v 10.1.1.1 1433

Windows does not ship nc. The closest PowerShell equivalent is a TCP connect test:

1

Test-NetConnection -ComputerName 10.1.1.1 -Port 1433

Short alias:

1

tnc 10.1.1.1 -Port 1433

What to look at

TcpTestSucceeded:

More detail, closer to nc -v:

1

Test-NetConnection -ComputerName 10.1.1.1 -Port 1433 -InformationLevel Detailed

Port only, no ping

Test-NetConnection also pings by default. TCP only:

1
2

Test-NetConnection -ComputerName 10.254.3.2 -Port 1433 -WarningAction SilentlyContinue |
  Select-Object ComputerName, RemoteAddress, RemotePort, TcpTestSucceeded

Very old PowerShell

If Test-NetConnection is missing, use the .NET TCP client:

1
2
3
4
5
6
7
8
9

$tcp = New-Object System.Net.Sockets.TcpClient
try {
    $tcp.Connect("10.254.3.2", 1433)
    Write-Host "Connected to 10.254.3.2:1433"
} catch {
    Write-Host "Failed: $($_.Exception.Message)"
} finally {
    $tcp.Close()
}

• Kubernetes
• Linux
• AWS
• GCP

If you need ssh user@2003-box from a modern jump host (scripts, scp, legacy Ansible, pull logs without RDP), install a third-party SSH server. In the lab I use Bitvise SSH Server; 7.x is the line validated on Server 2003.

Bitvise still lists XP SP3 and Server 2003 in its compatibility matrix. On a host unpatched since 2015, stick to an installer you have already validated (7.x if that is your golden image) or try current only on a clone — not blind on production.

Why not the alternatives

Install (short)

1. On a machine with a modern browser, download the installer from bitvise.com/ssh-server-download (or the 7.x package you keep for the lab).
2. Check the digital signature on the .exe (Properties → Digital Signatures → Bitvise).
3. Copy the installer to the 2003 box (USB, SMB, iLO virtual media). Do not fetch it with IE8 from a random mirror.
4. Run the installer as Administrator.
5. Reboot when Bitvise asks (recommended after first install, especially for key auth).
6. Open the Bitvise SSH Server Control Panel and start the service if it is not set to Automatic.

Official install notes: Installing Bitvise SSH Server.

Minimum config you should not skip

• Accounts: allow only the Windows user(s) you need (or Bitvise virtual accounts). Do not leave “any local account.”
• Port: 22 by default. On a management VLAN that is fine; otherwise change it and restrict with the host firewall / switch ACL.
• Windows 2003 firewall: open TCP/22 (or your chosen port) for the management network only.
• Key auth: import the jump-host pubkey; turn off password auth once keys work.
• SFTP root: mount only the path you need (e.g. D:\logs), not C:\.

From the client:

1
2
3

ssh Administrator@192.168.x.x
# or whichever account you enabled
scp logfile.log Administrator@192.168.x.x:D:/logs/

A modern OpenSSH client talks to Bitvise fine; you do not need the Bitvise client unless you want its GUI.

Security (legacy context)

The OS is still without Microsoft patches. Adding SSH improves operations; it increases network surface if the port is reachable beyond the jump host.

• VLAN / firewall: bastion IP only.
• Do not put the 2003 box on the Internet “because it has SSH now.”
• Licensing: Personal Edition is free for non-commercial use; Standard is eval / paid. Respect that in lab vs work.
• Real goal: migrate the workload. SSH is the bridge to pull data and retire the box.

Related

To open a modern HTTPS portal from the same host (IE8 will not cut it): modern browser on XP/2003 with Supermium.

Vendor docs and downloads: Bitvise SSH Server.

Topics

• ZFS — pools, snapshots, encryption, send/recv
• HP iLO — CLI, firmware, license key
• Proxmox — cloud-init templates and VMs
• Windows — legacy XP/2003, disk, SSH
• k3s — lightweight cluster, data-dir, disk moves

Tags

For a loose keyword (extension, tool, symptom): all tags.

1
2

du -xh --max-depth=1 / | sort -h
# or ncdu /

On Windows there is no decent du in the box. Explorer → Properties is slow and opaque. The tool I use in the lab and on desktops: WinDirStat — free, open source, interactive treemap.

Think of it as du on steroids: scan a drive or folder and see where the space went, not only how many GB.

What it does (beyond summing bytes)

The project site documents multithreaded scans, direct NTFS scanning when available, dark mode, and Explorer integration.

Quick use

1. Download the installer (or portable build) from windirstat.net / the project’s GitHub releases.
2. Run as Administrator if you are scanning all of C:\ (permissions on Windows, other user profiles, etc.).
3. Pick a volume or a specific folder (D:\VMs, C:\Users\…\AppData).
4. Wait for the scan. Large disks take time; this is not du on a tmpfs.
5. In the treemap, zoom into the fat block. Open the folder in Explorer, or delete from the UI only when you know what it is.

When to use it

• Disk is “full” and you do not know whether it is Downloads, hibernation, pagefile, ISOs, or VM snapshots.
• Before/after cleanup of a profile or a share.
• Showing a non-CLI person what owns the space (the treemap sells itself).

What it is not

• It does not replace du/ncdu on Linux or on a NAS over SSH.
• It is not a magic cleaner: deleting System Volume Information, WinSxS, or “big” databases because they look large will break the host.
• A full C:\ scan on a file server with millions of files can take a long time; narrow the root.

Mental map

Download and docs: https://windirstat.net/.

Thread: Import/convert/export RAW images to ZFS volume.

Snapshot and dd

If the VM is running, freeze the disk with a snapshot and read the snap (the @snap ZVOL shows up under /dev/zvol/…):

1
2
3

zfs snapshot rpool/data/vm-101-disk-0@export
dd if=/dev/zvol/rpool/data/vm-101-disk-0@export of=/mnt/backup/vm-101.raw bs=1M status=progress
zfs destroy rpool/data/vm-101-disk-0@export

VM off, you can read the live zvol:

1

dd if=/dev/zvol/rpool/data/vm-101-disk-0 of=/mnt/backup/vm-101.raw bs=1M status=progress

bs=1M matters: the 512 B default takes forever.

qemu-img (qcow2 / vmdk)

1

qemu-img convert -p -f raw -O qcow2 /mnt/backup/vm-101.raw /mnt/backup/vm-101.qcow2

Or straight from the zvol:

1

qemu-img convert -p -f raw -O qcow2 /dev/zvol/rpool/data/vm-101-disk-0 vm-101.qcow2

The other way (RAW → ZVOL)

1
2

zfs create -s -V 32G rpool/data/vm-101-disk-0
dd if=file.raw of=/dev/zvol/rpool/data/vm-101-disk-0 bs=1M status=progress

-s = sparse. The RAW must not be larger than the zvol.

See also: basic commands.

Connect

1

ssh Administrator@<ilo-ip>

Old firmware often needs a legacy cipher:

1

ssh -c aes256-cbc Administrator@<ilo-ip>

Accounts

Create a user with the usual admin groups (network changes reset iLO; accounts persist):

1

create /map1/accounts1 username=ops password='<password>' group=admin,config,oemhp_vm,oemhp_rc,oemhp_power

Change the Administrator password on first boot (Administrator / hpinvent is the HP default):

1

set /map1/accounts1/Administrator password='<new-password>'

Network (iLO reboots)

1
2
3

set map1/dhcpendpt1 EnabledState=no
set map1/enetport1/lanendpt1/ipendpt1 IPv4Address=192.168.1.213 SubnetMask=255.255.255.0
set map1/enetport1 SystemName=merlin

After each network set, wait for iLO to come back and SSH to the new address.

Mount an HTTP ISO and boot once

The ISO must be HTTP-reachable from iLO, not from your laptop.

1
2
3

set /map1/oemhp_vm1/cddr1 oemhp_image=http://10.254.0.50/ssp2017.iso
set /map1/oemhp_vm1/cddr1 oemhp_boot=Once
show /map1/oemhp_vm1/cddr1

Power

1
2
3
4

show /system1
start /system1
stop /system1
reset /system1

Ping from iLO

Useful to see whether iLO can reach the management network:

1

oemhp_ping /map1 10.1.1.50

Manual

HP’s CLP tree: iLO scripting and command line.

See also: upgrade firmware over SSH, read the license key.

See what is there

1

zpool import

Lists importable pools by name (VOL1 below) and whether they are FAULTED / UNAVAIL.

Import

1

zpool import -f VOL1

-f means “I know it might still look imported elsewhere”. That is the usual crash case: ZFS thinks the other side still holds it.

If -f is not enough because the last txgs were half-written:

1

zpool import -F VOL1

-F rewinds to an earlier consistent txg. You can lose the last seconds of writes. That is better than a pool that will not mount.

Dry-run:

1

zpool import -F -n VOL1

Afterwards: scrub

1
2

zpool scrub VOL1
zpool status VOL1

Scrub confirms the rewind did not leave checksum errors. Let it finish before you delete “just in case” snapshots.

Do not

• zpool create on the same sdX devices wipes the labels.
• -F is not the first knob: try -f first.
• If the pool is the root (rpool / Proxmox bpool), import from live media, not from the system that will not boot.

See also: replacing disks, basic commands.

Background: Ars native encryption, ZFS encrypted backups.

Create

1
2

zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt tank/encrypted
zfs create tank/encrypted/child1

child1 inherits encryption. Do not set encryption=off on a child: OpenZFS will not let you punch a plaintext hole under an encrypted parent.

Keyfile instead of prompt:

1
2

zfs create -o encryption=on -o keyformat=passphrase \
  -o keylocation=file:///root/zfs.key tank/encrypted

The key file does not live on the encrypted dataset. USB, TPM, or a path on another pool.

Load the key at boot

1
2

zfs load-key tank/encrypted
zfs mount tank/encrypted

Without load-key the dataset exists and zfs list shows it, but it will not mount.

Receive plaintext as encrypted

On the first recv you can set encryption properties:

1
2
3

zfs send tank/test@snap1 \
  | zfs recv -o encryption=on -o keyformat=passphrase \
      -o keylocation=file:///path/to/keyfile tank/encrypted

That receive is without -w. Raw -w copies the source wrapping key: only valid if the source was already encrypted. Details: unencrypted send into encrypted.

Rotate the passphrase (OpenZFS)

On Solaris the command was zfs key -c. On OpenZFS / Linux / TrueNAS:

1
2

zfs change-key tank/encrypted
zfs get keyformat,keylocation,keystatus tank/encrypted

New keyfile:

1

zfs change-key -o keylocation=file:///media/stick/key tank/encrypted

Wrapping-key rekey (does not rewrite every block; rotates the key that wraps master keys) is zfs change-key -i depending on version — read your zfs change-key man page before production.

See also: snapshot holds, off-host backups.

Use it on the snapshot you are zfs sending, or on the only known-good rollback after an upgrade.

Take the hold

1

zfs hold keep tank/home/cindys@snap1

Recursive on a whole tree (snapshot first, then hold):

1
2

zfs snapshot -r tank/home@now
zfs hold -r keep tank/home@now

The tag (keep) is yours, but unique per snapshot. You can have both keep and offsite on the same @now.

What destroy does

1
2

zfs destroy tank/home/cindys@snap1
# cannot destroy 'tank/home/cindys@snap1': dataset is busy

That is the hold, not a mounted filesystem. zfs destroy -d defers deletion until holds are gone; it does not override them.

List

1
2

zfs holds tank/home@now
zfs holds -r tank/home@now

1
2
3

NAME                  TAG   TIMESTAMP
tank/home@now         keep  Thu Jul 15 11:25:39 2010
tank/home/cindys@now  keep  Thu Jul 15 11:25:39 2010

Release, then destroy

1
2

zfs release -r keep tank/home@now
zfs destroy -r tank/home@now

If a long send is still running, wait: the hold exists so prune cannot eat your incremental cursor.

Oracle’s description (same mechanism on OpenZFS): Holding ZFS snapshots.

See also: send into an encrypted dataset.

Tested on Proxmox 8.0.4. The flow is this gist (fork of zidenis). Fedora 38 / Ubuntu 23.04 URLs go stale: point wget at the distro’s current cloud image.

1. Download the image

On the node, as root, on a storage with space (NFS, local, whatever you use):

1
2
3
4
5
6

export IMAGES_PATH="/mnt/pve/nfs-data/images/"
cd "${IMAGES_PATH}"

wget https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2
wget https://repo.almalinux.org/almalinux/9/cloud/x86_64/images/CHECKSUM -O SHA256SUMS
sha256sum -c SHA256SUMS --ignore-missing

Other distros (uncomment one):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

# Amazon Linux 2 — https://cdn.amazonlinux.com/os-images/latest/
# wget https://cdn.amazonlinux.com/os-images/2.0.20230727.0/kvm/amzn2-kvm-2.0.20230727.0-x86_64.xfs.gpt.qcow2

# CentOS Stream 9
# wget https://cloud.centos.org/centos/9-stream/x86_64/images/CentOS-Stream-GenericCloud-9-latest.x86_64.qcow2

# Fedora (bump the version)
# wget https://download.fedoraproject.org/pub/fedora/linux/releases/38/Cloud/x86_64/images/Fedora-Cloud-Base-38-1.6.x86_64.qcow2

# Oracle Linux 9 — the .qcow is not qcow2; convert it
# wget https://yum.oracle.com/templates/OracleLinux/OL9/u2/x86_64/OL9U2_x86_64-kvm-b197.qcow
# qemu-img convert -O qcow2 -o compat=0.10 OL9U2_x86_64-kvm-b197.qcow OL9U2_x86_64-kvm-b197.qcow2

# RHEL 9 — needs a logged-in download from access.redhat.com

# Rocky 9
# wget https://dl.rockylinux.org/pub/rocky/9/images/x86_64/Rocky-9-GenericCloud-Base.latest.x86_64.qcow2

# Ubuntu cloudimg
# wget https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img

2. VM and cloud-init variables

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

export QEMU_CPU_MODEL="host"
export VM_CPU_SOCKETS=1
export VM_CPU_CORES=2
export VM_MEMORY=4098
export VM_RESOURCE_POOL="CustomResourcePool"

export CLOUD_INIT_USER="user"
export CLOUD_INIT_SSHKEY="/home/user/.ssh/id_rsa.pub"
export CLOUD_INIT_IP="dhcp"
export CLOUD_INIT_NAMESERVER="1.1.1.1"
export CLOUD_INIT_SEARCHDOMAIN="example.com"

export TEMPLATE_ID=1001
export VM_NAME="alma9"
export VM_DISK_IMAGE="${IMAGES_PATH}/AlmaLinux-9-GenericCloud-latest.x86_64.qcow2"

--cpu host passes through the node CPU flags (faster; the template is not portable across CPU vendors). VM_MEMORY=4098 is the gist value; 4096 is fine.

The cloud-init user is not root. The key must be the .pub you will use from your laptop.

3. Create the base VM and import the disk

Swap local-lvm and vmbr0 for your real storage and bridge.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

qm create ${TEMPLATE_ID} --name ${VM_NAME} --cpu ${QEMU_CPU_MODEL} \
  --sockets ${VM_CPU_SOCKETS} --cores ${VM_CPU_CORES} --memory ${VM_MEMORY} \
  --numa 1 --net0 virtio,bridge=vmbr0 --ostype l26 --agent 1 \
  --pool ${VM_RESOURCE_POOL} --scsihw virtio-scsi-single

qm set ${TEMPLATE_ID} --scsi0 local-lvm:0,import-from=${VM_DISK_IMAGE}
qm set ${TEMPLATE_ID} --ide2 local-lvm:cloudinit --boot order=scsi0
qm set ${TEMPLATE_ID} --ipconfig0 ip=${CLOUD_INIT_IP} \
  --nameserver ${CLOUD_INIT_NAMESERVER} --searchdomain ${CLOUD_INIT_SEARCHDOMAIN}
qm set ${TEMPLATE_ID} --ciupgrade 1 --ciuser ${CLOUD_INIT_USER} --sshkeys ${CLOUD_INIT_SSHKEY}
qm cloudinit update ${TEMPLATE_ID}

--agent 1 asks for qemu-guest-agent. Alma/Rocky/Fedora ship it or cloud-init installs it (ciupgrade). Amazon Linux 2 and some Ubuntu cloudimgs will not show the guest IP in the Proxmox GUI until you install the agent yourself.

4. Convert to a template

1
2

qm set ${TEMPLATE_ID} --name "${VM_NAME}-Template"
qm template ${TEMPLATE_ID}

Do not start that VM again: clone it.

5. Clone and boot

1
2
3

export VM_ID=$(pvesh get /cluster/nextid)
qm clone ${TEMPLATE_ID} ${VM_ID} --name ${VM_NAME}
qm start ${VM_ID}

SSH with the private key that matches the .pub you fed cloud-init:

1

ssh user@192.168.0.123 -i ~/.ssh/id_rsa

If Proxmox does not show an IP: serial console, ip neigh on the bridge, or the router’s DHCP leases.

Source: gist inetshell/f1d0206d5319c11062845901e4f3d06b.

The endpoint

1

curl -sk "https://<ilo-ip>/xmldata?item=CpqKey"

In a browser: https://<ilo-ip>/xmldata?item=CpqKey.

The body is XML with the product and the key (or an unlicensed state). This is not Redfish; it is iLO’s xmldata interface, the one old HP SIM scripts used.

Notes

• -k because the iLO certificate is almost never from a public CA.
• Some iLOs still serve this over HTTP (http://<ilo-ip>/xmldata?item=CpqKey) if HTTPS will not negotiate.
• Treat the key as a secret. Do not commit it or paste it.
• Reading it does not activate anything: it only reads what is already stored on the BMC. Applying a different key is still license in CLP or the UI.

See also: iLO CLI commands.

Status

1

zpool status vol1

The dead member often shows up as a GUID:

1

1894156996840098641  OFFLINE

That happens when the kernel no longer has a /dev/sdX for that disk (you yanked it, it died, or the letter moved).

Replace

1

zpool replace -f vol1 1894156996840098641 /dev/disk/by-id/ata-NEWDISK

-f forces if ZFS still “remembers” the old disk. The replacement must be at least the same size (prefer by-id, not sde).

If the new disk is in the same slot and ZFS sees it:

1

zpool replace vol1 /dev/disk/by-id/ata-OLDDISK /dev/disk/by-id/ata-NEWDISK

or auto-detect a replacement in-place:

1

zpool replace vol1 /dev/disk/by-id/ata-OLDDISK

Wait for resilver

1
2

zpool status vol1
watch -n 5 zpool status vol1

Do not reboot, export, or start another replace on the same vdev until resilvered. On RAIDZ1 you are one disk down: a second failure in that window is data loss.

Afterwards

1
2

zpool detach vol1 1894156996840098641   # only if status still shows it as spare/old
zpool labelclear /dev/sdOLD             # if you reuse the old disk elsewhere

See also: import after a power outage.

Incremental to another host (the command I actually run)

1
2

zfs send -w -R -v -i vol1/secure/backups@initial vol1/secure/backups@new \
  | ssh root@10.0.0.1 zfs recv -s vol1/secure/backups

Hold @initial and @new while the send runs, or prune will break the chain.

The first full (no -i) is mandatory once:

1
2

zfs send -w -R -v vol1/secure/backups@initial \
  | ssh root@10.0.0.1 zfs recv -s vol1/secure/backups

Land it in S3

Same stream, other side of the pipe:

1
2
3

zfs send -w -R vol1/secure/backups@new \
  | gzip -1 \
  | aws s3 cp - s3://my-bucket/zfs/backups@new.zfs.gz

Restore:

1
2
3

aws s3 cp s3://my-bucket/zfs/backups@new.zfs.gz - \
  | gunzip \
  | zfs recv -s vol1/secure/backups

This is not a file sync. It is an opaque blob: you restore the whole stream (or the incremental on top of the full), or you restore nothing. Name @initial / @new and do not delete the full.

For incrementals in S3, tools like z3 keep a catalog of what already landed. A raw aws s3 cp - does not.

-w into S3 is correct if the dataset is already encrypted: AWS never sees plaintext. If the source is not encrypted, either encrypt on recv (post) or encrypt the object (KMS / client) — a cleartext stream in a bucket is a cleartext backup.

See also: native encryption.

1. Serve the .bin over HTTP

iLO pulls the firmware; you do not SCP it in. python3 -m http.server on the management network is enough, or nginx. Use the official HPE ilo*.bin.

2. Log in and load

1
2
3
4

ssh -c aes256-cbc Administrator@<ilo-ip>
show /map1/firmware1
cd /map1/firmware1
load -source http://<http-server>/ilofirmware.bin

-c aes256-cbc is the iLO 2/3 trick: modern OpenSSH will not offer the BMC’s ciphers unless you ask.

show /map1/firmware1 prints the current version before you touch anything. Keep it.

What to expect

• load takes minutes. Do not kill the session.
• iLO reboots itself. The host does not power off (firmware lives on the BMC).
• If HTTP is not reachable from the iLO NIC, load fails with a useless error: oemhp_ping the HTTP server.

After reboot, show /map1/firmware1 again and confirm the version string.

Write-up this is based on: Upgrade HP iLO via SSH.

See also: iLO CLI commands.

Pool

1
2
3
4
5
6

zpool create vol1 raidz sda sdb sdc
zpool status vol1
zpool export vol1
zpool import -f vol1
zpool destroy tank          # irreversible
zpool labelclear ada0       # wipe ZFS labels from a spare disk

raidz here is RAIDZ1 (one parity). For two, raidz2. Address disks as /dev/disk/by-id/…, not sdX, which reorder.

Mountpoint

1
2
3

zfs unmount vol1
zfs set mountpoint=/mnt/vol1 vol1
zfs mount vol1

L2ARC (cache) and SLOG (ZIL)

1
2
3
4

zpool add vol1 cache <disk-id>
zpool add vol1 log <disk-id>
zpool add vol1 log mirror <disk-id-a> <disk-id-b>
zpool remove vol1 <device>

Cache = hot reads. Log = sync writes (NFS, VMs). A “fast” USB SLOG is worse than no SLOG: if it dies mid-txg, you feel it. Mirror the log.

Compression

1
2
3

zfs set compression=lz4 newvol
zfs set compression=off newvol
zfs get compressratio newvol

lz4 is the sane default on modern OpenZFS. zstd compresses more and costs CPU. The property inherits to child datasets that do not override it.

See also

• Import after a power outage
• Replace disks
• Native encryption

Source thread: r/zfs.

Command

1
2
3
4
5
6

zfs snapshot tank/test@snap1
zfs send tank/test@snap1 \
  | zfs recv -o encryption=on \
      -o keyformat=passphrase \
      -o keylocation=file:///path/to/keyfile \
      tank/encrypted

Things that silently break the send:

1. Do not use -w. Raw send replicates the source encryption state. Plaintext in → plaintext out, and -o encryption=… is ignored or errors.
2. Name the destination dataset (tank/encrypted). A recv with no target does not create it.
3. Passphrase vs file: keylocation=prompt is a bad fit across a non-interactive pipe. Use a keyfile.
4. Later incrementals (-i) inherit encryption. Do not pass -o encryption=on again.

Check

1
2

zfs get encryption,keystatus,keyformat tank/encrypted
zfs load-key tank/encrypted   # if keystatus=unavailable

The other direction

Encrypted → encrypted with the same wrapping key: zfs send -w. Encrypted → new wrapping key: send without -w (ZFS decrypts on send; recv encrypts). That needs load-key on the source and is slower.

See also: native encryption.