1
2
3
4
5
6
7
8
9

# HEALTH_WARN: N daemons have recently crashed
ceph crash ls-new
ceph crash info <id>          # which daemon, when, and the backtrace
ceph crash archive-all        # the warning goes away; the crash stays in `ceph crash ls`

# Disks of a VM that no longer exists
rbd ls -l cephpool
pvesm free cephpool:vm-124-disk-1     # through the Proxmox storage layer
rbd rm cephpool/vm-124-disk-1         # or straight in Ceph

HEALTH_WARN: daemons have recently crashed

1
2
3

HEALTH_WARN 1 daemons have recently crashed
[WRN] RECENT_CRASH: 1 daemons have recently crashed
    osd.5 crashed on host pve-node2 at 2026-09-20T03:14:07.123456Z

It is the RECENT_CRASH check from the manager’s crash module: a daemon fell over, left a report, and the warning stays until an administrator archives it, or until two weeks pass (mgr/crash/warn_recent_interval). The daemon has almost always come back on its own; the warning does not say it is still down.

Before archiving, look at what it was:

1
2

ceph crash ls-new
ceph crash info 2026-09-20T03:14:07.123456Z_9f1c...

What matters from info is entity_name (which daemon), utc_timestamp, assert_msg or backtrace, and ceph_version. Then:

1
2
3
4
5

ceph osd stat                          # are they all still up/in?
systemctl status ceph-osd@5            # on the OSD's node
journalctl -u ceph-osd@5 --since '-1h'
dmesg -T | grep -iE 'sd[a-z]|nvme|i/o error'
smartctl -a /dev/sdX                   # if the crash lines up with disk errors

With that settled, archive:

1
2
3

ceph crash archive <id>
ceph crash archive-all
ceph -s                                # HEALTH_OK if nothing else is going on

Archiving fixes nothing; it only acknowledges you saw it. If the same OSD shows up in ls-new every few days, it is a disk or a controller, not an annoying notice: take it out of the cluster and replace it.

To clean the history: ceph crash rm <id> or ceph crash prune <days>. The warning can be turned off entirely with ceph config set mgr mgr/crash/warn_recent_interval 0, which is exactly what you do not want in production.

Orphaned VM disks in RBD

They appear when a VM is deleted while the storage is unreachable, a clone or migration fails halfway, someone deletes the .conf by hand, or a restore is left incomplete. They take space and do not show in the GUI because no .conf mentions them.

Finding them

Compare the pool’s images with the .conf files of every node (/etc/pve/nodes/ holds each node’s copy):

1
2
3
4
5

for img in $(rbd ls cephpool); do
  grep -rqs "cephpool:$img" /etc/pve/nodes/*/qemu-server /etc/pve/nodes/*/lxc \
    || echo "no owner: $img"
done
rbd ls -l cephpool          # sizes

pvesm list cephpool gives the same list with the VMID Proxmox infers from the name; handy to cross-check against qm list and pct list on each node.

If the VM still exists

Let Proxmox see it again and delete it from there:

1

qm rescan --vmid 124        # shows up as unused0 in the .conf

Then GUI → Hardware → the Unused disk → Remove, which does delete the image, or pvesm free cephpool:vm-124-disk-1. Careful: qm set 124 --delete unused0 only drops it from the .conf, it deletes no data.

If the VM no longer exists

1
2
3

pvesm free cephpool:vm-124-disk-1
# direct Ceph equivalent:
rbd rm cephpool/vm-124-disk-1

When rbd rm refuses:

Traps

• Cross-check against every node, not the qm list of the node you are on; the disk of a VM living on another node is not an orphan.
• base-* is not junk: those are the templates. Delete them only through qm destroy of the template.
• Space comes back slowly: ceph df takes a while to reflect the delete while the OSDs remove objects.
• A whole pool you no longer need: pveceph pool destroy <pool> --force --remove_storages; it takes everything inside with it.

See also

• Ceph: device classes and CRUSH rules
• VM is locked (snapshot-delete) — orphaned RBD snapshots show up there
• Ceph documentation: Health checks — RECENT_CRASH

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# 1. OSD class. If it already has one (automatic hdd/ssd/nvme), remove it first: Ceph will not overwrite it
ceph osd crush rm-device-class osd.5
ceph osd crush set-device-class sas osd.5

# 2. Rule that only places data on that class (root "default", failure domain "host")
ceph osd crush rule create-replicated replicated_sas default host sas

# 3. New pool on the rule, registered as Proxmox storage
pveceph pool create vm-sas --crush_rule replicated_sas --add_storages 1

# ...or move an existing pool (rebalances all of its data)
pveceph pool set vm-fast --crush_rule replicated_sas

Why

Ceph’s default rule (replicated_rule) spreads objects across all OSDs without looking at the disk underneath. In a mixed cluster that means the “fast” pool also lands on the slow disks and performs like the worst of them.

Since Luminous every OSD carries a device class. Ceph detects it on its own at startup (hdd, ssd, or nvme) and a CRUSH rule can be restricted to one class: internally it builds a shadow hierarchy (default~sas) with only those OSDs and places the whole pool there. The class is a free label: besides the automatic ones you can use sas, sata, archive, or whatever helps you separate.

Checking the state

1
2
3
4
5
6
7

ceph osd tree                        # CLASS column per OSD
ceph osd crush class ls              # existing classes
ceph osd crush class ls-osd sas      # OSDs in a class
ceph osd crush tree --show-shadow    # the default~<class> shadow hierarchies
ceph osd crush rule ls
ceph osd crush rule dump replicated_sas
ceph osd pool ls detail              # crush_rule <id> of each pool

Changing an OSD’s class

If the OSD already has a class, set-device-class does not overwrite it:

1
2

Error EBUSY: osd.5 has already bound to class 'hdd', can not reset class to 'sas';
use 'ceph osd crush rm-device-class <id>' to remove old class first

Hence the rm-device-class first. Several OSDs at once:

1
2

ceph osd crush rm-device-class osd.5 osd.6 osd.7
ceph osd crush set-device-class sas osd.5 osd.6 osd.7

There is no “create the class” step: the ceph osd crush class create from the early Luminous builds was removed, and the class appears with the first OSD that gets it. To rename it cluster-wide, ceph osd crush class rename sas sas-10k.

A manually assigned class survives OSD restarts: on startup the OSD tries to set its automatic class and gets the same EBUSY. If you would rather have new OSDs come up with no class so you assign it yourself, ceph config set osd osd_class_update_on_start false; the price is that no new OSD gets an automatic class.

The rule

1
2
3

ceph osd crush rule create-replicated <name> <root> <failure-domain> <class>
ceph osd crush rule create-replicated replicated_sas default host sas
ceph osd crush rule create-replicated replicated_nvme default host nvme

• root is default unless you built your own CRUSH tree.
• host as the failure domain puts each replica on a different node. With size 3 you need at least three nodes with OSDs of that class, or the PGs stay undersized.
• For erasure-coded pools the class goes into the profile: pveceph pool create ec-sas --erasure-coding k=4,m=2,device-class=sas.

The pool

New, with the Proxmox storage included:

1

pveceph pool create vm-sas --crush_rule replicated_sas --size 3 --min_size 2 --add_storages 1

Existing: pveceph pool set vm-fast --crush_rule replicated_sas, or in the GUI Ceph → Pools → Edit → Crush Rule. It is the same as ceph osd pool set vm-fast crush_rule replicated_sas. The change triggers a backfill of the whole pool onto the class’s OSDs; do it in a maintenance window and follow it with ceph -s.

Traps

• Count the OSDs before the rule. Three nodes, size 3, and only two with sas disks: the pool never reaches active+clean.
• Automatic classes get it wrong with RAID/HBA controllers that present SSDs as rotational. Check ceph osd tree after adding disks and correct with rm-device-class + set-device-class.
• Moving a big pool to another rule moves all of its data. Raise osd_max_backfills only if you know what you are doing; it is slow by default on purpose.
• Deleting a pool that refuses to go: pveceph pool destroy <pool> --force --remove_storages (the data goes with it).

See also

• Ceph on Proxmox: crashes and orphaned disks
• No-subscription repositories — includes the Ceph repo
• Ceph documentation: CRUSH Maps — Device classes

PVE 9 (Debian 13 trixie, deb822 format):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

cat > /etc/apt/sources.list.d/proxmox.sources <<'EOF'
Types: deb
URIs: http://download.proxmox.com/debian/pve
Suites: trixie
Components: pve-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
EOF

# Disable enterprise (without a subscription it returns 401 on every apt update)
grep -q '^Enabled:' /etc/apt/sources.list.d/pve-enterprise.sources \
  || echo 'Enabled: false' >> /etc/apt/sources.list.d/pve-enterprise.sources

apt update && apt full-upgrade

PVE 8 (Debian 12 bookworm, one-line format):

1
2
3
4
5

echo 'deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription' \
  > /etc/apt/sources.list.d/pve-no-subscription.list
sed -i 's/^deb /# deb /' /etc/apt/sources.list.d/pve-enterprise.list

apt update && apt full-upgrade

If the host runs Ceph, the Ceph enterprise repo has to be disabled too and swapped for its no-subscription (below).

What each repository does

Proxmox is clear about it: pve-no-subscription is for testing and non-production use; in production they want you to pay for a subscription and use enterprise. For a lab or a homelab it is the normal repo.

The enterprise 401 does not break anything by itself, but every apt update ends in an error and the other repos are left half refreshed. Disabling it is the fix; there is no need to delete it.

PVE 9: .sources files (deb822)

Since PVE 9 the installer already writes /etc/apt/sources.list.d/pve-enterprise.sources in deb822 format. A .sources file is a block of keys:

1
2
3
4
5
6
7

# /etc/apt/sources.list.d/pve-enterprise.sources
Types: deb
URIs: https://enterprise.proxmox.com/debian/pve
Suites: trixie
Components: pve-enterprise
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg
Enabled: false

Enabled: false keeps it written but inactive. The no-subscription block is the one in the TL;DR; the test one is the same with Components: pve-test.

If you carry .list files from an upgrade off PVE 8, apt 3 converts them for you:

1

apt modernize-sources

It rewrites each .list as .sources and leaves the original as .list.bak.

PVE 8: .list files

1
2
3
4
5

# /etc/apt/sources.list.d/pve-enterprise.list
# deb https://enterprise.proxmox.com/debian/pve bookworm pve-enterprise

# /etc/apt/sources.list.d/pve-no-subscription.list
deb http://download.proxmox.com/debian/pve bookworm pve-no-subscription

Commenting the enterprise line with # is the deb822 Enabled: false equivalent.

Ceph

The installer also leaves the Ceph enterprise repo. Same treatment; the repo name carries the Ceph release the cluster runs (ceph -v):

PVE 9:

1
2
3
4
5
6

# /etc/apt/sources.list.d/ceph.sources
Types: deb
URIs: http://download.proxmox.com/debian/ceph-tentacle
Suites: trixie
Components: no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

Swap ceph-tentacle for ceph-squid if you are still on Squid. If the original file had Components: enterprise, replace the whole file with this one.

PVE 8:

1
2
3

# /etc/apt/sources.list.d/ceph.list
# deb https://enterprise.proxmox.com/debian/ceph-squid bookworm enterprise
deb http://download.proxmox.com/debian/ceph-squid bookworm no-subscription

On bookworm there are ceph-quincy, ceph-reef, and ceph-squid; use the one matching your cluster, not the newest.

Proxmox Backup Server

Identical pattern with pbs in the URI and pbs-no-subscription as the component:

1
2
3
4
5
6

# /etc/apt/sources.list.d/proxmox.sources (PBS 4, trixie)
Types: deb
URIs: http://download.proxmox.com/debian/pbs
Suites: trixie
Components: pbs-no-subscription
Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg

And Enabled: false in pbs-enterprise.sources.

Traps

• Do not mix suites. bookworm on PVE 8, trixie on PVE 9. One line with the wrong suite breaks the next full-upgrade.
• The keyring is already there. /usr/share/keyrings/proxmox-archive-keyring.gpg ships with any ISO install. You only need to download it when installing Proxmox on top of a clean Debian (the official guide for that path covers it).
• Signed-By is mandatory in deb822; without that line apt ignores the repo with a warning.
• If apt fails with pve-apt-hook: not found, that is not a repo problem: separate post.
• The GUI’s “no valid subscription” notice is independent of the repos and is not fixed here; a Community subscription removes it and gives you the enterprise repo as a bonus.

Reference: Package Repositories on the Proxmox wiki.