Ceph on Proxmox: clearing the 'daemons have recently crashed' HEALTH_WARN and deleting orphaned VM disks in RBD

TL;DR

1
2
3
4
5
6
7
8
9

# HEALTH_WARN: N daemons have recently crashed
ceph crash ls-new
ceph crash info <id>          # which daemon, when, and the backtrace
ceph crash archive-all        # the warning goes away; the crash stays in `ceph crash ls`

# Disks of a VM that no longer exists
rbd ls -l cephpool
pvesm free cephpool:vm-124-disk-1     # through the Proxmox storage layer
rbd rm cephpool/vm-124-disk-1         # or straight in Ceph

HEALTH_WARN: daemons have recently crashed

1
2
3

HEALTH_WARN 1 daemons have recently crashed
[WRN] RECENT_CRASH: 1 daemons have recently crashed
    osd.5 crashed on host pve-node2 at 2026-09-20T03:14:07.123456Z

It is the RECENT_CRASH check from the manager’s crash module: a daemon fell over, left a report, and the warning stays until an administrator archives it, or until two weeks pass (mgr/crash/warn_recent_interval). The daemon has almost always come back on its own; the warning does not say it is still down.

Before archiving, look at what it was:

1
2

ceph crash ls-new
ceph crash info 2026-09-20T03:14:07.123456Z_9f1c...

What matters from info is entity_name (which daemon), utc_timestamp, assert_msg or backtrace, and ceph_version. Then:

1
2
3
4
5

ceph osd stat                          # are they all still up/in?
systemctl status ceph-osd@5            # on the OSD's node
journalctl -u ceph-osd@5 --since '-1h'
dmesg -T | grep -iE 'sd[a-z]|nvme|i/o error'
smartctl -a /dev/sdX                   # if the crash lines up with disk errors

With that settled, archive:

1
2
3

ceph crash archive <id>
ceph crash archive-all
ceph -s                                # HEALTH_OK if nothing else is going on

Archiving fixes nothing; it only acknowledges you saw it. If the same OSD shows up in ls-new every few days, it is a disk or a controller, not an annoying notice: take it out of the cluster and replace it.

To clean the history: ceph crash rm <id> or ceph crash prune <days>. The warning can be turned off entirely with ceph config set mgr mgr/crash/warn_recent_interval 0, which is exactly what you do not want in production.

Orphaned VM disks in RBD

They appear when a VM is deleted while the storage is unreachable, a clone or migration fails halfway, someone deletes the .conf by hand, or a restore is left incomplete. They take space and do not show in the GUI because no .conf mentions them.

Finding them

Compare the pool’s images with the .conf files of every node (/etc/pve/nodes/ holds each node’s copy):

1
2
3
4
5

for img in $(rbd ls cephpool); do
  grep -rqs "cephpool:$img" /etc/pve/nodes/*/qemu-server /etc/pve/nodes/*/lxc \
    || echo "no owner: $img"
done
rbd ls -l cephpool          # sizes

pvesm list cephpool gives the same list with the VMID Proxmox infers from the name; handy to cross-check against qm list and pct list on each node.

If the VM still exists

Let Proxmox see it again and delete it from there:

1

qm rescan --vmid 124        # shows up as unused0 in the .conf

Then GUI → Hardware → the Unused disk → Remove, which does delete the image, or pvesm free cephpool:vm-124-disk-1. Careful: qm set 124 --delete unused0 only drops it from the .conf, it deletes no data.

If the VM no longer exists

1
2
3

pvesm free cephpool:vm-124-disk-1
# direct Ceph equivalent:
rbd rm cephpool/vm-124-disk-1

When rbd rm refuses:

Traps

• Cross-check against every node, not the qm list of the node you are on; the disk of a VM living on another node is not an orphan.
• base-* is not junk: those are the templates. Delete them only through qm destroy of the template.
• Space comes back slowly: ceph df takes a while to reflect the delete while the OSDs remove objects.
• A whole pool you no longer need: pveceph pool destroy <pool> --force --remove_storages; it takes everything inside with it.

See also

• Ceph: device classes and CRUSH rules
• VM is locked (snapshot-delete) — orphaned RBD snapshots show up there
• Ceph documentation: Health checks — RECENT_CRASH