Crush on inetshellhttps://inet.sh/en/tags/crush/Recent content in Crush on inetshellHugoenFri, 25 Sep 2026 00:00:00 +0000Ceph on Proxmox: device classes and CRUSH rules to keep SAS, SSD and NVMe aparthttps://inet.sh/en/posts/proxmox/ceph-device-classes-crush-rules/Fri, 25 Sep 2026 00:00:00 +0000https://inet.sh/en/posts/proxmox/ceph-device-classes-crush-rules/How to make a Ceph pool on Proxmox VE use only one kind of disk: assign device classes to OSDs (set-device-class, with rm-device-class first if they already had one), create a replicated CRUSH rule per class and point the pool at that rule with pveceph or ceph osd pool set. Includes custom classes such as sas.TL;DR 
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# 1. OSD class. If it already has one (automatic hdd/ssd/nvme), remove it first: Ceph will not overwrite it
ceph osd crush rm-device-class osd.5
ceph osd crush set-device-class sas osd.5

# 2. Rule that only places data on that class (root "default", failure domain "host")
ceph osd crush rule create-replicated replicated_sas default host sas

# 3. New pool on the rule, registered as Proxmox storage
pveceph pool create vm-sas --crush_rule replicated_sas --add_storages 1

# ...or move an existing pool (rebalances all of its data)
pveceph pool set vm-fast --crush_rule replicated_sas

Why

Ceph’s default rule (replicated_rule) spreads objects across all OSDs without looking at the disk underneath. In a mixed cluster that means the “fast” pool also lands on the slow disks and performs like the worst of them.

Since Luminous every OSD carries a device class. Ceph detects it on its own at startup (hdd, ssd, or nvme) and a CRUSH rule can be restricted to one class: internally it builds a shadow hierarchy (default~sas) with only those OSDs and places the whole pool there. The class is a free label: besides the automatic ones you can use sas, sata, archive, or whatever helps you separate.

Checking the state

1
2
3
4
5
6
7

ceph osd tree                        # CLASS column per OSD
ceph osd crush class ls              # existing classes
ceph osd crush class ls-osd sas      # OSDs in a class
ceph osd crush tree --show-shadow    # the default~<class> shadow hierarchies
ceph osd crush rule ls
ceph osd crush rule dump replicated_sas
ceph osd pool ls detail              # crush_rule <id> of each pool

Changing an OSD’s class

If the OSD already has a class, set-device-class does not overwrite it:

1
2

Error EBUSY: osd.5 has already bound to class 'hdd', can not reset class to 'sas';
use 'ceph osd crush rm-device-class <id>' to remove old class first

Hence the rm-device-class first. Several OSDs at once:

1
2

ceph osd crush rm-device-class osd.5 osd.6 osd.7
ceph osd crush set-device-class sas osd.5 osd.6 osd.7

There is no “create the class” step: the ceph osd crush class create from the early Luminous builds was removed, and the class appears with the first OSD that gets it. To rename it cluster-wide, ceph osd crush class rename sas sas-10k.

A manually assigned class survives OSD restarts: on startup the OSD tries to set its automatic class and gets the same EBUSY. If you would rather have new OSDs come up with no class so you assign it yourself, ceph config set osd osd_class_update_on_start false; the price is that no new OSD gets an automatic class.

The rule

1
2
3

ceph osd crush rule create-replicated <name> <root> <failure-domain> <class>
ceph osd crush rule create-replicated replicated_sas default host sas
ceph osd crush rule create-replicated replicated_nvme default host nvme
  • root is default unless you built your own CRUSH tree.
  • host as the failure domain puts each replica on a different node. With size 3 you need at least three nodes with OSDs of that class, or the PGs stay undersized.
  • For erasure-coded pools the class goes into the profile: pveceph pool create ec-sas --erasure-coding k=4,m=2,device-class=sas.

The pool

New, with the Proxmox storage included:

1

pveceph pool create vm-sas --crush_rule replicated_sas --size 3 --min_size 2 --add_storages 1

Existing: pveceph pool set vm-fast --crush_rule replicated_sas, or in the GUI Ceph → Pools → Edit → Crush Rule. It is the same as ceph osd pool set vm-fast crush_rule replicated_sas. The change triggers a backfill of the whole pool onto the class’s OSDs; do it in a maintenance window and follow it with ceph -s.

Traps

  • Count the OSDs before the rule. Three nodes, size 3, and only two with sas disks: the pool never reaches active+clean.
  • Automatic classes get it wrong with RAID/HBA controllers that present SSDs as rotational. Check ceph osd tree after adding disks and correct with rm-device-class + set-device-class.
  • Moving a big pool to another rule moves all of its data. Raise osd_max_backfills only if you know what you are doing; it is slow by default on purpose.
  • Deleting a pool that refuses to go: pveceph pool destroy <pool> --force --remove_storages (the data goes with it).

See also

]]>