TL;DR

Boot the Proxmox ISO → Advanced Options → Install Proxmox VE (Debug mode); at the first shell press Ctrl+D to reach the second one, which already has LVM:

1
2
3
4

vgscan
vgchange -ay
lvcreate -L 4M pve -n grubtemp   # any change to the LVM metadata will do
reboot

With the node back up, clean up and make the fix permanent:

1
2
3

lvremove pve/grubtemp
[ -d /sys/firmware/efi ] && apt install grub-efi-amd64     # UEFI
# legacy BIOS: grub-install /dev/sdX && update-grub        # the disk, not the partition

The error

1
2
3

error: disk 'lvmid/Xxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxxx/Yyyyyy-yyyy-yyyy-yyyy-yyyy-yyyy-yyyyyy' not found.
Entering rescue mode...
grub rescue>

The disk is fine, the pve VG is fine, and the data is fine. Sometimes the same bug warns you earlier, during an apt full-upgrade:

1

/usr/sbin/grub-probe: error: disk `lvmid/...' not found.

If you see that in update-grub, apply the fix before rebooting.

Why it happens

LVM stores its on-disk metadata in a ring buffer. Every change (creating, deleting, or resizing an LV) writes a new copy further along the buffer, and sooner or later the text wraps around the end. The GRUB 2.06 in Debian 11 (PVE 7, 2.06-3) cannot read that wrap: it does not find the VG, does not find the pve/root LV, and drops to grub rescue.

That is why “fixing” it is as odd as making any LVM change: the metadata gets rewritten at another position, usually without a wrap, and GRUB reads it again. And that is why it comes back weeks or months later, when the buffer wraps again, if the GRUB installed on the disk is still the old one.

In the lab the change that got us out was lvextend -L +1G /dev/pve/root. It works just the same, but undoing it with lvreduce on a mounted root is playing with fire; if you did it that way, better keep the 1G (resize2fs /dev/pve/root) or use the throwaway LV from the TL;DR.

Full procedure

  1. Get into the system. Two ways from the Proxmox ISO:

    • Rescue Boot (Advanced Options): the ISO finds the installation and boots its kernel, bypassing the broken GRUB. If it works you are already inside the real system and can jump to step 3.
    • Debug mode: rescue shell. Ctrl+D at the first prompt; the second one has the LVM modules loaded.

  2. Trigger a metadata change (only if you came in through Debug mode):

    1
2
3
4

    vgscan
vgchange -ay
lvcreate -L 4M pve -n grubtemp
reboot

  3. Back in the normal system, remove the temporary LV and check whether the metadata still wraps:

    1
2

    lvremove pve/grubtemp
vgscan -vvv 2>&1 | grep 'Reading metadata'

    Output ending in (+0) means no wrap; any other value means the problem will repeat until you reinstall GRUB.

  4. Reinstall the GRUB that has the fix. The PVE 8 and 9 package (2.06-13 or newer) already carries it, but the binary sitting in the ESP or the MBR is only rewritten when the package gets (re)installed:

    1
2
3
4
5
6
7
8

    # UEFI: nodes upgraded from PVE 7 often have grub-pc instead of grub-efi-amd64
dpkg -l | grep -E '^ii +grub-'
apt install grub-efi-amd64
efibootmgr -v            # confirm the "proxmox" entry

# legacy BIOS
grub-install /dev/sdX
update-grub

    On PVE 7 the real fix is upgrading to PVE 8; the workaround only buys time.

Traps

  • grub rescue> will not get you out of this. ls (lvm/pve-root) shows nothing because the lvm module is exactly what cannot read the metadata. Do not waste time there.
  • Root on ZFS does not go through here. Nodes with rpool boot via proxmox-boot-tool; that is another path with other errors.
  • Point at the right devices if you run grub-install by hand: the disk (/dev/sda), not /dev/sda1; the ESP mounted on /boot/efi for UEFI.
  • In a cluster, once one node is fixed check the others with vgscan -vvv | grep 'Reading metadata'; same installation, same buffer.

References: Recover From Grub Failure on the Proxmox wiki and the forum thread.