With the disk already enlarged on the hypervisor side (qm resize 100 scsi0 +50G on Proxmox, Edit disk on VMware, your cloud console):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

# 1. Make the kernel see the new size (virtio-scsi/SATA; virtio-blk detects it alone)
echo 1 > /sys/class/block/sda/device/rescan
lsblk /dev/sda

# 2. Enlarge the partition (package cloud-guest-utils on Debian/Ubuntu, cloud-utils-growpart on RHEL)
growpart /dev/sda 3

# 3a. With LVM: PV, LV and filesystem
pvresize /dev/sda3
lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv      # -r: resize2fs or xfs_growfs depending on the fs

# 3b. Without LVM
resize2fs /dev/sda3        # ext4
xfs_growfs /               # xfs: takes the mount point

No reboot. df -h / confirms.

Before starting

• Snapshot or backup on the hypervisor. Growing is safe; picking the wrong partition is not.
• Look at the layout: lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS. It shows the partition number, whether LVM sits underneath, and which filesystem you are dealing with.
• Names: virtio-blk is /dev/vda; virtio-scsi and SATA, /dev/sda; NVMe, /dev/nvme0n1 with partitions p1, p2… (growpart /dev/nvme0n1 1).
• Sometimes the space is already there: Ubuntu Server creates the LV using only part of the VG. If vgs shows a large VFree, jump straight to lvextend.

1. Make the kernel see the new disk

lsblk must show the new size on the disk before moving on; otherwise growpart will say NOCHANGE.

2. growpart

1
2

growpart /dev/sda 3
# CHANGED: partition=3 start=4198400 old: size=… end=… new: size=… end=…

growpart stretches the partition to the end of the disk, moves the GPT backup header, and tells the kernel about the new table even with the partition mounted (it uses partx --update). It only works on the last partition; if a swap partition sits behind the root, remove it first (swapoff, delete it with fdisk, recreate it at the end).

Without growpart: parted /dev/sda resizepart 3 100% does the same with a confirmation prompt.

3a. LVM

1
2
3
4

pvresize /dev/sda3                                  # the PV fills the now-larger partition
vgs                                                 # VFree > 0
lvextend -r -l +100%FREE /dev/ubuntu-vg/ubuntu-lv   # all the space
lvextend -r -L +20G /dev/ubuntu-vg/ubuntu-lv        # or only part of it

-r calls resize2fs or xfs_growfs depending on what lives in the LV. Without -r, do it yourself: resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv or xfs_growfs /.

3b. Without LVM

1
2
3

resize2fs /dev/sda3                 # ext4: online
xfs_growfs /                        # xfs: online, on the mount point
btrfs filesystem resize max /       # btrfs

Raspberry Pi and SD cards

Same script with other names (Fedora ARM image with LVM):

1
2
3
4

growpart /dev/mmcblk0 3
pvresize /dev/mmcblk0p3
lvextend -l +100%FREE /dev/fedora/root
xfs_growfs -d /

Traps

• growpart: NOCHANGE: the kernel has not seen the new size yet. Back to step 1.
• growpart fails in a non-English locale (it parses sfdisk output): LC_ALL=C growpart /dev/sda 3.
• GPT PMBR size mismatch in fdisk -l after growing the disk: the GPT backup header stayed where the old disk ended. growpart moves it; if you did it by hand, sgdisk -e /dev/sda.
• The partition is not the last one. Swap or /home behind the root: either move those, or add a new disk and join it to the VG (pvcreate /dev/sdb && vgextend ubuntu-vg /dev/sdb), which in LVM is faster than reordering partitions.
• Shrinking is another story: ext4 only unmounted; XFS never. Think about it before handing out 2 TB “just in case”.
• Windows grows from Disk Management → Extend Volume; none of this applies.

See also

• cloud-init templates on Proxmox: the host-side qm resize lives there.
• LVM: the volume group will not activate, in case step 3 gets weird.
• Move k3s data to another disk, the alternative when what fills up is one specific service.

1
2
3
4
5
6

vgscan
pvs; vgs; lvs -a -o +devices,lv_health_status     # is a PV missing? which LV will not activate and why?

vgchange -ay                                       # normal attempt
vgchange -ay --activationmode degraded             # LVM raid1/raid5/raid6 LVs with a missing PV
vgchange -ay --activationmode partial              # any LV with a missing PV: ONLY to copy data, mount ro

If the failure is at boot (emergency shell because / or /home does not show up), run the same commands from the initramfs prompt, prefixed with lvm if there are no standalone binaries (lvm vgchange -ay), then exit.

What to look at first

The three activation modes (man vgchange): complete (the default) only activates LVs with all their PVs present; degraded activates LVM RAID LVs missing a leg; partial activates any LV even with missing PVs, filling the gaps with errors or zeros, and is only for recovery.

Cases

1. A PV is missing

Fix the hardware first: reseat, change ports, check the controller. As soon as pvs sees it, vgchange -ay and done.

If the disk does not come back and the LV was linear across several PVs, whatever lived on that disk is gone. Rescue the rest read-only:

1
2
3

vgchange -ay --activationmode partial
mount -o ro /dev/vg01/data /mnt/rescue
rsync -a /mnt/rescue/ /somewhere/else/

And only after copying, clean up the VG: vgreduce --removemissing --force vg01.

2. LVM RAID LVs (raid1, raid5, raid6)

With a missing leg they come up in degraded mode:

1

vgchange -ay --activationmode degraded

Then replace the PV: vgextend vg01 /dev/sdX with the new disk and lvconvert --repair vg01/data. If the LV will not activate even so (reload ioctl … failed, _rimage_ sub-LVs active but not the LV), you are in the scenario where LVM RAID tooling falls short; restoring from backup is the honest way out. For parity RAID, mdadm under LVM gives better recovery tools.

3. Thin pool

Thin pools suffer the most from an outage: at boot, thin_check finds the metadata inconsistent and LVM refuses:

1

Check of pool vg01/pool0 failed (status:1). Manual repair required!

The fix is what the message says, with the pool inactive:

1
2

lvconvert --repair vg01/pool0
vgchange -ay vg01

--repair rebuilds the metadata into a new LV and keeps the old one as pool0_meta0; once you confirm the thin volumes mount fine, lvremove vg01/pool0_meta0. A thin pool with a missing PV will not activate even in degraded mode; only partial remains, to copy.

4. Broken metadata (PVs present, VG “incomplete” or “unknown”)

1
2
3

vgcfgrestore --list vg01                                # files in /etc/lvm/archive with date and reason
vgcfgrestore -f /etc/lvm/archive/vg01_00042-1234567890.vg vg01
vgchange -ay vg01

Metadata only: if what is missing is a disk, restoring the configuration does not bring it back.

5. The disk is there, but LVM ignores it

On RHEL 9, Fedora, and derivatives LVM only looks at the devices listed in /etc/lvm/devices/system.devices. If the outage made a disk come back under another identifier (controller change, different WWID, cloned disk), LVM does not consider it even though lsblk shows it:

1
2
3
4

lvmdevices                          # current list
lvmdevices --adddev /dev/sdc        # add the device
vgimportdevices -a                  # or import every VG it finds
vgchange -ay

After activating

An outage breaks filesystems more often than LVM. Before mounting read-write:

1
2

fsck -n /dev/vg01/data        # ext4: look only
xfs_repair -n /dev/vg01/data  # xfs: look only

And if the result looks scary, mount ro, copy, and repair afterwards.

Traps

• partial fixes nothing: what is missing reads as zeros or errors and any write there is lost. Mount read-only and copy.
• Do not vgreduce --removemissing before copying. It is the last step, not the first.
• /etc/lvm/backup and /etc/lvm/archive are gold and live on the same disk that can fail. Copy them off the machine now and then (vgcfgbackup -f /external/path/%s.vg).
• In VMs the “power failure” is a forced stop; same recipe. And if what does not boot is GRUB itself with root on LVM, that is another problem.

References: man 8 vgchange, man 7 lvmthin, and man 8 lvmdevices.

Boot the Proxmox ISO → Advanced Options → Install Proxmox VE (Debug mode); at the first shell press Ctrl+D to reach the second one, which already has LVM:

1
2
3
4

vgscan
vgchange -ay
lvcreate -L 4M pve -n grubtemp   # any change to the LVM metadata will do
reboot

With the node back up, clean up and make the fix permanent:

1
2
3

lvremove pve/grubtemp
[ -d /sys/firmware/efi ] && apt install grub-efi-amd64     # UEFI
# legacy BIOS: grub-install /dev/sdX && update-grub        # the disk, not the partition

The error

1
2
3

error: disk 'lvmid/Xxxxxx-xxxx-xxxx-xxxx-xxxx-xxxx-xxxxxx/Yyyyyy-yyyy-yyyy-yyyy-yyyy-yyyy-yyyyyy' not found.
Entering rescue mode...
grub rescue>

The disk is fine, the pve VG is fine, and the data is fine. Sometimes the same bug warns you earlier, during an apt full-upgrade:

1

/usr/sbin/grub-probe: error: disk `lvmid/...' not found.

If you see that in update-grub, apply the fix before rebooting.

Why it happens

LVM stores its on-disk metadata in a ring buffer. Every change (creating, deleting, or resizing an LV) writes a new copy further along the buffer, and sooner or later the text wraps around the end. The GRUB 2.06 in Debian 11 (PVE 7, 2.06-3) cannot read that wrap: it does not find the VG, does not find the pve/root LV, and drops to grub rescue.

That is why “fixing” it is as odd as making any LVM change: the metadata gets rewritten at another position, usually without a wrap, and GRUB reads it again. And that is why it comes back weeks or months later, when the buffer wraps again, if the GRUB installed on the disk is still the old one.

In the lab the change that got us out was lvextend -L +1G /dev/pve/root. It works just the same, but undoing it with lvreduce on a mounted root is playing with fire; if you did it that way, better keep the 1G (resize2fs /dev/pve/root) or use the throwaway LV from the TL;DR.

Full procedure

1. Get into the system. Two ways from the Proxmox ISO:

   • Rescue Boot (Advanced Options): the ISO finds the installation and boots its kernel, bypassing the broken GRUB. If it works you are already inside the real system and can jump to step 3.
   • Debug mode: rescue shell. Ctrl+D at the first prompt; the second one has the LVM modules loaded.

2. Trigger a metadata change (only if you came in through Debug mode):

   1 2 3 4

   vgscan vgchange -ay lvcreate -L 4M pve -n grubtemp reboot

3. Back in the normal system, remove the temporary LV and check whether the metadata still wraps:

   1 2	lvremove pve/grubtemp vgscan -vvv 2>&1 | grep 'Reading metadata'

   Output ending in (+0) means no wrap; any other value means the problem will repeat until you reinstall GRUB.

4. Reinstall the GRUB that has the fix. The PVE 8 and 9 package (2.06-13 or newer) already carries it, but the binary sitting in the ESP or the MBR is only rewritten when the package gets (re)installed:

   1 2 3 4 5 6 7 8

   # UEFI: nodes upgraded from PVE 7 often have grub-pc instead of grub-efi-amd64 dpkg -l | grep -E '^ii +grub-' apt install grub-efi-amd64 efibootmgr -v # confirm the "proxmox" entry # legacy BIOS grub-install /dev/sdX update-grub

   On PVE 7 the real fix is upgrading to PVE 8; the workaround only buys time.

Traps

• grub rescue> will not get you out of this. ls (lvm/pve-root) shows nothing because the lvm module is exactly what cannot read the metadata. Do not waste time there.
• Root on ZFS does not go through here. Nodes with rpool boot via proxmox-boot-tool; that is another path with other errors.
• Point at the right devices if you run grub-install by hand: the disk (/dev/sda), not /dev/sda1; the ESP mounted on /boot/efi for UEFI.
• In a cluster, once one node is fixed check the others with vgscan -vvv | grep 'Reading metadata'; same installation, same buffer.

References: Recover From Grub Failure on the Proxmox wiki and the forum thread.