View Issue Details

IDProjectCategoryView StatusLast Update
0015749CentOS-7grub2public2019-01-27 00:54
Reporterloganv 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformOSCentOSOS Version7
Product Version7.6.1810 
Target VersionFixed in Version 
Summary0015749: Grub does not support ext4 meta_bg flag
DescriptionWhen a filesystem containing /boot is resized to >1024 times its original size, such as on a cloud image that is deployed to the disk and then growpart used, grub2-install can no longer read the filesystem. This is because grub reserves GDT blocks to allow growing the filesystem to 1024 times its original size by default, and when resizing past that point, a filesystem flag (meta_bg) is added which makes the filesystem unrecognizable to CentOS 7's grub.

In my case, I have a <1GB CentOS image containing a single partition, and when deploying to a 2TB or larger volume this occurs every time.

This occurs because the GDT reserved blocks are exhausted due to resizing past 1024x the original filesystem size. When this happens, the resize_inodes flag is removed, the meta_bg flag is added, and a kernel message is logged "Converting file system to meta_bg". See https://www.spinics.net/lists/linux-ext4/msg33898.html

After this change from resize_inodes -> meta_bg flag on the ext4 FS, grub-install can no longer read the filesystem. It fails with "grub2-install: error: unknown filesystem."

According to a changelog, grub2 added support for meta_bg in 2015, however it seems like this support does not exist in CentOS 7:
https://fossies.org/linux/grub/ChangeLog
2015-02-16 Vladimir Serbinenko <phcoder@gmail.com>
ext2: Support META_BG.
This fixes bug that system would become unbootable after ext*
online resize if no resize_inode was created at ext* format time.
Steps To ReproduceCreate a small ext4 filesystem containing boot
Run grub2-install with the path to the filesystem and see that it works
Resize the filesystem to > 1024 times its original size
Note the kernel message logged:
kernel: EXT4-fs (sda2): Converting file system to meta_bg
Attempt to grub-install the resized filesystem and see that grub fails with "unknown filesystem"
Additional Information--------------------------------------------------------------------------------
BEFORE RESIZE
--------------------------------------------------------------------------------
bash-4.2# parted /dev/sda print
Model: ATA WDC WD3000FYYZ-0 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1049kB 9437kB 8389kB BSP bios_grub
 2 9437kB 1368MB 1359MB ext4 root
 3 3001GB 3001GB 67.1MB fat16 primary

bash-4.2# tune2fs -l /dev/sda2
tune2fs 1.42.9 (28-Dec-2013)
Filesystem volume name: cloudimg-rootfs
Last mounted on: /
Filesystem UUID: 94feb0b3-cc60-448e-b101-e68c7b0bdae6
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 331936
Block count: 331776
Reserved block count: 16588
Free blocks: 80668
Free inodes: 301701
First block: 0
Block size: 4096
Fragment size: 4096
Reserved GDT blocks: 80
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 30176
Inode blocks per group: 1886
Flex block group size: 16
Filesystem created: Thu Jan 24 03:21:20 2019
Last mount time: Sat Jan 26 13:08:51 2019
Last write time: Sat Jan 26 13:08:51 2019
Mount count: 3
Maximum mount count: -1
Last checked: Thu Jan 24 03:21:20 2019
Check interval: 0 (<none>)
Lifetime writes: 1337 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: de9623cb-c755-4ca3-94ec-1b651d8994f9
Journal backup: inode blocks
bash-4.2# grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.

--------------------------------------------------------------------------------
RESIZE OPERATION
--------------------------------------------------------------------------------
bash-4.2# growpart /dev/sda 2
CHANGED: disk=/dev/sda partition=2: start=18432 old: size=2654207,end=2672639 new: size=5860383663,end=5860402095
bash-4.2# resize2fs /dev/sda2
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/sda2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 175
The filesystem on /dev/sda2 is now 732547958 blocks long.

--------------------------------------------------------------------------------
AFTER RESIZE
--------------------------------------------------------------------------------
Kernel log:
[Sat Jan 26 13:16:02 2019] EXT4-fs (sda2): resizing filesystem from 331776 to 732547958 blocks
[Sat Jan 26 13:16:03 2019] EXT4-fs (sda2): resizing filesystem from 339738624 to 732547958 blocks
[Sat Jan 26 13:16:03 2019] EXT4-fs (sda2): Converting file system to meta_bg
[Sat Jan 26 13:16:03 2019] EXT4-fs (sda2): resizing filesystem from 339738624 to 732547958 blocks
[Sat Jan 26 13:16:05 2019] EXT4-fs (sda2): resized filesystem to 732547958

bash-4.2# parted /dev/sda print
Model: ATA WDC WD3000FYYZ-0 (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1049kB 9437kB 8389kB BSP bios_grub
 2 9437kB 3001GB 3001GB ext4 root
 3 3001GB 3001GB 67.1MB fat16 primary

 bash-4.2# tune2fs -l /dev/sda2
tune2fs 1.42.9 (28-Dec-2013)
Filesystem volume name: cloudimg-rootfs
Last mounted on: /
Filesystem UUID: 94feb0b3-cc60-448e-b101-e68c7b0bdae6
Filesystem magic number: 0xEF53
Filesystem revision #: 1 (dynamic)
Filesystem features: has_journal ext_attr dir_index filetype needs_recovery meta_bg extent flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize
Filesystem flags: signed_directory_hash
Default mount options: user_xattr acl
Filesystem state: clean
Errors behavior: Continue
Filesystem OS type: Linux
Inode count: 674614656
Block count: 732547958
Reserved block count: 29303361
Free blocks: 690108140
Free inodes: 674584421
First block: 0
Block size: 4096
Fragment size: 4096
Blocks per group: 32768
Fragments per group: 32768
Inodes per group: 30176
Inode blocks per group: 1886
First meta block group: 81
Flex block group size: 16
Filesystem created: Thu Jan 24 03:21:20 2019
Last mount time: Sat Jan 26 13:08:51 2019
Last write time: Sat Jan 26 13:08:51 2019
Mount count: 3
Maximum mount count: -1
Last checked: Thu Jan 24 03:21:20 2019
Check interval: 0 (<none>)
Lifetime writes: 1337 MB
Reserved blocks uid: 0 (user root)
Reserved blocks gid: 0 (group root)
First inode: 11
Inode size: 256
Required extra isize: 28
Desired extra isize: 28
Journal inode: 8
Default directory hash: half_md4
Directory Hash Seed: de9623cb-c755-4ca3-94ec-1b651d8994f9
Journal backup: inode blocks

bash-4.2# grub2-install /dev/sda
Installing for i386-pc platform.
grub2-install: error: unknown filesystem.
TagsNo tags attached.
abrt_hash
URL

Activities

loganv

loganv

2019-01-26 18:30

reporter   ~0033694

I forgot to mention but the system becomes unbootable after the resize. Grub drops to the rescue shell since it cannot read the filesystem anymore.

As a workaround, one could reserve more GDT blocks at mkfs time using the "mkfs.ext4 -E resize=<blocks>" argument. This is not ideal because you have to use non-default mkfs arguments at creation time to save yourself from an unbootable system down the road, even though you might never expect to resize past the 1024x size at the time the filesystem is created.

Also, there is no immediate impact since the online resize completes successfully. Running grub2-install does expose the problem prior to reboot, but if you were to perform the resize and then reboot afterward with or without running grub2-install (possibly months or years later), you end up with an unbootable system dropping to the grub rescue shell with no idea why, since you have a perfectly valid ext4 filesystem sitting on the disk.
jrd

jrd

2019-01-26 20:29

manager   ~0033697

If you have not already done so I would strongly encourage you to file this upstream at https://bugzilla.redhat.com - I did brief search and I did not spot anything that
seems similar to this. Any fix for this would have to come from upstream unless we put out a fasttrack update; but even in that case upstream should be aware of it sooner rather than later. You can reference this bug url in the "Add External Bug:" fields, selecting CentOS and this bug ID or just paste the url itself in the field provided.
loganv

loganv

2019-01-27 00:54

reporter   ~0033698

Done - https://bugzilla.redhat.com/show_bug.cgi?id=1669772

Thanks!

Issue History

Date Modified Username Field Change
2019-01-26 18:22 loganv New Issue
2019-01-26 18:30 loganv Note Added: 0033694
2019-01-26 20:29 jrd Note Added: 0033697
2019-01-27 00:54 loganv Note Added: 0033698