View Issue Details

IDProjectCategoryView StatusLast Update
0017802CentOS-8[All Projects] generalpublic2020-10-15 16:14
Reporterusai 
PriorityurgentSeverityblockReproducibilityalways
Status newResolutionopen 
PlatformAmazon EC2OSCentOS 8OS VersionARM64
Product Version8.2.2004 
Target VersionFixed in Version 
Summary0017802: AWS: Root volume cannot be resized on CentOS8 ARM EC2 AMIs
DescriptionI'm trying to launch EC2 instances for Centos8 ARM.
The amis are taken from here: https://www.centos.org/download/aws-images/

The default size of the root volume is 10GB and it's impossible to resize it.
The growpart command is not executed correctly because the order of the partition is wrong in the base AMI.

I tested the AMIs in us-east-1 and eu-west-1.

```
$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
├─nvme0n1p1 259:1 0 600M 0 part /boot/efi
├─nvme0n1p2 259:2 0 7.8G 0 part /
└─nvme0n1p3 259:3 0 1M 0 part
```

cloud-init.log:
```
2020-10-14 08:13:49,941 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/nvme0n1', '2'] with allowed return codes [0] (shell=False, capture=True)
2020-10-14 08:13:50,139 - util.py[DEBUG]: resize_devices took 0.209 seconds
2020-10-14 08:13:50,139 - cc_growpart.py[DEBUG]: '/' NOCHANGE: no change necessary (/dev/nvme0n1, 2)
2020-10-14 08:13:50,139 - handlers.py[DEBUG]: finish: init-network/config-growpart: SUCCESS: config-growpart ran successfully
```
Steps To Reproduce# ARM instance c6g.xlarge
I launched an empty Cento8 ARM instance, starting from ami-01d2b21e37b37aaa7 in eu-west-1 and assigning 30 GB to the root volume, but the size is wrong.

```
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.7G 0 3.7G 0% /dev
tmpfs 3.7G 0 3.7G 0% /dev/shm
tmpfs 3.7G 22M 3.7G 1% /run
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/nvme0n1p2 7.8G 1.4G 6.5G 17% /
/dev/nvme0n1p1 599M 6.4M 593M 2% /boot/efi
tmpfs 749M 0 749M 0% /run/user/1000

$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
├─nvme0n1p1 259:1 0 600M 0 part /boot/efi
├─nvme0n1p2 259:2 0 7.8G 0 part /
└─nvme0n1p3 259:3 0 1M 0 part
```

The size of the 2 partition is 7.8G instead of 30G.

The growpart package is correctly installed:
```
$ rpm -qa | grep growpart
cloud-utils-growpart-0.29-3.el8.noarch
```

By looking at cloud-init.log I can see that the growpart command has been skipped.
```
2020-10-14 08:13:49,941 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/nvme0n1', '2'] with allowed return codes [0] (shell=False, capture=True)
2020-10-14 08:13:50,139 - util.py[DEBUG]: resize_devices took 0.209 seconds
2020-10-14 08:13:50,139 - cc_growpart.py[DEBUG]: '/' NOCHANGE: no change necessary (/dev/nvme0n1, 2)
2020-10-14 08:13:50,139 - handlers.py[DEBUG]: finish: init-network/config-growpart: SUCCESS: config-growpart ran successfully
```

Executing it by hand I can see:
```
$ sudo growpart --dry-run /dev/nvme0n1 2 -v
update-partition set to true
resizing 2 on /dev/nvme0n1 using resize_sfdisk_gpt
62914560 sectors of 512. total size=32212254720 bytes
## sfdisk --unit=S --dump /dev/nvme0n1
label: gpt
label-id: E97B9FFA-2C13-474E-A0E4-ABF1572CD20C
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 62914526

/dev/nvme0n1p1 : start= 2048, size= 1228800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=4C880299-6848-4FDE-9479-12D8F1410F6E, name="EFI System Partition"
/dev/nvme0n1p2 : start= 1230848, size= 16281600, type=0FC63DAF-8483-4772-8E79-3D69D8477DE4, uuid=7CA2D98D-3FD7-4B25-9400-854C5D9CC3F7
/dev/nvme0n1p3 : start= 17512448, size= 2048, type=21686148-6449-6E6F-744E-656564454649, uuid=69C14DB7-5A4D-4014-A565-5A894712AF1D
max_end=17512448 tot=62914560 pt_end=17512448 pt_start=1230848 pt_size=16281600
NOCHANGE: partition 2 is size 16281600. it cannot be grown
```

the problem seems the fact that there is the 3rd partition that start right after the 2nd so the 2nd cannot be resized.

```
$ sudo parted -l
Warning: Not all of the space available to /dev/nvme0n1 appears to be used, you
can fix the GPT to use all of the space (an extra 41943040 blocks) or continue
with the current setting?
Fix/Ignore? I
Model: NVMe Device (nvme)
Disk /dev/nvme0n1: 32.2GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1049kB 630MB 629MB fat32 EFI System Partition boot, esp
 2 630MB 8966MB 8336MB xfs
 3 8966MB 8967MB 1049kB bios_grub
```

Additional Information# x86_64 instance c5.xlarge

I launched an empty Cento8 x86_64 instance, starting from ami-0bfa4fefe067b7946 in eu-west-1, to see the differences.

```
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.7G 0 3.7G 0% /dev
tmpfs 3.7G 0 3.7G 0% /dev/shm
tmpfs 3.7G 8.4M 3.7G 1% /run
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/nvme0n1p2 30G 1.5G 29G 5% /
tmpfs 746M 0 746M 0% /run/user/1000

$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
├─nvme0n1p1 259:1 0 1M 0 part
└─nvme0n1p2 259:2 0 30G 0 part /
```

In cloud-init.log I can see that the command is executed correctly
```
2020-10-14 08:23:30,023 - util.py[DEBUG]: Running command ['growpart', '--dry-run', '/dev/nvme0n1', '2'] with allowed return codes [0] (shell=False, capture=True)
2020-10-14 08:23:30,081 - util.py[DEBUG]: Running command ['growpart', '/dev/nvme0n1', '2'] with allowed return codes [0] (shell=False, capture=True)
2020-10-14 08:23:30,397 - util.py[DEBUG]: resize_devices took 0.389 seconds
2020-10-14 08:23:30,397 - cc_growpart.py[INFO]: '/' resized: changed (/dev/nvme0n1, 2) from 8336179200 to 32210140672
2020-10-14 08:23:30,397 - handlers.py[DEBUG]: finish: init-network/config-growpart: SUCCESS: config-growpart ran successfully

$ sudo sfdisk --unit=S --dump /dev/nvme0n1
label: dos
label-id: 0xa323d5eb
device: /dev/nvme0n1
unit: sectors
/dev/nvme0n1p1 : start= 2048, size= 2048, type=83
/dev/nvme0n1p2 : start= 4096, size= 62910431, type=83, bootable
[centos@ip-172-31-3-176 ~]$ sudo parted -l
Model: NVMe Device (nvme)
Disk /dev/nvme0n1: 32.2GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags:

Number Start End Size Type File system Flags
 1 1049kB 2097kB 1049kB primary
 2 2097kB 32.2GB 32.2GB primary xfs boot
```

# RedHat 8 ARM AMI on us-east-1 - ami-07bd28c96286169fa

I verified the RedHat 8 ami in us-east-1 and the partitions are in the correct order and the root volume has the right size.
```
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.6G 0 3.6G 0% /dev
tmpfs 3.7G 0 3.7G 0% /dev/shm
tmpfs 3.7G 22M 3.6G 1% /run
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/nvme0n1p3 30G 1.1G 29G 4% /
/dev/nvme0n1p2 507M 130M 377M 26% /boot
/dev/nvme0n1p1 200M 6.2M 194M 4% /boot/efi
tmpfs 740M 0 740M 0% /run/user/1000

$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
├─nvme0n1p1 259:1 0 200M 0 part /boot/efi
├─nvme0n1p2 259:2 0 512M 0 part /boot
└─nvme0n1p3 259:3 0 29.3G 0 part /

$ curl http://169.254.169.254/latest/meta-data/ami-id
ami-07bd28c96286169fa
```

# Centos 8 ARM AMI on us-east-1 - ami-0c7ec43a152e9c107

I verified the 2 amis available for centos8 on us-east-1

```
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.7G 0 3.7G 0% /dev
tmpfs 3.7G 0 3.7G 0% /dev/shm
tmpfs 3.7G 22M 3.7G 1% /run
tmpfs 3.7G 0 3.7G 0% /sys/fs/cgroup
/dev/nvme0n1p2 7.8G 1.4G 6.5G 17% /
/dev/nvme0n1p1 599M 6.4M 593M 2% /boot/efi
tmpfs 746M 0 746M 0% /run/user/1000

$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
├─nvme0n1p1 259:1 0 600M 0 part /boot/efi
├─nvme0n1p2 259:2 0 7.8G 0 part /
└─nvme0n1p3 259:3 0 1M 0 part

$ curl http://169.254.169.254/latest/meta-data/ami-id
ami-0c7ec43a152e9c107
```

# Centos 8 ARM AMI on us-east-1 - ami-05c0f629cac7683bb

```
$ df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.6G 0 7.6G 0% /dev
tmpfs 7.6G 0 7.6G 0% /dev/shm
tmpfs 7.6G 22M 7.6G 1% /run
tmpfs 7.6G 0 7.6G 0% /sys/fs/cgroup
/dev/nvme0n1p2 7.8G 1.4G 6.5G 17% /
/dev/nvme0n1p1 599M 6.4M 593M 2% /boot/efi
tmpfs 1.6G 0 1.6G 0% /run/user/1000

$ sudo lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme0n1 259:0 0 30G 0 disk
├─nvme0n1p1 259:1 0 600M 0 part /boot/efi
├─nvme0n1p2 259:2 0 7.8G 0 part /
└─nvme0n1p3 259:3 0 1M 0 part

$ curl http://169.254.169.254/latest/meta-data/ami-id
ami-05c0f629cac7683bb
```
TagsAmazon, AMI, ARM, ec2

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2020-10-15 16:02 usai New Issue
2020-10-15 16:02 usai Tag Attached: Amazon
2020-10-15 16:02 usai Tag Attached: AMI
2020-10-15 16:02 usai Tag Attached: ec2
2020-10-15 16:02 usai Tag Attached: ARM