View Issue Details

IDProjectCategoryView StatusLast Update
0013836CentOS-7Cloud-Imagespublic2018-04-19 20:17
Reporterfmbiete 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
PlatformAmazon Web ServicesOSCentOSOS Version7.4
Product Version 
Target VersionFixed in Version 
Summary0013836: RHBA-2017:2283 - AWS AMI rebuild with ENA support
DescriptionWe need to rebuild the CentOS 7 AWS AMI to include the latest fixes and improvements from upstream:

https://access.redhat.com/errata/RHBA-2017:2283
https://bugzilla.redhat.com/show_bug.cgi?id=1410047

The changes from 7.3 are not included either.
TagsNo tags attached.
abrt_hash
URL

Activities

fmbiete

fmbiete

2017-10-14 10:45

reporter   ~0030373

This only requires adding 2 flags to the AMI build process, when registering the image into AWS.

--sriov-net-support simple --ena-support

No change in the kickstart part is required

Example:

aws ec2 register-image --name 'CentOS-7 HVM with updates' --description 'fixed networking support' --virtualization-type hvm --root-device-name /dev/xvda1 --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs": { "SnapshotId": "snap-whatever", "VolumeSize":8, "DeleteOnTermination": false, "VolumeType": "gp2"}}]' --architecture x86_64 --sriov-net-support simple --ena-support
fmbiete

fmbiete

2017-12-18 08:40

reporter   ~0030775

No activity in this ticket, but AMI 1708_11.01 includes those attributes (ENA & SRIOV support). I suppose someone noticed that CentOS AMI couldn't run in the new instance types. Is this bugzilla even read by the Cloud/Virt members?

{
            "ProductCodes": [
                {
                    "ProductCodeId": "aw0evgkw8e5c1q413zgy5pjce",
                    "ProductCodeType": "marketplace"
                }
            ],
            "Description": "CentOS Linux 7 x86_64 HVM EBS 1708_11.01",
            "VirtualizationType": "hvm",
            "Hypervisor": "xen",
            "ImageOwnerAlias": "aws-marketplace",
            "EnaSupport": true,
            "SriovNetSupport": "simple",
            "ImageId": "ami-192a9460",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/sda1",
                    "Ebs": {
                        "Encrypted": false,
                        "DeleteOnTermination": false,
                        "VolumeType": "standard",
                        "VolumeSize": 8,
                        "SnapshotId": "snap-013406753fcf8e3df"
                    }
                }
            ],
            "Architecture": "x86_64",
            "ImageLocation": "aws-marketplace/CentOS Linux 7 x86_64 HVM EBS 1708_11.01-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-95096eef.4",
            "RootDeviceType": "ebs",
            "OwnerId": "679593333241",
            "RootDeviceName": "/dev/sda1",
            "CreationDate": "2017-12-05T14:49:45.000Z",
            "Public": true,
            "ImageType": "machine",
            "Name": "CentOS Linux 7 x86_64 HVM EBS 1708_11.01-b7ee8a69-ee97-4a49-9e68-afaee216db2e-ami-95096eef.4"
        },
arronax

arronax

2017-12-27 11:02

reporter   ~0030829

Stumbled upon this issue while investigating ENA on the older AMIs, so I think it could be useful to update it. There are quite a lot of other issues on C5/M5 in the bug list, too, but this is the one popping up on `centos7 ena` search.


Short summary: some instances started with ami-192a9460 may show connectivity issues. Can be fixed by restart or updating network device names.


As fmbiete writes, ami-192a9460 does include necessary flags, and it even has everything for ENA in the system itself, but it MAY NOT properly work with newer instance types without some fixes.

Sometimes, if you spin up an m5 instance off ami-192a9460, it will start up (no complains on ENA from AWS) but you won't be able to connect to it. AWS EC2 console shows "Instance reachability check failed"

I started up a bunch of instances, most m5, and one c5, and hit the issue twice. Actually, I hit the issue on the first one I started, so that's why I looked into it. I later hit this again on 5th instance started.

I checked this a bit and was able to make ENA work on "bad" instance started from ami-192a9460, at least according to checks provided by AWS documentation.

Documentation is here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

According to the doc, system might show connection issues if there's no eth0 device present. I changed instance type to m4 and confirmed device is named ens3. Renamed ens3 to eth0 and disabled Predictable Network Interface Names. After that, once I made sure that cloud-init is happy with new device name and generates ifcfg-eth0 properly, I changed instance type again to m5. Instance then started up and I was able to connect to it, and it generally seems to be fine.

I also noted following: when m5 instance starts up properly, device name seems to always be ens5.

Later I found that restarting a faulty instance may also fix it, and device name is ens5 after that. This makes me believe that another restart might actually break this again, if device name changes. However, few restarts didn't show this behavior, so maybe I'm wrong.


It seems that updating AMI and setting default device name to eth0 would be the best way to prevent instances from being faulty in the first place.


Following steps were needed to make "bad" instance based on ami-192a9460 work properly:
* start instance as m4 or anything non-ENA
* add `net.ifnames=0` to `GRUB_CMDLINE_LINUX` line in `/etc/default/grub` and regenerate grub config with `grub2-mkconfig -o /boot/grub2/grub.cfg`
* update `/etc/udev/rules.d/70-persistent-net.rules` and rename device to eth0
* change instance type to m5 and start it
* check that ena is used as a driver for eth0: `ethtool -i eth0`

Note: not sure disabling predictable interface names is even necessary, but it's in the docs.
siebrand

siebrand

2018-01-23 20:57

reporter   ~0031010

> AMI 1708_11.01 includes those attributes (ENA & SRIOV support)

Unfortunately the latest CentOS images (1801_01) do not appear to contain these settings, as they cannot be used with instance type m5 on AWS.

(I've tried to look really hard for a up to date changelog or release notes for the cloud images, but I wasn't able to find any -- expected was https://wiki.centos.org/Manuals/ReleaseNotes/CentOS7 or similar, but that doesn't appear to mention any cloud image builds).
luckyknight

luckyknight

2018-01-30 16:19

reporter   ~0031112

Can confirm that the latest AWS image does not contain these settings and will not let me build a C5 or M5 server due to lack of ENA.

This is the message from AWS:

"Instance type is disabled.
This instance type requires ENA support. To enable this instance type, return to the previous step and select an AMI that is enabled for ENA."
nan008

nan008

2018-02-05 15:32

reporter   ~0031157

Any news if this issue will be fixed soon?

M5s are cheaper and we would like to replace the existing machines we have with m5. They all run Centos 7 latest build

We are getting the same error as luckyknight above my post after trying to launch the ami-6e28b517 and ami-4bf3d731 (1801_01) from AWS Marketplace. M5/C5 are greyed out lacking ENA support
icemanncsu

icemanncsu

2018-02-09 21:48

reporter   ~0031200

You can use AMI: ami-02e98f78 in the mean time. Just add this to your user-data and it will be updated to 1801 and rebooted ready to use!

---------
package_upgrade: true

power_state:
  delay: "now"
  mode: reboot
  message: Rebooting-System
  timeout: 30
  condition: True
---------
icemanncsu

icemanncsu

2018-03-12 15:56

reporter   ~0031415

Is there no update on this issue?
siebrand

siebrand

2018-03-12 16:16

reporter   ~0031416

> Is there no update on this issue?

Apparently not. I've tried to find someone on IRC, no luck. I've sent a tweet to someone who thought might have updated the wiki page about CentOS cloud images recently, and no reply. I'm at a loss on who to even poke.
icemanncsu

icemanncsu

2018-03-29 17:47

reporter   ~0031525

How does an issue with Priority: high & Severity: major, go months with no action?
BlueH2O

BlueH2O

2018-04-03 20:56

reporter   ~0031552

Hoping someone blows the dust off this report soon!
jwitko

jwitko

2018-04-04 17:02

reporter   ~0031561

Hi, I've also been waiting for this fix since October 2017. It would be greatly appreciated to be applied to CentOS 6 as well.
lucy

lucy

2018-04-08 15:40

reporter   ~0031585

I noticed that version 1803_01 was added to the AWS marketplace, and it supports ENA!
AMI ID in N.Virginia is ami-b81dbfc5

Did anyone try it already?
jwitko

jwitko

2018-04-08 16:17

reporter   ~0031586

Yes, I have the AWS ENA drivers installed and running on M5 instances with CentOS 7 1803_01 AMI.

It does not appear that CentOS 6 1803_01 has the requirements to use the ENA drivers.
lucy

lucy

2018-04-08 17:01

reporter   ~0031587

When you say that you have the drivers installed, you mean that you installed them manually, or that you confirm that the ones that come with the image by default work properly?
Also, does anyone know where the release notes of CentOS 7 1803_01 are? or at least some list of changes from the previous version?
Thanks!
jordan.davies

jordan.davies

2018-04-19 20:17

reporter   ~0031641

This also applies to the x1e instance types.

Any update on when CentOS 7 will be compatible with the newer instance types?

Issue History

Date Modified Username Field Change
2017-09-17 10:45 fmbiete New Issue
2017-10-14 10:45 fmbiete Note Added: 0030373
2017-12-18 08:40 fmbiete Note Added: 0030775
2017-12-27 11:02 arronax Note Added: 0030829
2018-01-23 20:57 siebrand Note Added: 0031010
2018-01-30 16:19 luckyknight Note Added: 0031112
2018-02-05 15:32 nan008 Note Added: 0031157
2018-02-09 21:48 icemanncsu Note Added: 0031200
2018-03-12 15:56 icemanncsu Note Added: 0031415
2018-03-12 16:16 siebrand Note Added: 0031416
2018-03-29 17:47 icemanncsu Note Added: 0031525
2018-04-03 20:56 BlueH2O Note Added: 0031552
2018-04-04 17:02 jwitko Note Added: 0031561
2018-04-08 15:40 lucy Note Added: 0031585
2018-04-08 16:17 jwitko Note Added: 0031586
2018-04-08 17:01 lucy Note Added: 0031587
2018-04-19 20:17 jordan.davies Note Added: 0031641