View Issue Details

IDProjectCategoryView StatusLast Update
0014585Cloud Instance SIGcloud-initpublic2018-07-11 08:40
Reporterekgermann 
PrioritynormalSeveritymajorReproducibilityalways
Status newResolutionopen 
Platformx86_64 on AWSOSCentOS 7 (x86_64) - with UpdatesOS Version1801_01
Summary0014585: Latest yum update bricks all network reachability with IPv6 enabled on AWS
DescriptionRunning "yum update" bricks network connectivity for AWS instances which have IPv6 enabled per the AWS docs.

Image is the "CentOS 7 (x86_64) - with Updates HVM" AMI

Unfortunately, if you update a production instance without a snap, you have no console access to try and recover it by disabling IPv6.

Did a lot of testing over the past several days and it seems to be related to cloud-init and the config of the network stack. In the steps to reproduce, there are two scenarios, one where it doesn't work and a workaround which bypasses cloud-init and does work.

Versions of kernel and cloud-init are in the "Steps" dialogue.
Steps To ReproduceLaunch t2.nano instance in region (Ohio)
  CentOS 7 (x86_64) - with Updates HVM
  Version 1801_01
  
Assign IPv6 address to instance via Actions -> Networking -> Manage IP Addresses (Auto assign)

Configuration at this point
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
            inet 100.86.16.176 netmask 255.255.255.0 broadcast 100.86.16.255
            inet6 fe80::8e2:f9ff:feec:ed8 prefixlen 64 scopeid 0x20<link>
            ether 0a:e2:f9:ec:0e:d8 txqueuelen 1000 (Ethernet)
            RX packets 10449 bytes 14627521 (13.9 MiB)
            RX errors 0 dropped 0 overruns 0 frame 0
            TX packets 3162 bytes 265467 (259.2 KiB)
            TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

Configure instance to pick up IPv6 address according to AWS docs
  (https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-migrate-ipv6.html#ipv6-dhcpv6-rhel)

Reboot to pick up IPv6 address
  
Configuration at this point
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
            inet 100.86.16.176 netmask 255.255.255.0 broadcast 100.86.16.255
            inet6 2600:1f16:940:9410:3321:d25:dde0:493e prefixlen 64 scopeid 0x0<global>
            inet6 fe80::8e2:f9ff:feec:ed8 prefixlen 64 scopeid 0x20<link>
            ether 0a:e2:f9:ec:0e:d8 txqueuelen 1000 (Ethernet)
            RX packets 225 bytes 26736 (26.1 KiB)
            RX errors 0 dropped 0 overruns 0 frame 0
            TX packets 255 bytes 29282 (28.5 KiB)
            TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
            
Versions at this point:
    cloud-init-0.7.9-9.el7.centos.2.x86_64
    
Run update
    yum update -y
    
Version installed:
    cloud-init-0.7.9-9.el7.centos.6.x86_64
    
Reboot
    
Instance is now unreachable on IPv4 and IPv6

"Get instance screenshot" shows system console at login prompt

Kernel shown is 3.10.0-693.21.1.el7.x86_64

Terminate bricked instance

===============================================================================================

Launch t2.nano instance in region (Ohio)
  CentOS 7 (x86_64) - with Updates HVM
  Version 1801_01
  
Assign IPv6 address to instance via Actions -> Networking -> Manage IP Addresses (Auto assign)

Configuration at this point
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
            inet 100.86.16.92 netmask 255.255.255.0 broadcast 100.86.16.255
            inet6 fe80::8fc:83ff:fed0:1790 prefixlen 64 scopeid 0x20<link>
            ether 0a:fc:83:d0:17:90 txqueuelen 1000 (Ethernet)
            RX packets 296 bytes 33314 (32.5 KiB)
            RX errors 0 dropped 0 overruns 0 frame 0
            TX packets 296 bytes 32800 (32.0 KiB)
            TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
            
Configure instance to pick up IPv6 address via following process
    References : https://serverfault.com/questions/866696/how-do-i-enable-ipv6-in-rhel-7-4-on-amazon-ec2
                    https://www.rootusers.com/configure-ipv6-addresses-and-basic-troubleshooting-in-linux/
                    
    Create /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg and add
        network: {config: disabled}
    
    Append to /etc/sysconfig/network-scripts/ifcfg-eth0
        DHCPV6C=yes
        IPV6INIT=no
        IPV6_AUTOCONF=yes
        IPV6_DEFROUTE=yes
        IPV6_FAILURE_FATAL=no
        IPV6_PEERDNS=yes
        IPV6_PEERROUTES=yes
        

Reboot to pick up IPv6 address

Configuration at this point
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
            inet 100.86.16.92 netmask 255.255.255.0 broadcast 100.86.16.255
            inet6 fe80::8fc:83ff:fed0:1790 prefixlen 64 scopeid 0x20<link>
            inet6 2600:1f16:940:9410:8eed:895e:d270:5d09 prefixlen 64 scopeid 0x0<global>
            ether 0a:fc:83:d0:17:90 txqueuelen 1000 (Ethernet)
            RX packets 254 bytes 29554 (28.8 KiB)
            RX errors 0 dropped 0 overruns 0 frame 0
            TX packets 274 bytes 31116 (30.3 KiB)
            TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
            
Versions at this point
    cloud-init-0.7.9-9.el7.centos.2.x86_64
    
Run update
    yum update -y

Version installed:
    cloud-init-0.7.9-9.el7.centos.6.x86_64
    
Reboot

Instance is reachable

Config at this point
    eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
            inet 100.86.16.92 netmask 255.255.255.0 broadcast 100.86.16.255
            inet6 fe80::8fc:83ff:fed0:1790 prefixlen 64 scopeid 0x20<link>
            inet6 2600:1f16:940:9410:8eed:895e:d270:5d09 prefixlen 64 scopeid 0x0<global>
            ether 0a:fc:83:d0:17:90 txqueuelen 1000 (Ethernet)
            RX packets 243 bytes 28561 (27.8 KiB)
            RX errors 0 dropped 0 overruns 0 frame 0
            TX packets 276 bytes 30844 (30.1 KiB)
            TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
        
Kernel shown is 3.10.0-693.21.1.el7.x86_64

Additional InformationVerified reproducible in Ohio, Northern VA and Paris regions

Verified on t2.nano and m4.2xlarge
TagsNo tags attached.

Activities

ldennison

ldennison

2018-05-25 23:56

reporter   ~0031919

This is related to or the same as ~14760. The notes on that ticket include helpful information to get around this problem without disabling IPv6 functionality.

Issue History

Date Modified Username Field Change
2018-03-15 20:14 ekgermann New Issue
2018-05-25 23:56 ldennison Note Added: 0031919