View Issue Details

IDProjectCategoryView StatusLast Update
0014760CentOS-7Cloud-Imagespublic2018-12-15 17:16
Reporterkyang 
PriorityurgentSeverityminorReproducibilityalways
Status newResolutionopen 
Platformx86_64OSCentOS LinuxOS Version7.5.1804
Product Version7.5.1804 
Target VersionFixed in Version 
Summary0014760: cloud-init failed and crash the server after update to CentOS 7.5
DescriptionI updated my AWS server to CentOS 7.5, and after doing a reboot I cannot make any connection to my server and it failed in instance status checks. then I have a look at the system log and found out it is cause cloud-init.

I enabled ipv6 on my server following the AWS Userguide (https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-migrate-ipv6.html#ipv6-dhcpv6-rhel)

--------------------------------------------------
network:
  version: 1
  config:
  - type: physical
    name: eth0
    subnets:
      - type: dhcp6
--------------------------------------------------

Add this setting to /etc/cloud/cloud.cfg.d works well on CentOS 7.4,
but after update to 7.5, I found under this setting, cloud-init will generate a bad configure in /etc/sysconfig/network-scripts/ifcfg-eth0 as follow

--------------------------------------------------
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=none
DEVICE=eth0
DHCPV6C=yes
IPV6INIT=yes
IPV6_AUTOCONF=no
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
------------------------------------------------------

the "BOOTPROTO=none" disable dhcpv4 client and the server will nerver get ipv4 address during boot, and make server can not be connected. I disabled cloud-init and reconfigure ifcfg-eth0 with following and it works very well.

--------------------------------------------------
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
DHCPV6C=yes
IPV6INIT=yes
IPV6_AUTOCONF=yes
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
------------------------------------------------------
Steps To Reproduce1. update a server in AWS from CentOS 7.4 to CentOS 7.5

2. enable ipv6 by add following configure in /etc/cloud/ cloud.cfg.d/99_custom_networking.cfg

--------------------------------------------------
network:
  version: 1
  config:
  - type: physical
    name: eth0
    subnets:
      - type: dhcp6
--------------------------------------------------

3. reboot
Additional InformationI found the source in /usr/lib/python2.7/site-packages/cloudinit/net/sysconfig.py

279 @classmethod
280 def _render_subnets(cls, iface_cfg, subnets):
281 # setting base values
282 iface_cfg['BOOTPROTO'] = 'none'
283
284 # modifying base values according to subnets
285 for i, subnet in enumerate(subnets, start=len(iface_cfg.children)):
286 subnet_type = subnet.get('type')
287 if subnet_type == 'dhcp6':
288 iface_cfg['IPV6INIT'] = True
289 iface_cfg['DHCPV6C'] = True
290 iface_cfg['IPV6_AUTOCONF'] = False
291 elif subnet_type in ['dhcp4', 'dhcp']:
292 iface_cfg['BOOTPROTO'] = 'dhcp'
293 elif subnet_type == 'static':
294 # grep BOOTPROTO sysconfig.txt -A2 | head -3
295 # BOOTPROTO=none|bootp|dhcp
296 # 'bootp' or 'dhcp' cause a DHCP client
297 # to run on the device. Any other
298 # value causes any static configuration
299 # in the file to be applied.
300 # ==> the following should not be set to 'static'
301 # but should remain 'none'
302 # if iface_cfg['BOOTPROTO'] == 'none':
303 # iface_cfg['BOOTPROTO'] = 'static'
304 if _subnet_is_ipv6(subnet):
305 iface_cfg['IPV6INIT'] = True
306 else:
307 raise ValueError("Unknown subnet type '%s' found"
308 " for interface '%s'" % (subnet_type,
309 iface_cfg.name))

in subnet_type == 'dhcp6', there is no code to add BOOTPROTO=dhcp
Tags7.5, cloud-init
abrt_hash
URL

Activities

pomac

pomac

2018-05-25 11:29

reporter   ~0031915

Same bug - this is a trace from what happens when booting on OpenStack.

-- Logs begin at fre 2018-05-25 10:51:00 UTC, end at fre 2018-05-25 10:53:55 UTC. --
maj 25 10:51:07 centos-1.novalocal systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: Cloud-init v. 0.7.9 running 'init' at Fri, 25 May 2018 10:51:08 +0000. Up 16.59 seconds.
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: ++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: +--------+------+-----------+-------------+-------+-------------------+
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: +--------+------+-----------+-------------+-------+-------------------+
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | lo: | True | . | . | d | . |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | eth0: | True | 10.0.0.15 | 255.255.0.0 | . | fa:16:3e:1d:45:00 |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | eth0: | True | . | . | d | fa:16:3e:1d:45:00 |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: +--------+------+-----------+-------------+-------+-------------------+
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: ++++++++++++++++++++++++++++++Route IPv4 info+++++++++++++++++++++++++++++++
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | 0 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.0.0 | eth0 | U |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: | 2 | 169.254.169.254 | 10.0.0.4 | 255.255.255.255 | eth0 | UGH |
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: 2018-05-25 10:51:08,646 - util.py[WARNING]: failed stage init
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: failed run of stage init
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ------------------------------------------------------------
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: Traceback (most recent call last):
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: File "/usr/lib/python2.7/site-packages/cloudinit/cmd/main.py", line 513, in status_wrapper
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ret = functor(name, args)
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: File "/usr/lib/python2.7/site-packages/cloudinit/cmd/main.py", line 281, in main_init
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: init.update()
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 357, in update
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: self._store_vendordata()
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: File "/usr/lib/python2.7/site-packages/cloudinit/stages.py", line 381, in _store_vendordata
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: util.write_file(self._get_ipath('vendordata_raw'), raw_vd, 0o600)
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 1755, in write_file
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: content = encode_text(content)
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: File "/usr/lib/python2.7/site-packages/cloudinit/util.py", line 154, in encode_text
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: return text.encode(encoding)
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: AttributeError: 'dict' object has no attribute 'encode'
maj 25 10:51:08 centos-1.novalocal cloud-init[787]: ------------------------------------------------------------
maj 25 10:51:08 centos-1.novalocal systemd[1]: cloud-init.service: main process exited, code=exited, status=1/FAILURE
maj 25 10:51:08 centos-1.novalocal systemd[1]: Failed to start Initial cloud-init job (metadata service crawler).
maj 25 10:51:08 centos-1.novalocal systemd[1]: Unit cloud-init.service entered failed state.
maj 25 10:51:08 centos-1.novalocal systemd[1]: cloud-init.service failed.
ldennison

ldennison

2018-05-25 23:50

reporter   ~0031918

This is related to or the same as #0014585.

## TL;DR version: As a workaround, I used the below cloud-init configuration instead of AWS' listed cloud-init configuration, and my CentOS 7.5 machine is reachable with IPv4 and IPv6.
--------------------------------------------------
network:
  version: 1
  config:
  - type: physical
    name: eth0
    subnets:
      - type: dhcp4
      - type: dhcp6

bootcmd:
  - "sed -i 's/^IPV6_AUTOCONF=.*$/IPV6_AUTOCONF=yes/g' /etc/sysconfig/network-scripts/ifcfg-eth0"
  - systemctl restart network
--------------------------------------------------


## Long version:
I encountered this problem when trying to deploy a CentOS 7.5 EC2 instance (1804_2) and attempted to find a "better" workaround than just disabling cloud-init network config. I don't presume to be an expert at this, but I did get my instances working with IPv6 and cloud-init again. The following describes the steps I worked through.

AWS lists this cloud-init config for RHEL/CentOS 7.4 (as originally posted on this ticket):
--------------------------------------------------
network:
  version: 1
  config:
  - type: physical
    name: eth0
    subnets:
      - type: dhcp6
--------------------------------------------------

Here's an updated configuration that results in the instance still being reachable over IPv4. Additionally, the IPv6 seems to get allocated to the interface (as reported by ifconfig). The difference is just an added line under subnets to specifically define both types (IPv4 and IPv6).
--------------------------------------------------
network:
  version: 1
  config:
  - type: physical
    name: eth0
    subnets:
      - type: dhcp4
      - type: dhcp6
--------------------------------------------------

However, none of the instances I tested with this configuration were actually reachable over IPv6 despite the interface listing the proper IPv6 attached. So after a lot of tweaking and testing, I narrowed that problem down to the "IPV6_AUTOCONF=no" value being written to /etc/sysconfig/network-scripts/ifcfg-eth0. Here's what that file looked like with the updated cloud-init config above:
--------------------------------------------------
# Created by cloud-init on instance boot automatically, do not edit.
#
BOOTPROTO=dhcp
DEVICE=eth0
DHCPV6C=yes
IPV6INIT=yes
IPV6_AUTOCONF=no
ONBOOT=yes
TYPE=Ethernet
USERCTL=no
--------------------------------------------------

I came up with the following final cloud-init configuration to set "IPV6_AUTOCONF=yes" on each boot, which results in the instance being reachable over IPv6. It's "ugly", but it seems to work without issue. It obviously requires the sed binary to do a string replacement on the ifcfg-eth0 file after cloud-init generates it.
--------------------------------------------------
network:
  version: 1
  config:
  - type: physical
    name: eth0
    subnets:
      - type: dhcp4
      - type: dhcp6

bootcmd:
  - "sed -i 's/^IPV6_AUTOCONF=.*$/IPV6_AUTOCONF=yes/g' /etc/sysconfig/network-scripts/ifcfg-eth0"
  - systemctl restart network
--------------------------------------------------
irvingpop

irvingpop

2018-12-15 17:15

reporter   ~0033357

I've noted the same thing on EC2, it happens if you've disabled NetworkManager only.
irvingpop

irvingpop

2018-12-15 17:16

reporter   ~0033358

I should add, this also happens on CentOS 7.6. I've discovered this because I'm creating my own minimal images.

Issue History

Date Modified Username Field Change
2018-05-10 15:58 kyang New Issue
2018-05-10 15:58 kyang Tag Attached: 7.5
2018-05-10 15:58 kyang Tag Attached: cloud-init
2018-05-25 11:29 pomac Note Added: 0031915
2018-05-25 23:50 ldennison Note Added: 0031918
2018-12-15 17:15 irvingpop Note Added: 0033357
2018-12-15 17:16 irvingpop Note Added: 0033358