View Issue Details

IDProjectCategoryView StatusLast Update
0017240administrationsecuritypublic2020-04-11 02:50
Reportercchekuri 
PriorityhighSeveritymajorReproducibilityalways
Status newResolutionopen 
Product Version 
Target VersionFixed in Version 
Summary0017240: CentOS Instances do not complete Cloud-Init (SSH Keys)
Description if CentOS went to a version of cloud-init that is not working correctly. Basically it looks like newer versions of cloud aware CentOS images fail cloud-init because its injecting a local only interface that it tries to use to pull metadata, which eventually fails and no ssh key gets injected.
Symptoms and Ticket Summary
Steps To Reproduceit seems to be faulty behavior of cloud-init, it should not add link local addresses to route table. see https://bugzilla.redhat.com/show_bug.cgi?id=1558641, practically the same issue. and it was fixed for fedora. maybe now centos has this issue again, even tho the cloud-init version is higher in our case.

And the details for this instance is below

nova show 390ba7a9-164e-403e-856b-af613a61deb4

+------------------------------------------+-------------------------------------------------------------------+

| Property | Value |

+------------------------------------------+-------------------------------------------------------------------+

| OS-DCF:diskConfig | AUTO |

| OS-EXT-AZ:availability_zone | qa2 |

| OS-EXT-SRV-ATTR:host | xxxx |

| OS-EXT-SRV-ATTR:hostname | centos7-1811-security-group-testvm-1 |

| OS-EXT-SRV-ATTR:hypervisor_hostname | xxxx |

| OS-EXT-SRV-ATTR:instance_name | instance-000018bf |

| OS-EXT-SRV-ATTR:kernel_id | |

| OS-EXT-SRV-ATTR:launch_index | 0 |

| OS-EXT-SRV-ATTR:ramdisk_id | |

| OS-EXT-SRV-ATTR:reservation_id | r-z9ou8gsy |

| OS-EXT-SRV-ATTR:root_device_name | /dev/vda |

| OS-EXT-SRV-ATTR:user_data | - |

| OS-EXT-STS:power_state | 1 |

| OS-EXT-STS:task_state | - |

| OS-EXT-STS:vm_state | active |

| OS-SRV-USG:launched_at | 2020-03-16T15:04:56.000000 |

| OS-SRV-USG:terminated_at | - |

| P6 RISE-SS-DATA Network (Shared) network | 10.30.85.3 |

| accessIPv4 | |

| accessIPv6 | |

| config_drive | |

| created | 2020-03-16T15:04:29Z |

| description | - |

| flavor:disk | 20 |

| flavor:ephemeral | 0 |

| flavor:extra_specs | {"hw:cpu_policy": "dedicated", "hw:cpu_thread_policy": "require"} |

| flavor:original_name | m1.small |

| flavor:ram | 2048 |

| flavor:swap | 0 |

| flavor:vcpus | 2 |

| hostId | e6fb9b1fca03487495be216cad660203dfaa0ce9af586749f56876b3 |

| host_status | UP |

| id | 390ba7a9-164e-403e-856b-af613a61deb4 |

| image | CentOS7Cloud-1811 (d8f7609a-73b2-44dd-b0c3-ef6eec00e468) |

| key_name | relmore |

| locked | False |

| metadata | {} |

| name | CentOS7-1811-Security-Group-TESTVM-1 |

| os-extended-volumes:volumes_attached | [] |

| progress | 0 |

| security_groups | default |

| status | ACTIVE |

| tags | [] |

| tenant_id | d37f964c75fa402fa4c6e603f1cf6c40 |

| updated | 2020-03-16T15:12:45Z |

| user_id | f8e50f79718e46f7b9dc0004290f2194 |



 

1805 Build (note no 169.254.169.254 interface):

 

[ 7.406156] cloud-init[838]: Cloud-init v. 0.7.9 running 'init' at Fri, 13 Mar 2020 14:13:13 +0000. Up 7.37 seconds.

[[32m OK [0m] Started Dynamic System Tuning Daemon.

[ 7.446127] cloud-init[838]: ci-info: ++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++

[ 7.453628] cloud-init[838]: ci-info: +--------+------+------------+-----------------+-------+-------------------+

[ 7.458276] cloud-init[838]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |

[ 7.462687] cloud-init[838]: ci-info: +--------+------+------------+-----------------+-------+-------------------+

[ 7.467543] cloud-init[838]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |

[ 7.480549] cloud-init[838]: ci-info: | lo: | True | . | . | d | . |

[ 7.493483] cloud-init[838]: ci-info: | eth0: | True | 10.30.85.5 | 255.255.255.192 | . | 02:9f:e7:b1:19:94 |

[ 7.503540] cloud-init[838]: ci-info: | eth0: | True | . | . | d | 02:9f:e7:b1:19:94 |

[ 7.515506] cloud-init[838]: ci-info: +--------+------+------------+-----------------+-------+-------------------+

[ 7.521945] cloud-init[838]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++

[ 7.529501] cloud-init[838]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+

[ 7.537183] cloud-init[838]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |

[ 7.543571] cloud-init[838]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+

[ 7.551476] cloud-init[838]: ci-info: | 0 | 0.0.0.0 | 10.30.85.1 | 0.0.0.0 | eth0 | UG |

[ 7.558422] cloud-init[838]: ci-info: | 1 | 10.30.85.0 | 0.0.0.0 | 255.255.255.192 | eth0 | U |

[ 7.568302] cloud-init[838]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+

 

 

1811 Build:

 

[ 7.647102] cloud-init[916]: Cloud-init v. 18.2 running 'init' at Fri, 13 Mar 2020 01:41:41 +0000. Up 7.61 seconds.

[ 7.682098] cloud-init[916]: ci-info: +++++++++++++++++++++++++++++++Net device info+++++++++++++++++++++++++++++++

[ 7.692571] cloud-init[916]: ci-info: +--------+------+-------------+-----------------+-------+-------------------+

[ 7.706564] cloud-init[916]: ci-info: | Device | Up | Address | Mask | Scope | Hw-Address |

[ 7.716568] cloud-init[916]: ci-info: +--------+------+-------------+-----------------+-------+-------------------+

[ 7.726587] cloud-init[916]: ci-info: | eth0: | True | 10.30.85.11 | 255.255.255.192 | . | 02:da:6d:36:19:9c |

[ 7.736628] cloud-init[916]: ci-info: | eth0: | True | . | . | d | 02:da:6d:36:19:9c |

[ 7.747396] cloud-init[916]: ci-info: | lo: | True | 127.0.0.1 | 255.0.0.0 | . | . |

[ 7.753394] cloud-init[916]: ci-info: | lo: | True | . | . | d | . |

[ 7.759543] cloud-init[916]: ci-info: +--------+------+-------------+-----------------+-------+-------------------+

[ 7.773573] cloud-init[916]: ci-info: +++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++

[ 7.786722] cloud-init[916]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+

[ 7.792142] cloud-init[916]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |

[ 7.796739] cloud-init[916]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+

[ 7.803558] cloud-init[916]: ci-info: | 0 | 0.0.0.0 | 10.30.85.1 | 0.0.0.0 | eth0 | UG |

[ 7.812274] cloud-init[916]: ci-info: | 1 | 10.30.85.0 | 0.0.0.0 | 255.255.255.192 | eth0 | U |

[ 7.818805] cloud-init[916]: ci-info: | 2 | 169.254.0.0 | 0.0.0.0 | 255.255.0.0 | eth0 | U |

[ 7.826905] cloud-init[916]: ci-info: +-------+-------------+------------+-----------------+-----------+-------+

 

 

Here is the 1811 instance trying to connect via that 169 interface (it does this for 2 minutes):

 

[ 14.877285] cloud-init[2897]: 2020-03-13 02:28:41,362 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [2/120s]: request error [('Connection aborted.', error(113, 'No route to host'))]

[ 17.882664] cloud-init[2897]: 2020-03-13 02:28:44,368 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [5/120s]: request error [('Connection aborted.', error(113, 'No route to host'))]

[ 20.888679] cloud-init[2897]: 2020-03-13 02:28:47,374 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [8/120s]: request error [('Connection aborted.', error(113, 'No route to host'))]

[ 23.894483] cloud-init[2897]: 2020-03-13 02:28:50,380 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [11/120s]: request error [('Connection aborted.', error(113, 'No route to host'))]

[ 26.900562] cloud-init[2897]: 2020-03-13 02:28:53,386 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [14/120s]: request error [('Connection aborted.', error(113, 'No route to host'))]

[ 29.906682] cloud-init[2897]: 2020-03-13 02:28:56,392 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/2009-04-04/meta-data/instance-id' failed [18/120s]: request error [('Connection aborted.', error(113, 'No route to host'))]

 

Eventually the attempt to pull the metadata fails, cloud-init then stops before it has a chance to inject our ssh keys (where we both saw permission denied).
TagsNo tags attached.

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2020-04-11 02:50 cchekuri New Issue