View Issue Details

IDProjectCategoryView StatusLast Update
0018046CentOS-8cloud-initpublic2021-02-04 09:32
Reportermrdracon Assigned To 
PrioritynormalSeverityminorReproducibilityalways
Status newResolutionopen 
Product Version8.3.2011 
Summary0018046: openSSH fails to start after cloud-init cleared host ssh keys
DescriptionHey folks,

On current Centos-8 and Centos-8-Stream Generic cloud images there exists a bug which causes OpenSSH server to fail once at start.
That happens because cloud-init now clears host ssh keys on startup, but that happens after sshd-keygen.target.

So openSSH server thinks it has new fresh generated keys, but because cloud-init clears them, it fails to start.

In logs it looks like:
1) openSSH generates server keys
Feb 03 15:24:14 localhost.localdomain systemd[1]: sshd-keygen@ed25519.service: Succeeded.
Feb 03 15:24:14 localhost.localdomain systemd[1]: Started OpenSSH ed25519 Server Key Generation.
Feb 03 15:24:14 localhost.localdomain systemd[1]: sshd-keygen@ecdsa.service: Succeeded.
Feb 03 15:24:14 localhost.localdomain systemd[1]: Started OpenSSH ecdsa Server Key Generation.
Feb 03 15:24:14 localhost.localdomain systemd[1]: sshd-keygen@rsa.service: Succeeded.
Fеb 03 15:24:14 localhost.localdomain systemd[1]: Started OpenSSH rsa Server Key Generation.
Fеb 03 15:24:14 localhost.localdomain systemd[1]: Reached target sshd-keygen.target.

2) cloud-init deletes them
2021-02-03 15:24:25,042 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key
2021-02-03 15:24:25,042 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key.pub
2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key
2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key.pub
2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key
2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key.pub
...
2021-02-03 15:24:25,057 - handlers.py[DEBUG]: finish: init-network/config-ssh: SUCCESS: config-ssh ran successfully

3) openSSH can't load
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: Unable to load host key: /etc/ssh/ssh_host_rsa_key
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: Unable to load host key: /etc/ssh/ssh_host_ecdsa_key
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: Unable to load host key: /etc/ssh/ssh_host_ed25519_key
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: sshd: no hostkeys available -- exiting.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sm-notify[1032]: Version 2.3.3 starting
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started Command Scheduler.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd.service: Failed with result 'exit-code'.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Failed to start OpenSSH server daemon.

Its not really a big deal, because systemd instantly tries to restart OpenSSH

Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Stopped OpenSSH server daemon.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Stopped target sshd-keygen.target.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Stopping sshd-keygen.target.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH ed25519 Server Key Generation...
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH rsa Server Key Generation...
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH ecdsa Server Key Generation...
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd-keygen@ed25519.service: Succeeded.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started OpenSSH ed25519 Server Key Generation.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd-keygen@ecdsa.service: Succeeded.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started OpenSSH ecdsa Server Key Generation.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd-keygen@rsa.service: Succeeded.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started OpenSSH rsa Server Key Generation.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Reached target sshd-keygen.target.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH server daemon...
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1287]: Server listening on 0.0.0.0 port 22.
Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1287]: Server listening on :: port 22.

but you don't really like to see errors on boot on your fresh VM
Steps To Reproduce1) take CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2
2) import it to your favorite cloud
3) provision VM out of this image
4) check in journald logs that OpenSSH failed to start
Additional InformationIt all happens now because of https://bugzilla.redhat.com/show_bug.cgi?id=1598831
on previous images cloud-init did not touch host ssh keys.

issue starts with cloud-init > than cloud-init-19.4-11.el8_3.1.noarch
TagsNo tags attached.

Activities

There are no notes attached to this issue.

Issue History

Date Modified Username Field Change
2021-02-04 09:32 mrdracon New Issue