View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0018046 | CentOS-8 | cloud-init | public | 2021-02-04 09:32 | 2021-02-04 09:32 |
| Reporter | mrdracon | Assigned To | |||
| Priority | normal | Severity | minor | Reproducibility | always |
| Status | new | Resolution | open | ||
| Product Version | 8.3.2011 | ||||
| Summary | 0018046: openSSH fails to start after cloud-init cleared host ssh keys | ||||
| Description | Hey folks, On current Centos-8 and Centos-8-Stream Generic cloud images there exists a bug which causes OpenSSH server to fail once at start. That happens because cloud-init now clears host ssh keys on startup, but that happens after sshd-keygen.target. So openSSH server thinks it has new fresh generated keys, but because cloud-init clears them, it fails to start. In logs it looks like: 1) openSSH generates server keys Feb 03 15:24:14 localhost.localdomain systemd[1]: sshd-keygen@ed25519.service: Succeeded. Feb 03 15:24:14 localhost.localdomain systemd[1]: Started OpenSSH ed25519 Server Key Generation. Feb 03 15:24:14 localhost.localdomain systemd[1]: sshd-keygen@ecdsa.service: Succeeded. Feb 03 15:24:14 localhost.localdomain systemd[1]: Started OpenSSH ecdsa Server Key Generation. Feb 03 15:24:14 localhost.localdomain systemd[1]: sshd-keygen@rsa.service: Succeeded. Fеb 03 15:24:14 localhost.localdomain systemd[1]: Started OpenSSH rsa Server Key Generation. Fеb 03 15:24:14 localhost.localdomain systemd[1]: Reached target sshd-keygen.target. 2) cloud-init deletes them 2021-02-03 15:24:25,042 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key 2021-02-03 15:24:25,042 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ed25519_key.pub 2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key 2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_ecdsa_key.pub 2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key 2021-02-03 15:24:25,048 - util.py[DEBUG]: Attempting to remove /etc/ssh/ssh_host_rsa_key.pub ... 2021-02-03 15:24:25,057 - handlers.py[DEBUG]: finish: init-network/config-ssh: SUCCESS: config-ssh ran successfully 3) openSSH can't load Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: Unable to load host key: /etc/ssh/ssh_host_rsa_key Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: Unable to load host key: /etc/ssh/ssh_host_ecdsa_key Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: Unable to load host key: /etc/ssh/ssh_host_ed25519_key Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1029]: sshd: no hostkeys available -- exiting. Feb 03 15:24:25 cent-8-generic.ru-central1.internal sm-notify[1032]: Version 2.3.3 starting Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started Command Scheduler. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd.service: Failed with result 'exit-code'. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Failed to start OpenSSH server daemon. Its not really a big deal, because systemd instantly tries to restart OpenSSH Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Stopped OpenSSH server daemon. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Stopped target sshd-keygen.target. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Stopping sshd-keygen.target. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH ed25519 Server Key Generation... Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH rsa Server Key Generation... Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH ecdsa Server Key Generation... Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd-keygen@ed25519.service: Succeeded. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started OpenSSH ed25519 Server Key Generation. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd-keygen@ecdsa.service: Succeeded. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started OpenSSH ecdsa Server Key Generation. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: sshd-keygen@rsa.service: Succeeded. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Started OpenSSH rsa Server Key Generation. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Reached target sshd-keygen.target. Feb 03 15:24:25 cent-8-generic.ru-central1.internal systemd[1]: Starting OpenSSH server daemon... Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1287]: Server listening on 0.0.0.0 port 22. Feb 03 15:24:25 cent-8-generic.ru-central1.internal sshd[1287]: Server listening on :: port 22. but you don't really like to see errors on boot on your fresh VM | ||||
| Steps To Reproduce | 1) take CentOS-8-GenericCloud-8.3.2011-20201204.2.x86_64.qcow2 2) import it to your favorite cloud 3) provision VM out of this image 4) check in journald logs that OpenSSH failed to start | ||||
| Additional Information | It all happens now because of https://bugzilla.redhat.com/show_bug.cgi?id=1598831 on previous images cloud-init did not touch host ssh keys. issue starts with cloud-init > than cloud-init-19.4-11.el8_3.1.noarch | ||||
| Tags | No tags attached. | ||||
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2021-02-04 09:32 | mrdracon | New Issue |
