I do not believe the ganeti/3.0.2-3 (arm64, ppc64el) failures on Lunar are caused by the openssh update. In both failures[1][2], the __main__.TestX509Certificates.test and __main__.TestX509Certificates.testNoEmpty unit tests fail. This exact same failure can be seen in an earlier migration-reference/0 tests[3][4] of ganeti on both arm64 and ppc64el, so it seems this test is just flaky.
[1] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/arm64/g/ganeti/20230605_011521_99bea@/log.gz [2] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/ppc64el/g/ganeti/20230605_011157_99bea@/log.gz [3] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/arm64/g/ganeti/20230118_074257_458fa@/log.gz [4] https://autopkgtest.ubuntu.com/results/autopkgtest-lunar/lunar/ppc64el/g/ganeti/20230118_072256_077d0@/log.gz -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2011458 Title: ssh fails to rebind when it is killed with -HUP Status in openssh package in Ubuntu: Fix Released Status in openssh source package in Kinetic: Fix Committed Status in openssh source package in Lunar: Fix Committed Bug description: [Impact] The sshd re-execution logic is generally broken with systemd socket activation, which means that (1) sshd fails when it is told to re-exec via SIGHUP (e.g. systemctl reload ssh), and (2) sshd fails when started in debug mode. [Test Case] (1) Test systemctl reload ssh: * On a machine with openssh-server installed, make a connection to localhost to activate ssh.service (the connection does not need to be complete, so you can just say "no" at the host key verification stage): $ ssh localhost [...] * Send SIGHUP to sshd by calling systemctl reload ssh: $ systemctl reload ssh * Check the service state: $ systemctl status ssh × ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled) Drop-In: /etc/systemd/system/ssh.service.d └─00-socket.conf Active: failed (Result: exit-code) since Mon 2023-04-17 20:43:27 UTC; 4s ago Duration: 2min 44.132s TriggeredBy: ● ssh.socket Docs: man:sshd(8) man:sshd_config(5) Process: 1112 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION) Process: 1152 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) Process: 1153 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS) Main PID: 1112 (code=exited, status=255/EXCEPTION) CPU: 79ms Apr 17 20:40:43 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server. Apr 17 20:41:06 lunar sshd[1113]: Connection closed by 127.0.0.1 port 54666 [preauth] Apr 17 20:43:27 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure Shell server... Apr 17 20:43:27 lunar sshd[1112]: Received SIGHUP; restarting. Apr 17 20:43:27 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell server. Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use. Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on :: failed: Address already in use. Apr 17 20:43:27 lunar sshd[1112]: fatal: Cannot bind any address. Apr 17 20:43:27 lunar systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION Apr 17 20:43:27 lunar systemd[1]: ssh.service: Failed with result 'exit-code'. * On an affected machine, the service will fail as shown above. (2) Test debug mode: * On a machine with openssh-server installed, edit /etc/default/ssh to configure debug mode for sshd: $ cat /etc/default/ssh # Default settings for openssh-server. This file is sourced by /bin/sh from # /etc/init.d/ssh. # Options to pass to sshd SSHD_OPTS=-ddd * Attempt to make a connection to localhost: $ ssh localhost kex_exchange_identification: read: Connection reset by peer Connection reset by 127.0.0.1 port 22 * On an affected machine, the attempt will fail as shown above, and the service will be in a failed state: $ systemctl status ssh × ssh.service - OpenBSD Secure Shell server Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled) Drop-In: /etc/systemd/system/ssh.service.d └─00-socket.conf Active: failed (Result: exit-code) since Mon 2023-04-17 20:46:34 UTC; 2min 27s ago Duration: 5ms TriggeredBy: ● ssh.socket Docs: man:sshd(8) man:sshd_config(5) Process: 1166 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS) Process: 1167 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited, status=255/EXCEPTION) Main PID: 1167 (code=exited, status=255/EXCEPTION) CPU: 40ms Apr 17 20:46:34 lunar sshd[1167]: Server listening on :: port 22. Apr 17 20:46:34 lunar sshd[1167]: debug3: fd 4 is not O_NONBLOCK Apr 17 20:46:34 lunar sshd[1167]: debug1: Server will not fork when running in debugging mode. Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: entering fd = 7 config len 3456 Apr 17 20:46:34 lunar sshd[1167]: debug3: ssh_msg_send: type 0 Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: done Apr 17 20:46:34 lunar sshd[1167]: debug1: rexec start in 4 out 4 newsock 4 pipe -1 sock 7 Apr 17 20:46:34 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell server. Apr 17 20:46:34 lunar systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION Apr 17 20:46:34 lunar systemd[1]: ssh.service: Failed with result 'exit-code'. [Where problems could occur] The fix expands Ubuntu's patch for systemd socket activation to try and make sure that any fds passed from systemd are not closed across re-executions of sshd. If we saw a problem, it would most likely be an attempt to operate on a closed fd, or the wrong fd, as a result of an edge case in one of the re-execution paths. [Original Description] In kinetic and lunar gce images we are facing an issue when ssh is being killed with -HUP SSH is failing to rebind port 22. It is not failing in other previous systems. It can be reproduced by running # pkill -o -HUP sshd || true # journalctl -n 20 Mar 13 14:58:52 mar131454-025105 sshd[1371]: Received SIGHUP; restarting. Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use. Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on :: failed: Address already in use. Mar 13 14:58:52 mar131454-025105 sshd[1371]: fatal: Cannot bind any address. Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Failed with result 'exit-code'. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2011458/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
