I have confirmed the fix using openssh 1:9.0p1-1ubuntu7.3 from kinetic-
proposed:
Test 1:
root@kinetic:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu7.3
Candidate: 1:9.0p1-1ubuntu7.3
Version table:
*** 1:9.0p1-1ubuntu7.3 500
500 http://archive.ubuntu.com/ubuntu kinetic-proposed/main amd64
Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu7.1 500
500 http://archive.ubuntu.com/ubuntu kinetic-updates/main amd64 Packages
1:9.0p1-1ubuntu7 500
500 http://archive.ubuntu.com/ubuntu kinetic/main amd64 Packages
root@kinetic:~# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:xmU0f5QuqvJ8GZVhZihANPUgBhUzvYo96entI7JuoRk.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Welcome to Ubuntu 22.10 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
root@kinetic:~# systemctl reload ssh.service
root@kinetic:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Fri 2023-06-02 13:48:58 UTC; 42s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1012 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1046 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1047 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
status=0/SUCCESS)
Main PID: 1013 (sshd)
Tasks: 1 (limit: 18854)
Memory: 3.2M
CPU: 94ms
CGroup: /system.slice/ssh.service
└─1013 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
Jun 02 13:48:58 kinetic systemd[1]: Starting OpenBSD Secure Shell server...
Jun 02 13:48:58 kinetic sshd[1013]: Server listening on :: port 22.
Jun 02 13:48:58 kinetic systemd[1]: Started OpenBSD Secure Shell server.
Jun 02 13:49:00 kinetic sshd[1014]: Accepted publickey for root from ::1 port
44690 ssh2: RSA SHA256:A+T8Yt6KxQdy0hmT8>
Jun 02 13:49:00 kinetic sshd[1014]: pam_unix(sshd:session): session opened for
user root(uid=0) by (uid=0)
Jun 02 13:49:00 kinetic sshd[1014]: pam_env(sshd:session): deprecated reading
of user environment enabled
Jun 02 13:49:26 kinetic systemd[1]: Reloading OpenBSD Secure Shell server...
Jun 02 13:49:26 kinetic sshd[1013]: Received SIGHUP; restarting.
Jun 02 13:49:26 kinetic systemd[1]: Reloaded OpenBSD Secure Shell server.
Jun 02 13:49:27 kinetic sshd[1013]: Server listening on :: port 22.
Test 2:
root@kinetic:~# vi /etc/default/ssh
root@kinetic:~# cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
root@kinetic:~# ssh localhost
Welcome to Ubuntu 22.10 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Fri Jun 2 13:49:00 2023 from ::1
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
debug3: Copy environment: XDG_SESSION_ID=104
debug3: Copy environment: XDG_RUNTIME_DIR=/run/user/0
debug3: Copy environment: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
debug3: Copy environment: XDG_SESSION_TYPE=tty
debug3: Copy environment: XDG_SESSION_CLASS=user
debug3: Copy environment:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
debug3: Copy environment: LANG=en_US.UTF-8
Environment:
LANG=en_US.UTF-8
USER=root
LOGNAME=root
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL=/bin/bash
TERM=xterm-256color
XDG_SESSION_ID=104
XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
SSH_CLIENT=::1 47582 22
SSH_CONNECTION=::1 47582 ::1 22
SSH_TTY=/dev/pts/2
root@kinetic:~# systemctl status ssh.service
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Fri 2023-06-02 13:56:50 UTC; 6s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 142 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 143 (sshd)
Tasks: 0 (limit: 18854)
Memory: 1.8M
CPU: 42ms
CGroup: /system.slice/ssh.service
‣ 143 "sshd: root@pts/2"
Jun 02 13:56:51 kinetic sshd[143]: debug1: server_input_channel_req: channel 0
request shell reply 1
Jun 02 13:56:51 kinetic sshd[143]: debug1: session_by_channel: session 0
channel 0
Jun 02 13:56:51 kinetic sshd[143]: debug1: session_input_channel_req: session 0
req shell
Jun 02 13:56:51 kinetic sshd[143]: Starting session: shell on pts/2 for root
from ::1 port 47582 id 0
Jun 02 13:56:51 kinetic sshd[143]: debug2: fd 4 setting TCP_NODELAY
Jun 02 13:56:51 kinetic sshd[143]: debug3: set_sock_tos: set socket 4
IPV6_TCLASS 0x10
Jun 02 13:56:51 kinetic sshd[143]: debug2: channel 0: rfd 10 isatty
Jun 02 13:56:51 kinetic sshd[143]: debug2: fd 10 setting O_NONBLOCK
Jun 02 13:56:51 kinetic sshd[143]: debug3: fd 7 is O_NONBLOCK
Jun 02 13:56:51 kinetic sshd[143]: debug3: send packet: type 99
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2011458
Title:
ssh fails to rebind when it is killed with -HUP
Status in openssh package in Ubuntu:
Fix Released
Status in openssh source package in Kinetic:
Fix Committed
Status in openssh source package in Lunar:
Fix Committed
Bug description:
[Impact]
The sshd re-execution logic is generally broken with systemd socket
activation, which means that (1) sshd fails when it is told to re-exec
via SIGHUP (e.g. systemctl reload ssh), and (2) sshd fails when started in
debug mode.
[Test Case]
(1) Test systemctl reload ssh:
* On a machine with openssh-server installed, make a connection to
localhost to activate ssh.service (the connection does not need to be
complete, so you can just say "no" at the host key verification
stage):
$ ssh localhost
[...]
* Send SIGHUP to sshd by calling systemctl reload ssh:
$ systemctl reload ssh
* Check the service state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset:
enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:43:27 UTC; 4s
ago
Duration: 2min 44.132s
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1112 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited,
status=255/EXCEPTION)
Process: 1152 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1153 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
status=0/SUCCESS)
Main PID: 1112 (code=exited, status=255/EXCEPTION)
CPU: 79ms
Apr 17 20:40:43 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:41:06 lunar sshd[1113]: Connection closed by 127.0.0.1 port 54666
[preauth]
Apr 17 20:43:27 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure
Shell server...
Apr 17 20:43:27 lunar sshd[1112]: Received SIGHUP; restarting.
Apr 17 20:43:27 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on 0.0.0.0 failed:
Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on :: failed:
Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: fatal: Cannot bind any address.
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Main process exited,
code=exited, status=255/EXCEPTION
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
* On an affected machine, the service will fail as shown above.
(2) Test debug mode:
* On a machine with openssh-server installed, edit /etc/default/ssh to
configure debug mode for sshd:
$ cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
* Attempt to make a connection to localhost:
$ ssh localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
* On an affected machine, the attempt will fail as shown above, and
the service will be in a failed state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset:
enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:46:34 UTC;
2min 27s ago
Duration: 5ms
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1166 ExecStartPre=/usr/sbin/sshd -t (code=exited,
status=0/SUCCESS)
Process: 1167 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited,
status=255/EXCEPTION)
Main PID: 1167 (code=exited, status=255/EXCEPTION)
CPU: 40ms
Apr 17 20:46:34 lunar sshd[1167]: Server listening on :: port 22.
Apr 17 20:46:34 lunar sshd[1167]: debug3: fd 4 is not O_NONBLOCK
Apr 17 20:46:34 lunar sshd[1167]: debug1: Server will not fork when running
in debugging mode.
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: entering fd = 7
config len 3456
Apr 17 20:46:34 lunar sshd[1167]: debug3: ssh_msg_send: type 0
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: done
Apr 17 20:46:34 lunar sshd[1167]: debug1: rexec start in 4 out 4 newsock 4
pipe -1 sock 7
Apr 17 20:46:34 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Main process exited,
code=exited, status=255/EXCEPTION
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
[Where problems could occur]
The fix expands Ubuntu's patch for systemd socket activation to try
and make sure that any fds passed from systemd are not closed across
re-executions of sshd. If we saw a problem, it would most likely be an
attempt to operate on a closed fd, or the wrong fd, as a result of an
edge case in one of the re-execution paths.
[Original Description]
In kinetic and lunar gce images we are facing an issue when ssh is being
killed with -HUP
SSH is failing to rebind port 22. It is not failing in other previous systems.
It can be reproduced by running
# pkill -o -HUP sshd || true
# journalctl -n 20
Mar 13 14:58:52 mar131454-025105 sshd[1371]: Received SIGHUP; restarting.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on
0.0.0.0 failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on ::
failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: fatal: Cannot bind any address.
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Main process
exited, code=exited, status=255/EXCEPTION
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Failed with result
'exit-code'.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2011458/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
