I have verified the fix using openssh 1:9.0p1-1ubuntu8.2 from lunar-
proposed:
Test 1:
root@lunar:~# apt-cache policy openssh-server
openssh-server:
Installed: 1:9.0p1-1ubuntu8.2
Candidate: 1:9.0p1-1ubuntu8.2
Version table:
*** 1:9.0p1-1ubuntu8.2 500
500 http://archive.ubuntu.com/ubuntu lunar-proposed/main amd64 Packages
100 /var/lib/dpkg/status
1:9.0p1-1ubuntu8 500
500 http://archive.ubuntu.com/ubuntu lunar/main amd64 Packages
root@lunar:~# ssh localhost
The authenticity of host 'localhost (::1)' can't be established.
ED25519 key fingerprint is SHA256:pGn2A4yGeFN/7Rlk830HLUMQqsybaSmP36MPGpmKXmQ.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Welcome to Ubuntu 23.04 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
root@lunar:~# systemctl reload ssh
root@lunar:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Fri 2023-06-02 14:13:03 UTC; 11s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1030 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1062 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1063 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
status=0/SUCCESS)
Main PID: 1031 (sshd)
Tasks: 1 (limit: 18854)
Memory: 3.2M
CPU: 101ms
CGroup: /system.slice/ssh.service
└─1031 "sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups"
Jun 02 14:13:03 lunar systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Jun 02 14:13:03 lunar sshd[1031]: Server listening on :: port 22.
Jun 02 14:13:03 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Jun 02 14:13:04 lunar sshd[1032]: Accepted publickey for root from ::1 port
53474 ssh2: RSA SHA256:KzrBiY2sWqLQ5WWGGUo>
Jun 02 14:13:04 lunar sshd[1032]: pam_unix(sshd:session): session opened for
user root(uid=0) by (uid=0)
Jun 02 14:13:04 lunar sshd[1032]: pam_env(sshd:session): deprecated reading of
user environment enabled
Jun 02 14:13:10 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure Shell
server...
Jun 02 14:13:10 lunar sshd[1031]: Received SIGHUP; restarting.
Jun 02 14:13:10 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell
server.
Jun 02 14:13:10 lunar sshd[1031]: Server listening on :: port 22.
Test 2:
root@lunar:~# vi /etc/default/ssh
root@lunar:~# cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
root@lunar:~# ssh localhost
Welcome to Ubuntu 23.04 (GNU/Linux 6.2.0-20-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
Last login: Fri Jun 2 14:13:04 2023 from ::1
debug1: PAM: reinitializing credentials
debug1: permanently_set_uid: 0/0
debug3: Copy environment: XDG_SESSION_ID=111
debug3: Copy environment: XDG_RUNTIME_DIR=/run/user/0
debug3: Copy environment: DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
debug3: Copy environment: XDG_SESSION_TYPE=tty
debug3: Copy environment: XDG_SESSION_CLASS=user
debug3: Copy environment:
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
debug3: Copy environment: LANG=en_US.UTF-8
Environment:
LANG=en_US.UTF-8
USER=root
LOGNAME=root
HOME=/root
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
SHELL=/bin/bash
TERM=xterm-256color
XDG_SESSION_ID=111
XDG_RUNTIME_DIR=/run/user/0
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/0/bus
XDG_SESSION_TYPE=tty
XDG_SESSION_CLASS=user
SSH_CLIENT=::1 58114 22
SSH_CONNECTION=::1 58114 ::1 22
SSH_TTY=/dev/pts/2
root@lunar:~# systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset: enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
/run/systemd/system/service.d
└─zzz-lxc-service.conf
Active: active (running) since Fri 2023-06-02 14:14:30 UTC; 3s ago
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 146 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Main PID: 147 (sshd)
Tasks: 0 (limit: 18854)
Memory: 1.8M
CPU: 58ms
CGroup: /system.slice/ssh.service
‣ 147 "sshd: root@pts/2"
Jun 02 14:14:31 lunar sshd[147]: debug1: server_input_channel_req: channel 0
request shell reply 1
Jun 02 14:14:31 lunar sshd[147]: debug1: session_by_channel: session 0 channel 0
Jun 02 14:14:31 lunar sshd[147]: debug1: session_input_channel_req: session 0
req shell
Jun 02 14:14:31 lunar sshd[147]: Starting session: shell on pts/2 for root from
::1 port 58114 id 0
Jun 02 14:14:31 lunar sshd[147]: debug2: fd 4 setting TCP_NODELAY
Jun 02 14:14:31 lunar sshd[147]: debug3: set_sock_tos: set socket 4 IPV6_TCLASS
0x10
Jun 02 14:14:31 lunar sshd[147]: debug2: channel 0: rfd 10 isatty
Jun 02 14:14:31 lunar sshd[147]: debug2: fd 10 setting O_NONBLOCK
Jun 02 14:14:31 lunar sshd[147]: debug3: fd 7 is O_NONBLOCK
Jun 02 14:14:31 lunar sshd[147]: debug3: send packet: type 99
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to openssh in Ubuntu.
https://bugs.launchpad.net/bugs/2011458
Title:
ssh fails to rebind when it is killed with -HUP
Status in openssh package in Ubuntu:
Fix Released
Status in openssh source package in Kinetic:
Fix Committed
Status in openssh source package in Lunar:
Fix Committed
Bug description:
[Impact]
The sshd re-execution logic is generally broken with systemd socket
activation, which means that (1) sshd fails when it is told to re-exec
via SIGHUP (e.g. systemctl reload ssh), and (2) sshd fails when started in
debug mode.
[Test Case]
(1) Test systemctl reload ssh:
* On a machine with openssh-server installed, make a connection to
localhost to activate ssh.service (the connection does not need to be
complete, so you can just say "no" at the host key verification
stage):
$ ssh localhost
[...]
* Send SIGHUP to sshd by calling systemctl reload ssh:
$ systemctl reload ssh
* Check the service state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset:
enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:43:27 UTC; 4s
ago
Duration: 2min 44.132s
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1112 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited,
status=255/EXCEPTION)
Process: 1152 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
Process: 1153 ExecReload=/bin/kill -HUP $MAINPID (code=exited,
status=0/SUCCESS)
Main PID: 1112 (code=exited, status=255/EXCEPTION)
CPU: 79ms
Apr 17 20:40:43 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:41:06 lunar sshd[1113]: Connection closed by 127.0.0.1 port 54666
[preauth]
Apr 17 20:43:27 lunar systemd[1]: Reloading ssh.service - OpenBSD Secure
Shell server...
Apr 17 20:43:27 lunar sshd[1112]: Received SIGHUP; restarting.
Apr 17 20:43:27 lunar systemd[1]: Reloaded ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on 0.0.0.0 failed:
Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: error: Bind to port 22 on :: failed:
Address already in use.
Apr 17 20:43:27 lunar sshd[1112]: fatal: Cannot bind any address.
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Main process exited,
code=exited, status=255/EXCEPTION
Apr 17 20:43:27 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
* On an affected machine, the service will fail as shown above.
(2) Test debug mode:
* On a machine with openssh-server installed, edit /etc/default/ssh to
configure debug mode for sshd:
$ cat /etc/default/ssh
# Default settings for openssh-server. This file is sourced by /bin/sh from
# /etc/init.d/ssh.
# Options to pass to sshd
SSHD_OPTS=-ddd
* Attempt to make a connection to localhost:
$ ssh localhost
kex_exchange_identification: read: Connection reset by peer
Connection reset by 127.0.0.1 port 22
* On an affected machine, the attempt will fail as shown above, and
the service will be in a failed state:
$ systemctl status ssh
× ssh.service - OpenBSD Secure Shell server
Loaded: loaded (/lib/systemd/system/ssh.service; disabled; preset:
enabled)
Drop-In: /etc/systemd/system/ssh.service.d
└─00-socket.conf
Active: failed (Result: exit-code) since Mon 2023-04-17 20:46:34 UTC;
2min 27s ago
Duration: 5ms
TriggeredBy: ● ssh.socket
Docs: man:sshd(8)
man:sshd_config(5)
Process: 1166 ExecStartPre=/usr/sbin/sshd -t (code=exited,
status=0/SUCCESS)
Process: 1167 ExecStart=/usr/sbin/sshd -D $SSHD_OPTS (code=exited,
status=255/EXCEPTION)
Main PID: 1167 (code=exited, status=255/EXCEPTION)
CPU: 40ms
Apr 17 20:46:34 lunar sshd[1167]: Server listening on :: port 22.
Apr 17 20:46:34 lunar sshd[1167]: debug3: fd 4 is not O_NONBLOCK
Apr 17 20:46:34 lunar sshd[1167]: debug1: Server will not fork when running
in debugging mode.
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: entering fd = 7
config len 3456
Apr 17 20:46:34 lunar sshd[1167]: debug3: ssh_msg_send: type 0
Apr 17 20:46:34 lunar sshd[1167]: debug3: send_rexec_state: done
Apr 17 20:46:34 lunar sshd[1167]: debug1: rexec start in 4 out 4 newsock 4
pipe -1 sock 7
Apr 17 20:46:34 lunar systemd[1]: Started ssh.service - OpenBSD Secure Shell
server.
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Main process exited,
code=exited, status=255/EXCEPTION
Apr 17 20:46:34 lunar systemd[1]: ssh.service: Failed with result 'exit-code'.
[Where problems could occur]
The fix expands Ubuntu's patch for systemd socket activation to try
and make sure that any fds passed from systemd are not closed across
re-executions of sshd. If we saw a problem, it would most likely be an
attempt to operate on a closed fd, or the wrong fd, as a result of an
edge case in one of the re-execution paths.
[Original Description]
In kinetic and lunar gce images we are facing an issue when ssh is being
killed with -HUP
SSH is failing to rebind port 22. It is not failing in other previous systems.
It can be reproduced by running
# pkill -o -HUP sshd || true
# journalctl -n 20
Mar 13 14:58:52 mar131454-025105 sshd[1371]: Received SIGHUP; restarting.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on
0.0.0.0 failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: error: Bind to port 22 on ::
failed: Address already in use.
Mar 13 14:58:52 mar131454-025105 sshd[1371]: fatal: Cannot bind any address.
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Main process
exited, code=exited, status=255/EXCEPTION
Mar 13 14:58:52 mar131454-025105 systemd[1]: ssh.service: Failed with result
'exit-code'.
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2011458/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
