Bug#1064347: openssh-server: sshd crashes under heavy traffic
Hello,
I am no maintainer, just tried to reproduce this issue which I could
inside a minimal Bullseye amd64 qemu VM with the instructions
from the linked Ubuntu bug.
I could not reproduce it within Bookworm or Trixie/testing.
Without "LogLevel DEBUG" it was also not observable.
Unfortunately did also not happen with a ssh package built with asan enabled.
And I upgraded step by step via snapshot.d.o, around 2021-11-15 it
stopped to be an issue. This step brought openssh 8.7p1-1.
Downgrading just openssh 8.4p1-6 in this exact VM showed it again.
Therefore I assume this issue got fixed between openssh 8.4p1-6 and 8.7p1-1.
Kind regards,
Bernhard
#13
#14 malloc_consolidate (av=av@entry=0x7faa3b64cb80 ) at
malloc.c:4518
#15 0x7faa3b5023d5 in _int_malloc (av=av@entry=0x7faa3b64cb80 ,
bytes=bytes@entry=8193) at malloc.c:3699
#16 0x7faa3b503063 in malloc_check (sz=8192, caller=) at
hooks.c:239
#17 0x7faa3b504cea in __libc_calloc (n=n@entry=1,
elem_size=elem_size@entry=8192) at malloc.c:3387
#18 0x7faa3b4f6ef4 in __GI___open_memstream
(bufloc=bufloc@entry=0x7ffe636eb6e0, sizeloc=sizeloc@entry=0x7ffe636eb6e8) at
memstream.c:83
#19 0x7faa3b5726e1 in __vsyslog_internal (pri=39, fmt=0x55b451dcb150
"%.500s", ap=0x7ffe636eb7d0, mode_flags=2) at ../misc/syslog.c:181
#20 0x7faa3b572d5f in __syslog_chk (pri=pri@entry=7, flag=flag@entry=1,
fmt=fmt@entry=0x55b451dcb150 "%.500s") at ../misc/syslog.c:136
#21 0x55b451d87e16 in syslog (__fmt=0x55b451dcb150 "%.500s", __pri=7) at
/usr/include/x86_64-linux-gnu/bits/syslog.h:31
#22 do_log (level=level@entry=SYSLOG_LEVEL_DEBUG1, fmt=fmt@entry=0x55b451dba421
"Forked child %ld.", args=args@entry=0x7ffe636ec110) at ../../log.c:484
#23 0x55b451d88254 in debug (fmt=fmt@entry=0x55b451dba421 "Forked child
%ld.") at ../../log.c:229
#24 0x55b451d3c86e in server_accept_loop (config_s=0x7ffe636ec270, newsock=, sock_out=, sock_in=) at
../../sshd.c:1377
#25 main (ac=, av=) at ../../sshd.c:2089
# 2024-04-23 Bullseye/stable amd64 qemu VM
apt update
apt dist-upgrade
apt install systemd-coredump moreutils parallel htop fakeroot mc ccache gdb
openssh-server-dbgsym
apt build-dep glibc
apt build-dep openssh-server
mkdir /home/benutzer/source/glibc/orig -p
cd/home/benutzer/source/glibc/orig
apt source glibc
mkdir /home/benutzer/source/openssh-server/orig -p
cd/home/benutzer/source/openssh-server/orig
apt source openssh-server
sed -i.bak 's/#LogLevel INFO/LogLevel DEBUG/g' /etc/ssh/sshd_config
systemctl restart sshd
ssh-keygen -b 4096
ssh-copy-id -i .ssh/id_rsa.pub benutzer@localhost
parallel -j 32 -N0 "ssh benutzer@localhost 'true'" ::: {1..2}
benutzer@debian:~/.ssh$ ssh-keygen -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/benutzer/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/benutzer/.ssh/id_rsa
Your public key has been saved in /home/benutzer/.ssh/id_rsa.pub
The key fingerprint is:
SHA256:Hgx6dUtFBhKiI0wBYKtXMkwZeRcP/eEZCUsU69bbO+o benutzer@debian
The key's randomart image is:
+---[RSA 4096]+
|+o== ++B+.++|
|.=+ ...=.++o |
| .*.+.. =oo+ |
|. = o = ++. |
|. . . . S o |
| . . o . o |
|. . .|
|.. |
| .E... |
+[SHA256]-+
benutzer@debian:~$ ssh-copy-id -i .ssh/id_rsa.pub benutzer@localhost
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: ".ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter
out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are
prompted now it is to install the new keys
benutzer@localhost's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'benutzer@localhost'"
and check to make sure that only the key(s) you wanted were added.
parallel -j 800 -N0 "ssh benutzer@localhost 'mount; sleep 1; cat /proc/cpuinfo;
free -h; dd if=/dev/zero of=/dev/null bs=1 count=8192; mount -av; sleep
$(($RANDOM % 5)); lscpu'" ::: {1..1}
# AMD Ryzen 1700, VM, 16 threads
root@debian:~# coredumpctl list
TIMEPID UID GID SIG COREFILE EXE
Tue 2024-04-23 00:20:53 CEST 124297 0 0 6 present /usr/sbin/sshd
Tue 2024-04-23 00:23:02 CEST 159284 0 0 6 present /usr/sbin/sshd
Tue 2024-04-23 00:23:47 CEST 229261 0 0 11 present /usr/sbin/sshd
Tue 2024-04-23 00:24:32 CEST 277265 0 0 11 present /usr/sbin/sshd
Tue 2024-04-23 00:24:54 CEST 301567 0 0 6 present /usr/sbin/sshd
root@debian:~# coredumpctl gdb 301567
PID: 301567 (sshd)
UID: 0 (root)
GID: 0 (root)
Signal: 6 (ABRT)
Timestamp: Tue 2024-04-23 00:24:53 CEST (47s ago)
Command Line: sshd: /usr/sbin/sshd -D [listener] 4 of 10-100 startups
Executable: /usr/sbin/sshd
Bug#1069706: systemd unit files lack ordering wrt nss-user-lookup.target
On Tue, 23 Apr 2024 12:23:21 +0100 Colin Watson wrote: > On Tue, Apr 23, 2024 at 09:32:00AM +0200, Rasmus Villemoes wrote: > > According to systemd.special(7) > > > > nss-user-lookup.target > > > > A target that should be used as synchronization point for all > > regular UNIX user/group name service lookups. [...] All > > services for which the availability of the full user/group > > database is essential should be ordered after this target, but > > not pull it in. All services which provide parts of the > > user/group database should be ordered before this target, and > > pull it in. > > > > I have a custom .service that does exactly as described in the second > > part, i.e. provides part of the user/group database and says > > Before=nss-user-lookup.target, Wants=nss-user-lookup.target > > (concretely, it modifies /etc/shadow to update a default password, but > > that's not really important). I believe sshd definitely belongs in the > > former category, i.e. sshd should not be started until any such > > service that updates the user/group database, such as updating > > /etc/shadow, have run. > > > > Hence the ssh.service and ssh.socket files should add > > > > After=nss-user-lookup.target > > > > in their [Unit] sections. This is a no-op on systems that do not have > > any service pulling in that target, but required for correctness on > > systems that do. > > > > Of course, I could, and currently do, handle this via a drop-in config > > fragment in some ssh.service.d/ directory. But this, and other similar > > synchronization targets, exist so that one does not necessarily need > > to know about every other service running on the system. > > This sounds like a reasonable proposal to me. I'm just CCing Debian's > systemd maintainers for a quick review to make sure I'm not missing > anything subtle. That sounds fine - maybe the service, but not the socket, so that connections can start to come in early. I also note there's accountsservice pulling that target in but it shouldn't, but that's a separate matter and can be handled upstream. -- Kind regards, Luca Boccassi signature.asc Description: This is a digitally signed message part
Bug#1069706: systemd unit files lack ordering wrt nss-user-lookup.target
On Tue, Apr 23, 2024 at 09:32:00AM +0200, Rasmus Villemoes wrote: > According to systemd.special(7) > > nss-user-lookup.target > > A target that should be used as synchronization point for all > regular UNIX user/group name service lookups. [...] All > services for which the availability of the full user/group > database is essential should be ordered after this target, but > not pull it in. All services which provide parts of the > user/group database should be ordered before this target, and > pull it in. > > I have a custom .service that does exactly as described in the second > part, i.e. provides part of the user/group database and says > Before=nss-user-lookup.target, Wants=nss-user-lookup.target > (concretely, it modifies /etc/shadow to update a default password, but > that's not really important). I believe sshd definitely belongs in the > former category, i.e. sshd should not be started until any such > service that updates the user/group database, such as updating > /etc/shadow, have run. > > Hence the ssh.service and ssh.socket files should add > > After=nss-user-lookup.target > > in their [Unit] sections. This is a no-op on systems that do not have > any service pulling in that target, but required for correctness on > systems that do. > > Of course, I could, and currently do, handle this via a drop-in config > fragment in some ssh.service.d/ directory. But this, and other similar > synchronization targets, exist so that one does not necessarily need > to know about every other service running on the system. This sounds like a reasonable proposal to me. I'm just CCing Debian's systemd maintainers for a quick review to make sure I'm not missing anything subtle. -- Colin Watson (he/him) [cjwat...@debian.org]
Bug#1069236: openssh-server: X over ssh fails with "cannot open display"
I'm not using a hostname with ssh, I'm sshing directly to an IPv4 address. *How* was it disabled? net.ipv6.conf.all.disable_ipv6 = 1 in /etc/sysctl.conf My point is that "AddressFamily any" should not fail to set $DISPLAY if IPv6 is not available. On Tue, Apr 23, 2024 at 5:38 AM Jonathan Dowland wrote: > > On Thu, Apr 18, 2024 at 06:33:00AM -0500, allan wrote: > > Resolved the issue by editing /etc/ssh/sshd_config and changing > > #AddressFamily any > > to > > AddressFamily inet > > This is not a reasonable change to make to the default configuration, > because it would mean that ssh did not work out of the box in IPv6 > environments. > > On Thu, Apr 18, 2024 at 07:53:52AM -0500, allan wrote: > > More info - IPv6 is disabled on all four machines. I think > > "AddressFamily any" should have supported an IPv4 connection. > > *How* is it disabled? More information will be needed to figure out > exactly what's gone on in your environment. > > I speculate that the hostnames you were trying to connect to were > resolving as IPv6 addresses, and the connection failing because the > hosts are rejecting IPv6 traffic. If that's right, the ultimate fix > is to correct whatever name resolution is giving you the wrong > addresses in your environment. > > If you are prepared to experiment, we might be able to drill down and > check that. If so, can you > > 1) reverse the sshd_config change you made on at least one of the >hosts, and restart that sshd > > 2) assuming the troublesome host is named "myhost" in your environment >(substitute as appropriate), from your client machine, report the >result of running > > getent hosts myhost > dig +short myhost > nslookup myhost > ping -c 1 myhost > > (one or more of these commands may not exist on your machine) > > 2) re-attempt to connect from your client, this time passing -vv or >-vvv, and capture the logging output
Bug#1069236: openssh-server: X over ssh fails with "cannot open display"
On Thu, Apr 18, 2024 at 06:33:00AM -0500, allan wrote: > Resolved the issue by editing /etc/ssh/sshd_config and changing > #AddressFamily any > to > AddressFamily inet This is not a reasonable change to make to the default configuration, because it would mean that ssh did not work out of the box in IPv6 environments. On Thu, Apr 18, 2024 at 07:53:52AM -0500, allan wrote: > More info - IPv6 is disabled on all four machines. I think > "AddressFamily any" should have supported an IPv4 connection. *How* is it disabled? More information will be needed to figure out exactly what's gone on in your environment. I speculate that the hostnames you were trying to connect to were resolving as IPv6 addresses, and the connection failing because the hosts are rejecting IPv6 traffic. If that's right, the ultimate fix is to correct whatever name resolution is giving you the wrong addresses in your environment. If you are prepared to experiment, we might be able to drill down and check that. If so, can you 1) reverse the sshd_config change you made on at least one of the hosts, and restart that sshd 2) assuming the troublesome host is named "myhost" in your environment (substitute as appropriate), from your client machine, report the result of running getent hosts myhost dig +short myhost nslookup myhost ping -c 1 myhost (one or more of these commands may not exist on your machine) 2) re-attempt to connect from your client, this time passing -vv or -vvv, and capture the logging output
Bug#1069706: systemd unit files lack ordering wrt nss-user-lookup.target
Package: openssh-server Version: 1:8.9p1-3ubuntu0.6 Severity: normal Dear Maintainer, According to systemd.special(7) nss-user-lookup.target A target that should be used as synchronization point for all regular UNIX user/group name service lookups. [...] All services for which the availability of the full user/group database is essential should be ordered after this target, but not pull it in. All services which provide parts of the user/group database should be ordered before this target, and pull it in. I have a custom .service that does exactly as described in the second part, i.e. provides part of the user/group database and says Before=nss-user-lookup.target, Wants=nss-user-lookup.target (concretely, it modifies /etc/shadow to update a default password, but that's not really important). I believe sshd definitely belongs in the former category, i.e. sshd should not be started until any such service that updates the user/group database, such as updating /etc/shadow, have run. Hence the ssh.service and ssh.socket files should add After=nss-user-lookup.target in their [Unit] sections. This is a no-op on systems that do not have any service pulling in that target, but required for correctness on systems that do. Of course, I could, and currently do, handle this via a drop-in config fragment in some ssh.service.d/ directory. But this, and other similar synchronization targets, exist so that one does not necessarily need to know about every other service running on the system. -- System Information: Debian Release: bookworm/sid APT prefers jammy-updates APT policy: (500, 'jammy-updates'), (500, 'jammy-security'), (500, 'jammy'), (100, 'jammy-backports') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 5.15.136-6-g3d6db53ae88c (SMP w/8 CPU threads) Kernel taint flags: TAINT_WARN Locale: LANG=en_DK.UTF-8, LC_CTYPE=en_DK.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en Shell: /bin/sh linked to /usr/bin/dash Init: systemd (via /run/systemd/system) Versions of packages openssh-server depends on: ii adduser3.118ubuntu5 ii debconf [debconf-2.0] 1.5.79ubuntu1 ii dpkg 1.21.1ubuntu2.3 ii init-system-helpers1.62 ii libaudit1 1:3.0.7-1build1 ii libc6 2.35-0ubuntu3.6 ii libcom-err21.46.5-2ubuntu1.1 ii libcrypt1 1:4.4.27-1 ii libgssapi-krb5-2 1.19.2-2ubuntu0.3 ii libkrb5-3 1.19.2-2ubuntu0.3 ii libpam-modules 1.4.0-11ubuntu2.4 ii libpam-runtime 1.4.0-11ubuntu2.4 ii libpam0g 1.4.0-11ubuntu2.4 ii libselinux13.3-1build2 ii libssl33.0.2-0ubuntu1.15 ii libsystemd0249.11-0ubuntu3.12 ii libwrap0 7.6.q-31build2 ii lsb-base 11.1.0ubuntu4 ii openssh-client 1:8.9p1-3ubuntu0.6 ii openssh-sftp-server1:8.9p1-3ubuntu0.6 ii procps 2:3.3.17-6ubuntu2.1 ii ucf3.0043 ii zlib1g 1:1.2.11.dfsg-2ubuntu9.2 Versions of packages openssh-server recommends: ii libpam-systemd [logind] 249.11-0ubuntu3.12 ii ncurses-term 6.3-2ubuntu0.1 ii ssh-import-id5.11-0ubuntu1 ii xauth1:1.1-1build2 Versions of packages openssh-server suggests: pn molly-guard pn monkeysphere ii ssh-askpass 1:1.2.4.1-13 ii ssh-askpass-fullscreen [ssh-askpass] 0.3-3.1build2 ii ssh-askpass-gnome [ssh-askpass] 1:8.9p1-3ubuntu0.6 ii ufw 0.36.1-4ubuntu0.1 -- debconf information excluded
