Bug#923508: systemd not restarting sshd on Debian 9 Stretch i386 due to RestartPreventExitStatus=255 in sshd.service file
Michael, thanks for your detailed reply. A few things I would like to clarify. 1) I'm not wanting to edit the sshd.service file As you say this will be overwritten, hence my comment that this was NOT a solution. 2) I was trying to use the sshd.service.d/override.conf file instead However this does not seem to be working. The systemctl edit option was editing this override file successfully, but it didn't seem to take effect. I was not aware that you had to reset any option before setting it again. I have seen this in relation to ExecStart= but I guess that applies as you can set multiple ExecStart options, and thus need to clear it first. It seems to have been the case that other options have not needed to be cleared in the override file before resetting, at least in my experience. Having an option set to both True and False for instance does not make a lot of sense. In any case I was resetting RestartPreventExitStatus to empty as I did not want any value assigned to it (as if it had never been set) Perhaps this is the issue, that as RestartPreventExitStatus is single valued, you CANNOT reset it to nothing, only to another value? If so that's a problem as there is no way of (safely) clearing the setting of RestartPreventExitStatus. Is there a way of setting systemd to dump out the config it is using for a specific service, after the override file has been applied? 3) With regard to sshd I don't really have a problem with what sshd is doing with RestartPreventExitStatus, provided I can override it. Although because the return value of 255 is not specific to issue it was added to address, then it may cause problems for others. As for using a non-default config, I have no apologies about that, as I don't consider the default anything like secure. Deciding on listening on a specific interface only, and using public key authentication is prudent. 4) Networking Certainly the networking configuration was not broken. Although I have discovered that is machine unlike the other debian boxes I have (installed from earlier releases and upgraded) is using NetworkManager, which I have now removed, as it does not play well on headless machines. Without NetworkManager the machine will start sshd with the default config (RestartPreventExitStatus=255) as the interface is started earlier, and the need to restart sshd is removed. But given NetworkManager is the default these days, that would seem to make removing it 'not a default config'. This does not negate the need to be able to override RestartPreventExitStatus, or at least have the ability to do so.
Bug#923508: systemd not restarting sshd on Debian 9 Stretch i386 due to RestartPreventExitStatus=255 in sshd.service file
Michael, as to what the bug report is about I guess there are three issues: 1) At some point the /etc/systemd/system/sshd.service file has been updated to include RestartPreventExitStatus=255 which has a negative effect stopping sshd restarting in order to work with a slow to start network connection. This does not seem to be a sensible default option. I don't know if this comes under sshd or systemd. 2) The override.conf mechanism does not seem to be working. Which is a definite systemd issue 3) systemd seems to be complaining about the need to reload units using daemon-reload even when you have just done daemon-reload These seem to have come in with the last point release, and as of yesterday, the latest updates on top of that had not addressed these issues. Andrew
Bug#923508: systemd not restarting sshd on Debian 9 Stretch i386 due to RestartPreventExitStatus=255 in sshd.service file
Package: systemd Version: 232-25+deb9u On Debian 9 stretch with the latest updates systemd no longer restarts sshd due to changes to the sshd.service unit file. The line: RestartPreventExitStatus=255 has been added to the master sshd unit file in recent systemd/sshd updates. This means if the network interface is not up by the time sshd tries to start, sshd will return a result code of 255, and systemd will NOT try to restart it. This is obviously a HUGE problem for anybody using sshd as their exclusive means of accessing a system. On many systems the network can take a considerable amount of time to start. When you finally do get physical access to a system, and try to override the default unit file by adding: RestartPreventExitStatus= to the end of /etc/systemd/system/sshd.service.d/override.conf and running systemctl daemon-reload and rebooting You find that the default of 255 is NOT overridden. I already had: [Service] Restart=always RestartSec=5 in the override.conf file. What part of ALWAYS does systemd not understand! I was forced to comment out the RestartPreventExitStatus=255 line in the /etc/systemd/system/sshd.service file, which is NOT a solution. Also the version of systemd currently being used has bugs which when you manually restart a unit using systemctl restart sshd.service it warns about the configuration needing reloading using systemctl daemon-reload, even though you have just done that. This bug appears to have been fixed once in systemd in 2016, so either an old version is being used, or it has crept back in. systemctl daemon-reload systemctl restart sshd.service Warning: sshd.service changed on disk. Run 'systemctl daemon-reload' to reload units. Of course if systemd didn't start networking dependent units until the network was actually up then none of this would be an issue... Of course then you get into arguments about which network interface... Clearly much more testing of systemd needs to be done before pushing updates of it. But the bottom line is NEVER break sshd. Systemd Version: ||/ Name Version Architecture Description +++-==---= ii systemd 232-25+deb9u i386 system and service manager Openssh Server version: ||/ Name Version Architecture Description +++-==---= ii openssh-server 1:7.4p1-10+d i386 secure shell (SSH) server, for se System Version: uname -a Linux sempron 4.9.0-8-686-pae #1 SMP Debian 4.9.144-3.1 (2019-02-19) i686 GNU/Linux cat /etc/os-release PRETTY_NAME="Debian GNU/Linux 9 (stretch)" apt show libc6 | grep ^Version Version: 2.24-11+deb9u4 Error messages from sshd when booting using default config (RestartPreventExitStatus=255 in /etc/systemd/system/sshd.service file) journalctl -b0 | grep ssh Mar 01 07:32:02 sempron sshd[432]: error: Bind to port on 192.168.1.54 failed: Cannot assign requested address. Mar 01 07:32:02 sempron sshd[432]: fatal: Cannot bind any address. Mar 01 07:32:02 sempron systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a Mar 01 07:32:02 sempron systemd[1]: ssh.service: Unit entered failed state. Mar 01 07:32:02 sempron systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 01 07:32:17 sempron systemd[700]: Listening on GnuPG cryptographic agent (ssh-agent emulation). where is the sshd port number /etc/systemd/system/sshd.service file used for above log file: [Unit] Description=OpenBSD Secure Shell server After=network.target auditd.service ConditionPathExists=!/etc/ssh/sshd_not_to_be_run [Service] EnvironmentFile=-/etc/default/ssh ExecStartPre=/usr/sbin/sshd -t ExecStart=/usr/sbin/sshd -D $SSHD_OPTS ExecReload=/usr/sbin/sshd -t ExecReload=/bin/kill -HUP $MAINPID KillMode=process #Restart=on-failure Restart=always RestartPreventExitStatus=255 Type=notify [Install] WantedBy=multi-user.target Alias=sshd.service /etc/systemd/system/sshd.service.d/override.conf [Unit] After=network.target [Service] Restart=always RestartSec=5 RestartPreventExitStatus= Commenting out RestartPreventExitStatus=255 in sshd.service, doing systemctl daemon-reload, and rebooting gives: journalctl -b0 | grep ssh Mar 01 07:38:24 sempron sshd[426]: error: Bind to port on 192.168.1.54 failed: Cannot assign requested address. Mar 01 07:38:24 sempron sshd[426]: fatal: Cannot bind any address. Mar 01 07:38:24 sempron systemd[1]: ssh.service: Main process exited, code=exited, status=255/n/a Mar 01 07:38:24 sempron systemd[1]: ssh.service: Unit entered failed state. Mar 01 07:38:24 sempron systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 01 07:38:24 sempron systemd[1]: ssh.service: Service hold-off time over, scheduling restart. Mar 01 07:38:25 sempron sshd[437]: error: Bind to port on 192.168.1.54 failed: Cannot assign requested address.
Bug#908927: Debian Linux version 3.16.0-7-586 (3.16.59-1) gives a partial fix for SMB 3.0 mounts
Ok, I installed 3.16.0-7-586 (3.16.59-1) which allowed me to mount the remote share using the version 3.0 protocol, but it got some console errors and a kernel oops in the journal, so there are still some issues here: Oct 05 04:30:38 pentium kernel: Linux version 3.16.0-7-586 (debian-ker...@lists.debian.org) (gcc version 4.9.2 (Debian 4.9.2-10+deb8u1) ) #1 Debian 3.16.59-1 (2018-10-03) ... Oct 05 04:35:08 pentium systemd[1]: Got automount request for /mnt/share, triggered by 954 (mount.cifs) Oct 05 04:35:08 pentium systemd[1]: Mounting /mnt/share... Oct 05 04:35:08 pentium kernel: Key type dns_resolver registered Oct 05 04:35:08 pentium kernel: FS-Cache: Netfs 'cifs' registered for caching Oct 05 04:35:08 pentium kernel: Key type cifs.spnego registered Oct 05 04:35:08 pentium kernel: Key type cifs.idmap registered Oct 05 04:35:09 pentium systemd[1]: Mounted /mnt/share. Oct 05 04:35:09 pentium kernel: BUG: unable to handle kernel NULL pointer dereference at 0034 Oct 05 04:35:09 pentium kernel: IP: [] crypto_shash_setkey+0xe/0xb0 Oct 05 04:35:09 pentium kernel: *pde = Oct 05 04:35:09 pentium kernel: Oops: [#1] Oct 05 04:35:09 pentium kernel: Modules linked in: arc4 ecb md4 hmac nls_utf8 cifs dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc ppdev snd_emu10k1 snd_util_mem snd_rawmidi snd_hwdep snd_seq_device snd_ac97_codec snd_pcm snd_timer evdev snd pcspkr serio_raw soundcore ac97_bus emu10k1_gp gameport parport_pc parport processor button fuse autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid sg hid sd_mod sr_mod crc_t10dif crct10dif_generic cdrom crct10dif_common ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore i2c_piix4 3c59x scsi_mod mii i2c_core thermal fan usb_common thermal_sys floppy Oct 05 04:35:09 pentium kernel: CPU: 0 PID: 954 Comm: mount.cifs Not tainted 3.16.0-7-586 #1 Debian 3.16.59-1 Oct 05 04:35:09 pentium kernel: Hardware name: /i430TX-SMC669, BIOS 4.51 PG 07/20/98 Oct 05 04:35:09 pentium kernel: task: ccf7e5a0 ti: cc06c000 task.ti: cc06c000 Oct 05 04:35:09 pentium kernel: EIP: 0060:[] EFLAGS: 00010296 CPU: 0 Oct 05 04:35:09 pentium kernel: EIP is at crypto_shash_setkey+0xe/0xb0 Oct 05 04:35:09 pentium kernel: EAX: EBX: c035e8e0 ECX: 0010 EDX: cd9630c4 Oct 05 04:35:09 pentium kernel: ESI: cc06dd18 EDI: cbffa000 EBP: cc06dc30 ESP: cc06dc18 Oct 05 04:35:09 pentium kernel: DS: 007b ES: 007b FS: GS: 00e0 SS: 0068 Oct 05 04:35:09 pentium kernel: CR0: 8005003b CR2: 0034 CR3: 0c03c000 CR4: 0010 Oct 05 04:35:09 pentium kernel: Stack: Oct 05 04:35:09 pentium kernel: 0246 c10f0692 00011200 c035e8e0 cc06dd18 cbffa000 cc06dc7c d0ee2e39 Oct 05 04:35:09 pentium kernel: c10f0692 0011 cc06dcd0 c035e8e0 cbffa008 c6711f1a 0002 c15e3ac0 Oct 05 04:35:09 pentium kernel: 0246 f439365f cdd23340 cd963000 Oct 05 04:35:09 pentium kernel: Call Trace: Oct 05 04:35:09 pentium kernel: [] ? mempool_alloc+0x42/0x120 Oct 05 04:35:09 pentium kernel: [] ? smb3_calc_signature+0xb9/0x2a0 [cifs] Oct 05 04:35:09 pentium kernel: [] ? mempool_alloc+0x42/0x120 Oct 05 04:35:09 pentium kernel: [] ? smb2_sign_rqst+0x2f/0x60 [cifs] Oct 05 04:35:09 pentium kernel: [] ? smb2_setup_request+0x8c/0x130 [cifs] Oct 05 04:35:09 pentium kernel: [] ? SendReceive2+0xac/0x3f0 [cifs] Oct 05 04:35:09 pentium kernel: [] ? set_next_entity+0x52/0x70 Oct 05 04:35:09 pentium kernel: [] ? SMB2_ioctl+0x133/0x2e0 [cifs] Oct 05 04:35:09 pentium kernel: [] ? smb3_validate_negotiate+0x123/0x310 [cifs] Oct 05 04:35:09 pentium kernel: [] ? SMB2_tcon+0x261/0x480 [cifs] Oct 05 04:35:09 pentium kernel: [] ? kstrdup+0x3a/0x50 Oct 05 04:35:09 pentium kernel: [] ? smb2_writev_callback+0xe0/0xe0 [cifs] Oct 05 04:35:09 pentium kernel: [] ? cifs_get_tcon+0x192/0x400 [cifs] Oct 05 04:35:09 pentium kernel: [] ? cifs_mount+0x49d/0xc40 [cifs] Oct 05 04:35:09 pentium kernel: [] ? cifs_do_mount+0xc9/0x5b0 [cifs] Oct 05 04:35:09 pentium kernel: [] ? cifs_drop_inode+0x40/0x40 [cifs] Oct 05 04:35:09 pentium kernel: [] ? mount_fs+0x36/0x190 Oct 05 04:35:10 pentium kernel: [] ? kstrdup+0x3a/0x50 Oct 05 04:35:10 pentium kernel: [] ? vfs_kern_mount+0x48/0xf0 Oct 05 04:35:10 pentium kernel: [] ? do_mount+0x1e8/0xa60 Oct 05 04:35:10 pentium kernel: [] ? strndup_user+0x39/0xc0 Oct 05 04:35:10 pentium kernel: [] ? copy_mount_options+0x2f/0x1c0 Oct 05 04:35:10 pentium kernel: [] ? SyS_mount+0x9c/0xf0 Oct 05 04:35:10 pentium kernel: [] ? syscall_call+0x10/0x10 Oct 05 04:35:10 pentium kernel: Code: 26 00 8b 55 f0 83 c4 10 5b 5e 89 d0 5f 5d c3 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 89 e5 57 56 53 83 ec 0c 3e 8d 74 26 00 <8b> 78 34 89 4d f0 89 c3 89 d6 8b 4f 1c 85 ca 74 59 89 c8 ba d0 Oct 05 04:35:10 pentium kernel: EIP: [] crypto_shash_setkey+0xe/0xb0 SS:ESP 0068:cc06dc18 Oct 05 04:35:10 pentium kernel: CR2: 0034 Oct 05 04:35:10 pentium kernel: ---[ end
Bug#908927: Issue seems fixed on Arch Linux machine by kernel 3.16.58-1-ARCH
I've updated the kernel on the Arch Linux box which was experiencing the same issue to: 3.16.58-1-ARCH This seems to fix the issue on that machine, so hopefully the fix is on the way here also. Thanks Andrew
Bug#908927: Linux Kernel oops in jessie 8.11 linux-image-3.16.0-6-i586 (3.16.57-2) mounting cifs share with vers=3.0
Package: linux-image Version: 3.16.0-6-i586 I am seeing a kernel NULL pointer deference mounting a cifs share on jessie 8.11. This only occurs when you try to mount with vers=3.0 in the fstab file://192.168.1.30/share /mnt/share cifs defaults,noauto,noserverino,x-systemd.automount,rw,uid=aroberts,gid=aroberts,credentials=/etc/cifs-credentials,vers=3.0 0 0 using vers=2.1 works ok. This bug also affects the SAME kernel (3.16.57-2) on ArchLinux ARM (odroid-c2, aarch64). Other boxes with different kernels (Debian stretch, ArchLinux raspberry pi, Fedora etc) are all fine, as was a previous kernel on odroid (3.14). See odroid-c2 bug report at Arch Linux Arm: https://archlinuxarm.org/forum/viewtopic.php?f=15=13045 The shares are served from a Centos (RHEL 7.5.1804 x86_64) box with the following options in the smb.conf file: hosts deny = ALL hosts allow = 127. 192.168.1. nt pipe support = no client max protocol = SMB3 client min protocol = SMB2_10 server min protocol = SMB2_10 I'm trying to use vers=3.0 as a CERT advisory suggested that previous versions of the protocol are not secure, so this is a securty issue. Obviously I'm still using debian 8.11 as this is a i586 box... uname -a Linux pentium 3.16.0-6-586 #1 Debian 3.16.57-2 (2018-07-14) i586 GNU/Linux journalctl -b0 ...Sep 16 06:06:38 pentium kernel: BUG: unable to handle kernel NULL pointer dereference at 0034 Sep 16 06:06:38 pentium kernel: IP: [] crypto_shash_setkey+0xe/0xb0 Sep 16 06:06:38 pentium kernel: *pde = Sep 16 06:06:38 pentium kernel: Oops: [#1] Sep 16 06:06:38 pentium kernel: Modules linked in: arc4 ecb md4 hmac nls_utf8 isofs udf crc_itu_t cifs dns_resolver nfsd auth_rpcgss oid_registry nfs_acl nfs lockd fscache sunrpc ppdev snd_emu10k1 snd_util_mem snd_rawmidi snd_hwdep snd_seq_device snd_ac97_codec evdev snd_pcm snd_timer serio_raw snd pcspkr soundcore emu10k1_gp ac97_bus gameport parport_pc parport processor button fuse autofs4 ext4 crc16 mbcache jbd2 hid_generic usbhid sg hid sd_mod sr_mod crc_t10dif crct10dif_generic cdrom crct10dif_common ata_generic ata_piix uhci_hcd libata ehci_hcd usbcore i2c_piix4 scsi_mod 3c59x mii i2c_core usb_common thermal fan thermal_sys floppy Sep 16 06:06:38 pentium kernel: CPU: 0 PID: 3932 Comm: mount.cifs Not tainted 3.16.0-6-586 #1 Debian 3.16.57-2 Sep 16 06:06:38 pentium kernel: Hardware name: /i430TX-SMC669, BIOS 4.51 PG 07/20/98 Sep 16 06:06:38 pentium kernel: task: cd184500 ti: cfbc task.ti: cfbc Sep 16 06:06:38 pentium kernel: EIP: 0060:[] EFLAGS: 00010296 CPU: 0 Sep 16 06:06:38 pentium kernel: EIP is at crypto_shash_setkey+0xe/0xb0 Sep 16 06:06:38 pentium kernel: EAX: EBX: cfab81e0 ECX: 0010 EDX: cd8daac4 Sep 16 06:06:38 pentium kernel: ESI: cfbc1d18 EDI: cdba4000 EBP: cfbc1c30 ESP: cfbc1c18 Sep 16 06:06:38 pentium kernel: DS: 007b ES: 007b FS: GS: 00e0 SS: 0068 Sep 16 06:06:38 pentium kernel: CR0: 8005003b CR2: 0034 CR3: 0fa79000 CR4: 0010 Sep 16 06:06:38 pentium kernel: Stack: Sep 16 06:06:38 pentium kernel: 0246 c10efd02 00011200 cfab81e0 cfbc1d18 cdba4000 cfbc1c7c d0f16e39 Sep 16 06:06:38 pentium kernel: c10efd02 0082 cfbc1cd0 cfab81e0 cdba4008 f85388a6 0002 c15e3ac0 Sep 16 06:06:38 pentium kernel: 0246 9b528262 cfa234c0 cd8daa00 Sep 16 06:06:38 pentium kernel: Call Trace: Sep 16 06:06:38 pentium kernel: [] ? mempool_alloc+0x42/0x120 Sep 16 06:06:38 pentium kernel: [] ? smb3_calc_signature+0xb9/0x2a0 [cifs] Sep 16 06:06:38 pentium kernel: [] ? mempool_alloc+0x42/0x120 Sep 16 06:06:38 pentium kernel: [] ? smb2_sign_rqst+0x2f/0x60 [cifs] Sep 16 06:06:38 pentium kernel: [] ? smb2_setup_request+0x8c/0x130 [cifs] Sep 16 06:06:38 pentium kernel: [] ? SendReceive2+0xac/0x3f0 [cifs] Sep 16 06:06:38 pentium kernel: [] ? set_security_override_from_ctx+0x7/0x40 Sep 16 06:06:38 pentium kernel: [] ? SMB2_ioctl+0x133/0x2e0 [cifs] Sep 16 06:06:38 pentium kernel: [] ? smb3_validate_negotiate+0x123/0x310 [cifs] Sep 16 06:06:38 pentium kernel: [] ? SMB2_tcon+0x261/0x480 [cifs] Sep 16 06:06:38 pentium kernel: [] ? kstrdup+0x3a/0x50 Sep 16 06:06:38 pentium kernel: [] ? smb2_writev_callback+0xe0/0xe0 [cifs] Sep 16 06:06:38 pentium kernel: [] ? cifs_get_tcon+0x192/0x400 [cifs] Sep 16 06:06:38 pentium kernel: [] ? cifs_mount+0x49d/0xc40 [cifs] Sep 16 06:06:38 pentium kernel: [] ? cifs_do_mount+0xc9/0x5b0 [cifs] Sep 16 06:06:38 pentium kernel: [] ? cifs_drop_inode+0x40/0x40 [cifs] Sep 16 06:06:38 pentium kernel: [] ? mount_fs+0x36/0x190 Sep 16 06:06:38 pentium kernel: [] ? kstrdup+0x3a/0x50 Sep 16 06:06:38 pentium kernel: [] ? vfs_kern_mount+0x48/0xf0 Sep 16 06:06:38 pentium kernel: [] ? do_mount+0x1e8/0xa60 Sep 16 06:06:38 pentium kernel: [] ? strndup_user+0x39/0xc0 Sep 16 06:06:38 pentium kernel: [] ? copy_mount_options+0x2f/0x1c0 Sep 16 06:06:38 pentium kernel:
