This is a follow-up to the problems reported in the following thread that already discussed an issue with socket activation for the ssh.service.
SSHd now uses socket-based activation (Ubuntu 22.10 and later) https://discourse.ubuntu.com/t/sshd-now-uses-socket-based-activation-ubuntu-22-10-and-later/30189/4 -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to openssh in Ubuntu. https://bugs.launchpad.net/bugs/2055806 Title: sshd.service ssh.socket systemd-tmpfiles-setup:Before= - Missing privilege separation directory: /run/sshd Status in openssh package in Ubuntu: New Bug description: Symptom: ssh.service is not running and not accepting new SSH connections and returns socket closed error upon attempted connection. ssh.service shows a failed with error message "Missing privilege separation directory: /run/sshd". The following line is needed in the SystemD Unit configuration of the ssh.service file for the openssh-server package to fix what appears to be a race condition when ssh.service is sometimes started before systemd-tmpfiles-setup.service is finished processing the file in /usr/lib/tmpfiles.d/openssh.server.conf that is required to create the /run/sshd directory that is required by ssh.service to start property. There might be more depth or an alternative reason to why /run/sshd/ still didn't exist after using ```systemctl reset-failed ssh.service```. So more investigation is needed by someone with more background in systemd-tmpfiles-setup.service and openssh-server package after it when to socket activation. = Error Messages = ``` $journalctl -u ssh.service -b-1 Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 21:30:56 server sshd[1271]: Missing privilege separation directory: /run/sshd Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:56 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1. Mar 02 21:30:56 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 21:30:56 server sshd[1369]: Missing privilege separation directory: /run/sshd Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:56 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2. Mar 02 21:30:56 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 21:30:56 server sshd[1454]: Missing privilege separation directory: /run/sshd Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3. Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:57 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 21:30:57 server sshd[1465]: Missing privilege separation directory: /run/sshd Mar 02 21:30:57 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4. Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:57 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 21:30:57 server sshd[1475]: Missing privilege separation directory: /run/sshd Mar 02 21:30:57 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5. Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 21:30:57 server systemd[1]: ssh.service: Start request repeated too quickly. Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:25 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:25 server sshd[47238]: Missing privilege separation directory: /run/sshd Mar 02 22:19:25 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:25 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:25 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:25 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1. Mar 02 22:19:25 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:25 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:25 server sshd[47240]: Missing privilege separation directory: /run/sshd Mar 02 22:19:25 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:25 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:25 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2. Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:26 server sshd[47241]: Missing privilege separation directory: /run/sshd Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3. Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:26 server sshd[47242]: Missing privilege separation directory: /run/sshd Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4. Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:26 server sshd[47243]: Missing privilege separation directory: /run/sshd Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5. Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:26 server systemd[1]: ssh.service: Start request repeated too quickly. Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:37 server sshd[47253]: Missing privilege separation directory: /run/sshd Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:37 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 1. Mar 02 22:19:37 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:37 server sshd[47254]: Missing privilege separation directory: /run/sshd Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:37 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 2. Mar 02 22:19:37 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:37 server sshd[47255]: Missing privilege separation directory: /run/sshd Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 3. Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:38 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:38 server sshd[47256]: Missing privilege separation directory: /run/sshd Mar 02 22:19:38 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 4. Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:38 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell server... Mar 02 22:19:38 server sshd[47257]: Missing privilege separation directory: /run/sshd Mar 02 22:19:38 server systemd[1]: ssh.service: Control process exited, code=exited, status=255/EXCEPTION Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart counter is at 5. Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:38 server systemd[1]: ssh.service: Start request repeated too quickly. Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. Mar 02 22:19:45 server systemd[1]: ssh.service: Start request repeated too quickly. Mar 02 22:19:45 server systemd[1]: ssh.service: Failed with result 'exit-code'. Mar 02 22:19:45 server systemd[1]: Failed to start ssh.service - OpenBSD Secure Shell server. ``` Note: Errors showed up a total of 3x5 times because the ```systemctl reset-failed ssh.service``` command was used and it still showed up. = Add to Package SystemD Unit File = ``` ''/lib/systemd/system/ssh.service'' [Unit] After=systemd-tmpfiles-setup.service ``` = System Configuration = ``` ''lsb_release -rd'' No LSB modules are available. Description: Ubuntu 23.10 Release: 23.10 ``` ``` ''cat /etc/lsb-release'' DISTRIB_ID=Ubuntu DISTRIB_RELEASE=23.10 DISTRIB_CODENAME=mantic DISTRIB_DESCRIPTION="Ubuntu 23.10" ``` ``` ''apt-cache policy openssh-server'' openssh-server: Installed: 1:9.3p1-1ubuntu3.2 Candidate: 1:9.3p1-1ubuntu3.2 Version table: *** 1:9.3p1-1ubuntu3.2 500 500 http://us.archive.ubuntu.com/ubuntu mantic-updates/main amd64 Packages 500 http://security.ubuntu.com/ubuntu mantic-security/main amd64 Packages 100 /var/lib/dpkg/status 1:9.3p1-1ubuntu3 500 500 http://us.archive.ubuntu.com/ubuntu mantic/main amd64 Packages ``` = Package Configuration Problems = ``` ''systemctl --property=After show ssh.service'' After=network.target sysinit.target basic.target auditd.service system.slice ssh.socket pollinate.service systemd-journald.socket ``` ``` ''systemctl --no-pager --property=Before show systemd-tmpfiles-setup.service'' Before=ssh.service man-db.service logrotate.service systemd-timesyncd.service vgauth.service ModemManager.service fwupd.service e2scrub_reap.service initrd-switch-root.target systemd-update-utmp.service bluetooth.service open-vm-tools.service systemd-resolved.service polkit.service systemd-logind.service cockpit.service sysinit.target upower.service shutdown.target ``` **Note: ssh.service needs to be in the Before= clause above to fix the race condition of the ssh.service starting before the /usr/lib/tmpfiles.d/openssh-server.conf is processed by systemd- tmpfiles-setup.service during start-up sequence otherwise /run/sshd directory won't exist.** ``` ''cat /usr/lib/tmpfiles.d/openssh-server.conf'' #Type Path Mode UID GID Age Arguments D /run/sshd 0755 root root - - ``` = Temporary Solution = Create the directory and the Unit file with the After= clause to populate the Before= clause for systemd-tmpfiles-setup.service ``` ''mkdir -p /etc/systemd/system/ssh.service.d'' cat >/etc/systemd/system/ssh.service.d/after.conf <<EOF [Unit] After=systemd-tmpfiles-setup.service EOF ``` Verify the file. ``` ''cat /etc/systemd/system/ssh.service.d/after.conf'' [Unit] After=systemd-tmpfiles-setup.service ``` ``` Reload SystemD config ``` systemctl daemon-reload ``` Verify new active config. ``` ''systemctl --no-pager --property=Before show systemd-tmpfiles-setup.service'' Before=... ssh.service ... ``` ``` ''systemctl --property=After show ssh.service'' After=... systemd-tmpfiles-setup.service ...``` === End of Bug Report === ProblemType: Bug DistroRelease: Ubuntu 23.10 Package: openssh-server 1:9.3p1-1ubuntu3.2 ProcVersionSignature: Ubuntu 6.5.0-21.21-generic 6.5.8 Uname: Linux 6.5.0-21-generic x86_64 NonfreeKernelModules: zfs ApportVersion: 2.27.0-0ubuntu5 Architecture: amd64 CasperMD5CheckResult: unknown Date: Sun Mar 3 02:22:39 2024 InstallationDate: Installed on 2020-12-14 (1174 days ago) InstallationMedia: Ubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731) ProcEnviron: LANG=en_US.UTF-8 PATH=(custom, no user) SHELL=/bin/bash TERM=putty SourcePackage: openssh UpgradeStatus: Upgraded to mantic on 2024-02-26 (6 days ago) To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2055806/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
