Public bug reported:
Symptom: ssh.service is not running and not accepting new SSH
connections and returns socket closed error upon attempted connection.
ssh.service shows a failed with error message "Missing privilege
separation directory: /run/sshd".
The following line is needed in the SystemD Unit configuration of the
ssh.service file for the openssh-server package to fix what appears to
be a race condition when ssh.service is sometimes started before
systemd-tmpfiles-setup.service is finished processing the file in
/usr/lib/tmpfiles.d/openssh.server.conf that is required to create the
/run/sshd directory that is required by ssh.service to start property.
There might be more depth or an alternative reason to why /run/sshd/
still didn't exist after using ```systemctl reset-failed ssh.service```.
So more investigation is needed by someone with more background in
systemd-tmpfiles-setup.service and openssh-server package after it when
to socket activation.
= Error Messages =
```
$journalctl -u ssh.service -b-1
Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 21:30:56 server sshd[1271]: Missing privilege separation directory:
/run/sshd
Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 21:30:56 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 1.
Mar 02 21:30:56 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 21:30:56 server sshd[1369]: Missing privilege separation directory:
/run/sshd
Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 21:30:56 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 2.
Mar 02 21:30:56 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 21:30:56 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 21:30:56 server sshd[1454]: Missing privilege separation directory:
/run/sshd
Mar 02 21:30:56 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 21:30:56 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 21:30:56 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 3.
Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 21:30:57 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 21:30:57 server sshd[1465]: Missing privilege separation directory:
/run/sshd
Mar 02 21:30:57 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 4.
Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 21:30:57 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 21:30:57 server sshd[1475]: Missing privilege separation directory:
/run/sshd
Mar 02 21:30:57 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 21:30:57 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 5.
Mar 02 21:30:57 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 21:30:57 server systemd[1]: ssh.service: Start request repeated too
quickly.
Mar 02 21:30:57 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 21:30:57 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:25 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:25 server sshd[47238]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:25 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:25 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:25 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:25 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 1.
Mar 02 22:19:25 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:25 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:25 server sshd[47240]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:25 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:25 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:25 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 2.
Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:26 server sshd[47241]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 3.
Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:26 server sshd[47242]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 4.
Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:26 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:26 server sshd[47243]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:26 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:26 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 5.
Mar 02 22:19:26 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:26 server systemd[1]: ssh.service: Start request repeated too
quickly.
Mar 02 22:19:26 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:26 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:37 server sshd[47253]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:37 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 1.
Mar 02 22:19:37 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:37 server sshd[47254]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:37 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 2.
Mar 02 22:19:37 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:37 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:37 server sshd[47255]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:37 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:37 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:37 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 3.
Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:38 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:38 server sshd[47256]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:38 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 4.
Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:38 server systemd[1]: Starting ssh.service - OpenBSD Secure Shell
server...
Mar 02 22:19:38 server sshd[47257]: Missing privilege separation directory:
/run/sshd
Mar 02 22:19:38 server systemd[1]: ssh.service: Control process exited,
code=exited, status=255/EXCEPTION
Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:38 server systemd[1]: ssh.service: Scheduled restart job, restart
counter is at 5.
Mar 02 22:19:38 server systemd[1]: Stopped ssh.service - OpenBSD Secure Shell
server.
Mar 02 22:19:38 server systemd[1]: ssh.service: Start request repeated too
quickly.
Mar 02 22:19:38 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:38 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
Mar 02 22:19:45 server systemd[1]: ssh.service: Start request repeated too
quickly.
Mar 02 22:19:45 server systemd[1]: ssh.service: Failed with result 'exit-code'.
Mar 02 22:19:45 server systemd[1]: Failed to start ssh.service - OpenBSD Secure
Shell server.
```
Note: Errors showed up a total of 3x5 times because the ```systemctl
reset-failed ssh.service``` command was used and it still showed up.
= Add to Package SystemD Unit File =
```
''/lib/systemd/system/ssh.service''
[Unit]
After=systemd-tmpfiles-setup.service
```
= System Configuration =
```
''lsb_release -rd''
No LSB modules are available.
Description: Ubuntu 23.10
Release: 23.10
```
```
''cat /etc/lsb-release''
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=23.10
DISTRIB_CODENAME=mantic
DISTRIB_DESCRIPTION="Ubuntu 23.10"
```
```
''apt-cache policy openssh-server''
openssh-server:
Installed: 1:9.3p1-1ubuntu3.2
Candidate: 1:9.3p1-1ubuntu3.2
Version table:
*** 1:9.3p1-1ubuntu3.2 500
500 http://us.archive.ubuntu.com/ubuntu mantic-updates/main amd64
Packages
500 http://security.ubuntu.com/ubuntu mantic-security/main amd64
Packages
100 /var/lib/dpkg/status
1:9.3p1-1ubuntu3 500
500 http://us.archive.ubuntu.com/ubuntu mantic/main amd64 Packages
```
= Package Configuration Problems =
```
''systemctl --property=After show ssh.service''
After=network.target sysinit.target basic.target auditd.service system.slice
ssh.socket pollinate.service systemd-journald.socket
```
```
''systemctl --no-pager --property=Before show systemd-tmpfiles-setup.service''
Before=ssh.service man-db.service logrotate.service systemd-timesyncd.service
vgauth.service ModemManager.service fwupd.service e2scrub_reap.service
initrd-switch-root.target systemd-update-utmp.service bluetooth.service
open-vm-tools.service systemd-resolved.service polkit.service
systemd-logind.service cockpit.service sysinit.target upower.service
shutdown.target
```
**Note: ssh.service needs to be in the Before= clause above to fix the
race condition of the ssh.service starting before the
/usr/lib/tmpfiles.d/openssh-server.conf is processed by systemd-
tmpfiles-setup.service during start-up sequence otherwise /run/sshd
directory won't exist.**
```
''cat /usr/lib/tmpfiles.d/openssh-server.conf''
#Type Path Mode UID GID Age
Arguments
D /run/sshd 0755 root root - -
```
= Temporary Solution =
Create the directory and the Unit file with the After= clause to
populate the Before= clause for systemd-tmpfiles-setup.service
```
''mkdir -p /etc/systemd/system/ssh.service.d''
cat >/etc/systemd/system/ssh.service.d/after.conf <<EOF
[Unit]
After=systemd-tmpfiles-setup.service
EOF
```
Verify the file.
```
''cat /etc/systemd/system/ssh.service.d/after.conf''
[Unit]
After=systemd-tmpfiles-setup.service
```
```
Reload SystemD config
```
systemctl daemon-reload
```
Verify new active config.
```
''systemctl --no-pager --property=Before show systemd-tmpfiles-setup.service''
Before=... ssh.service ...
```
```
''systemctl --property=After show ssh.service''
After=... systemd-tmpfiles-setup.service ...```
=== End of Bug Report ===
ProblemType: Bug
DistroRelease: Ubuntu 23.10
Package: openssh-server 1:9.3p1-1ubuntu3.2
ProcVersionSignature: Ubuntu 6.5.0-21.21-generic 6.5.8
Uname: Linux 6.5.0-21-generic x86_64
NonfreeKernelModules: zfs
ApportVersion: 2.27.0-0ubuntu5
Architecture: amd64
CasperMD5CheckResult: unknown
Date: Sun Mar 3 02:22:39 2024
InstallationDate: Installed on 2020-12-14 (1174 days ago)
InstallationMedia: Ubuntu 20.04.1 LTS "Focal Fossa" - Release amd64 (20200731)
ProcEnviron:
LANG=en_US.UTF-8
PATH=(custom, no user)
SHELL=/bin/bash
TERM=putty
SourcePackage: openssh
UpgradeStatus: Upgraded to mantic on 2024-02-26 (6 days ago)
** Affects: openssh (Ubuntu)
Importance: Undecided
Status: New
** Tags: amd64 apport-bug mantic
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2055806
Title:
sshd.service ssh.socket systemd-tmpfiles-setup:Before= - Missing
privilege separation directory: /run/sshd
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/openssh/+bug/2055806/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
