Re: Troubleshooting timedatectl and hostnamectl failed to activate service: timed out
It appears that I can fix this by editing the unit file and changing: ProtectSystem=strict to: ProtectSystem=full I'm not sure why that is but a resolution is good enough for me. Following up on my own thread in case this helps someone else in the future. Best, Sean On Thu, Dec 14, 2023 at 12:02 PM Sean Caron wrote: > Sorry, perhaps apparmor is not completely disabled, but I don't think it's > enforcing. I tried shutting it off completely with: > > systemctl stop apparmor > > And that doesn't seem to have made a difference. > > Best, > > Sean > > > On Thu, Dec 14, 2023 at 11:57 AM Sean Caron wrote: > >> Hi Mantas, >> >> Thanks for the suggestions! I took a look and I'm seeing entries like the >> following in the logs: >> >> Starting Hostname Service... >> systemd-hostnamed.service: Failed to set up mount namespacing: >> /run/systemd/unit-root/: Invalid argument >> systemd-hostnamed.service: Failed at step NAMESPACE spawning >> /lib/systemd/systemd-hostnamed: Invalid argument >> systemd-hostnamed.service: Main process exited, code=exited, >> status=226/NAMESPACE >> systemd-hostnamed.service: Failed with result 'exit-code'. >> Failed to start Hostname Service. >> >> Starting Time & Date Service... >> systemd-timedated.service: Failed to set up mount namespacing: >> /run/systemd/unit-root/: Invalid argument >> systemd-timedated.service: Failed at step NAMESPACE spawning >> /lib/systemd/systemd-timedated: Invalid argument >> systemd-timedated.service: Main process exited, code=exited, >> status=226/NAMESPACE >> systemd-timedated.service: Failed with result 'exit-code'. >> Failed to start Time & Date Service. >> >> Apparmor is disabled on all of our systems. >> >> The /run/systemd/unit-root directory exists on both working and >> nonworking systems and the ownership and permissions are identical on >> working and nonworking systems. >> >> I'm unfortunately not very conversant with everything that systemd does >> behind the scenes with this namespacing stuff. Does this raise any obvious >> flags? Any ideas for how I could further debug and/or resolve this would be >> so greatly appreciated! >> >> Best, >> >> Sean >> >> On Wed, Dec 13, 2023 at 1:22 PM Mantas Mikulėnas >> wrote: >> >>> Activation is not client-side, it's handled automatically by dbus-daemon >>> – which either spawns the service directly or starts it as a systemd >>> service. >>> >>> In this case, check whether your logs show systemd-hostnamed.service >>> attempting to start; either it fails to start (missing libraries? >>> Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?). >>> >>> On Wed, Dec 13, 2023, 19:45 Sean Caron wrote: >>> Hi everyone, I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245 (245.4-4ubuntu3.22). I have some systems where I am receiving the following error messages when people attempt to use timedatectl or hostnamectl: Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) Failed to query system properties: Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms) I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and it didn't really give me anything useful for determining the root cause of the issue. Here's an example of that output for timedatectl status: Bus n/a: changing state UNSET → OPENING Bus n/a: changing state OPENING → AUTHENTICATING Bus n/a: changing state AUTHENTICATING → HELLO Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a Got message type=method_return sender=org.freedesktop.DBus destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a Bus n/a: changing state HELLO → RUNNING Sent message type=method_call sender=n/a destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a Got message type=error sender=org.freedesktop.DBus destination=:1.15318 path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) Bus n/a: changing state RUNNING → CLOSED I read that sometimes these issues can be caused by filesystem permissions on subdirectories in /var such as /var/tmp
Re: Troubleshooting timedatectl and hostnamectl failed to activate service: timed out
Sorry, perhaps apparmor is not completely disabled, but I don't think it's enforcing. I tried shutting it off completely with: systemctl stop apparmor And that doesn't seem to have made a difference. Best, Sean On Thu, Dec 14, 2023 at 11:57 AM Sean Caron wrote: > Hi Mantas, > > Thanks for the suggestions! I took a look and I'm seeing entries like the > following in the logs: > > Starting Hostname Service... > systemd-hostnamed.service: Failed to set up mount namespacing: > /run/systemd/unit-root/: Invalid argument > systemd-hostnamed.service: Failed at step NAMESPACE spawning > /lib/systemd/systemd-hostnamed: Invalid argument > systemd-hostnamed.service: Main process exited, code=exited, > status=226/NAMESPACE > systemd-hostnamed.service: Failed with result 'exit-code'. > Failed to start Hostname Service. > > Starting Time & Date Service... > systemd-timedated.service: Failed to set up mount namespacing: > /run/systemd/unit-root/: Invalid argument > systemd-timedated.service: Failed at step NAMESPACE spawning > /lib/systemd/systemd-timedated: Invalid argument > systemd-timedated.service: Main process exited, code=exited, > status=226/NAMESPACE > systemd-timedated.service: Failed with result 'exit-code'. > Failed to start Time & Date Service. > > Apparmor is disabled on all of our systems. > > The /run/systemd/unit-root directory exists on both working and nonworking > systems and the ownership and permissions are identical on working and > nonworking systems. > > I'm unfortunately not very conversant with everything that systemd does > behind the scenes with this namespacing stuff. Does this raise any obvious > flags? Any ideas for how I could further debug and/or resolve this would be > so greatly appreciated! > > Best, > > Sean > > On Wed, Dec 13, 2023 at 1:22 PM Mantas Mikulėnas > wrote: > >> Activation is not client-side, it's handled automatically by dbus-daemon >> – which either spawns the service directly or starts it as a systemd >> service. >> >> In this case, check whether your logs show systemd-hostnamed.service >> attempting to start; either it fails to start (missing libraries? >> Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?). >> >> On Wed, Dec 13, 2023, 19:45 Sean Caron wrote: >> >>> Hi everyone, >>> >>> I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245 >>> (245.4-4ubuntu3.22). >>> >>> I have some systems where I am receiving the following error messages >>> when people attempt to use timedatectl or hostnamectl: >>> >>> >>> Failed to query server: Failed to activate service >>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) >>> >>> Failed to query system properties: Failed to activate service >>> 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms) >>> >>> >>> I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and >>> it didn't really give me anything useful for determining the root cause of >>> the issue. Here's an example of that output for timedatectl status: >>> >>> >>> Bus n/a: changing state UNSET → OPENING >>> Bus n/a: changing state OPENING → AUTHENTICATING >>> Bus n/a: changing state AUTHENTICATING → HELLO >>> Sent message type=method_call sender=n/a >>> destination=org.freedesktop.DBus path=/org/freedesktop/DBus >>> interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 >>> signature=n/a error-name=n/a error-message=n/a >>> Got message type=method_return sender=org.freedesktop.DBus >>> destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1 >>> reply_cookie=1 signature=s error-name=n/a error-message=n/a >>> Bus n/a: changing state HELLO → RUNNING >>> Sent message type=method_call sender=n/a >>> destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1 >>> interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 >>> reply_cookie=0 signature=s error-name=n/a error-message=n/a >>> Got message type=error sender=org.freedesktop.DBus destination=:1.15318 >>> path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s >>> error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to >>> activate service 'org.freedesktop.timedate1': timed out >>> (service_start_timeout=25000ms) >>> Failed to query server: Failed to activate service >>> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) >>> Bus n/a: changing state RUNNING → CLOSED >>> >>> >>> I read that sometimes these issues can be caused by filesystem >>> permissions on subdirectories in /var such as /var/tmp or /var/lib/systemd >>> but I checked these and compared against a working system and I don't see >>> any obvious differences. >>> >>> I have tried using strace on timedatectl and hostnamectl to try and see >>> what's hanging things up but that hasn't really provided any fruitful >>> direction, either. >>> >>> I didn't really know this was occurring until an end user reported it to >>> me so I don't necessarily know how long the issue has been
Re: Troubleshooting timedatectl and hostnamectl failed to activate service: timed out
Hi Mantas, Thanks for the suggestions! I took a look and I'm seeing entries like the following in the logs: Starting Hostname Service... systemd-hostnamed.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Invalid argument systemd-hostnamed.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-hostnamed: Invalid argument systemd-hostnamed.service: Main process exited, code=exited, status=226/NAMESPACE systemd-hostnamed.service: Failed with result 'exit-code'. Failed to start Hostname Service. Starting Time & Date Service... systemd-timedated.service: Failed to set up mount namespacing: /run/systemd/unit-root/: Invalid argument systemd-timedated.service: Failed at step NAMESPACE spawning /lib/systemd/systemd-timedated: Invalid argument systemd-timedated.service: Main process exited, code=exited, status=226/NAMESPACE systemd-timedated.service: Failed with result 'exit-code'. Failed to start Time & Date Service. Apparmor is disabled on all of our systems. The /run/systemd/unit-root directory exists on both working and nonworking systems and the ownership and permissions are identical on working and nonworking systems. I'm unfortunately not very conversant with everything that systemd does behind the scenes with this namespacing stuff. Does this raise any obvious flags? Any ideas for how I could further debug and/or resolve this would be so greatly appreciated! Best, Sean On Wed, Dec 13, 2023 at 1:22 PM Mantas Mikulėnas wrote: > Activation is not client-side, it's handled automatically by dbus-daemon – > which either spawns the service directly or starts it as a systemd service. > > In this case, check whether your logs show systemd-hostnamed.service > attempting to start; either it fails to start (missing libraries? > Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?). > > On Wed, Dec 13, 2023, 19:45 Sean Caron wrote: > >> Hi everyone, >> >> I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245 >> (245.4-4ubuntu3.22). >> >> I have some systems where I am receiving the following error messages >> when people attempt to use timedatectl or hostnamectl: >> >> >> Failed to query server: Failed to activate service >> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) >> >> Failed to query system properties: Failed to activate service >> 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms) >> >> >> I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and it >> didn't really give me anything useful for determining the root cause of the >> issue. Here's an example of that output for timedatectl status: >> >> >> Bus n/a: changing state UNSET → OPENING >> Bus n/a: changing state OPENING → AUTHENTICATING >> Bus n/a: changing state AUTHENTICATING → HELLO >> Sent message type=method_call sender=n/a destination=org.freedesktop.DBus >> path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello >> cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a >> Got message type=method_return sender=org.freedesktop.DBus >> destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1 >> reply_cookie=1 signature=s error-name=n/a error-message=n/a >> Bus n/a: changing state HELLO → RUNNING >> Sent message type=method_call sender=n/a >> destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1 >> interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 >> reply_cookie=0 signature=s error-name=n/a error-message=n/a >> Got message type=error sender=org.freedesktop.DBus destination=:1.15318 >> path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s >> error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to >> activate service 'org.freedesktop.timedate1': timed out >> (service_start_timeout=25000ms) >> Failed to query server: Failed to activate service >> 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) >> Bus n/a: changing state RUNNING → CLOSED >> >> >> I read that sometimes these issues can be caused by filesystem >> permissions on subdirectories in /var such as /var/tmp or /var/lib/systemd >> but I checked these and compared against a working system and I don't see >> any obvious differences. >> >> I have tried using strace on timedatectl and hostnamectl to try and see >> what's hanging things up but that hasn't really provided any fruitful >> direction, either. >> >> I didn't really know this was occurring until an end user reported it to >> me so I don't necessarily know how long the issue has been occurring or >> have a change in mind that could have broken things. I'm not sure if the >> upgrade from Ubuntu 18 to Ubuntu 20 broke it, or if some security >> configuration broke it. Or perhaps there is a missing dependency package on >> the broken systems? >> >> Could anyone out there please provide a little bit more guidance on how I >> might troubleshoot this and determine the root cause of the issue? I >>
Re: Troubleshooting timedatectl and hostnamectl failed to activate service: timed out
Activation is not client-side, it's handled automatically by dbus-daemon – which either spawns the service directly or starts it as a systemd service. In this case, check whether your logs show systemd-hostnamed.service attempting to start; either it fails to start (missing libraries? Apparmor?) or dbus-daemon fails to contact systemd (pid1 crashed?). On Wed, Dec 13, 2023, 19:45 Sean Caron wrote: > Hi everyone, > > I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245 > (245.4-4ubuntu3.22). > > I have some systems where I am receiving the following error messages when > people attempt to use timedatectl or hostnamectl: > > > Failed to query server: Failed to activate service > 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) > > Failed to query system properties: Failed to activate service > 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms) > > > I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and it > didn't really give me anything useful for determining the root cause of the > issue. Here's an example of that output for timedatectl status: > > > Bus n/a: changing state UNSET → OPENING > Bus n/a: changing state OPENING → AUTHENTICATING > Bus n/a: changing state AUTHENTICATING → HELLO > Sent message type=method_call sender=n/a destination=org.freedesktop.DBus > path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello > cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a > Got message type=method_return sender=org.freedesktop.DBus > destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1 > reply_cookie=1 signature=s error-name=n/a error-message=n/a > Bus n/a: changing state HELLO → RUNNING > Sent message type=method_call sender=n/a > destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1 > interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 > reply_cookie=0 signature=s error-name=n/a error-message=n/a > Got message type=error sender=org.freedesktop.DBus destination=:1.15318 > path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s > error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to > activate service 'org.freedesktop.timedate1': timed out > (service_start_timeout=25000ms) > Failed to query server: Failed to activate service > 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) > Bus n/a: changing state RUNNING → CLOSED > > > I read that sometimes these issues can be caused by filesystem permissions > on subdirectories in /var such as /var/tmp or /var/lib/systemd but I > checked these and compared against a working system and I don't see any > obvious differences. > > I have tried using strace on timedatectl and hostnamectl to try and see > what's hanging things up but that hasn't really provided any fruitful > direction, either. > > I didn't really know this was occurring until an end user reported it to > me so I don't necessarily know how long the issue has been occurring or > have a change in mind that could have broken things. I'm not sure if the > upgrade from Ubuntu 18 to Ubuntu 20 broke it, or if some security > configuration broke it. Or perhaps there is a missing dependency package on > the broken systems? > > Could anyone out there please provide a little bit more guidance on how I > might troubleshoot this and determine the root cause of the issue? I > really hate to bother folks here but I'm feeling stuck. > > Thank you! > > Sean >
Troubleshooting timedatectl and hostnamectl failed to activate service: timed out
Hi everyone, I'm on Ubuntu 20.04 LTS, kernel version 5.4.0-163-generic, systemd 245 (245.4-4ubuntu3.22). I have some systems where I am receiving the following error messages when people attempt to use timedatectl or hostnamectl: Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) Failed to query system properties: Failed to activate service 'org.freedesktop.hostname1': timed out (service_start_timeout=25000ms) I tried setting SYSTEMD_LOG_LEVEL=debug and rerunning the commands and it didn't really give me anything useful for determining the root cause of the issue. Here's an example of that output for timedatectl status: Bus n/a: changing state UNSET → OPENING Bus n/a: changing state OPENING → AUTHENTICATING Bus n/a: changing state AUTHENTICATING → HELLO Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=Hello cookie=1 reply_cookie=0 signature=n/a error-name=n/a error-message=n/a Got message type=method_return sender=org.freedesktop.DBus destination=:1.15318 path=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 signature=s error-name=n/a error-message=n/a Bus n/a: changing state HELLO → RUNNING Sent message type=method_call sender=n/a destination=org.freedesktop.timedate1 path=/org/freedesktop/timedate1 interface=org.freedesktop.DBus.Properties member=GetAll cookie=2 reply_cookie=0 signature=s error-name=n/a error-message=n/a Got message type=error sender=org.freedesktop.DBus destination=:1.15318 path=n/a interface=n/a member=n/a cookie=3 reply_cookie=2 signature=s error-name=org.freedesktop.DBus.Error.TimedOut error-message=Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) Failed to query server: Failed to activate service 'org.freedesktop.timedate1': timed out (service_start_timeout=25000ms) Bus n/a: changing state RUNNING → CLOSED I read that sometimes these issues can be caused by filesystem permissions on subdirectories in /var such as /var/tmp or /var/lib/systemd but I checked these and compared against a working system and I don't see any obvious differences. I have tried using strace on timedatectl and hostnamectl to try and see what's hanging things up but that hasn't really provided any fruitful direction, either. I didn't really know this was occurring until an end user reported it to me so I don't necessarily know how long the issue has been occurring or have a change in mind that could have broken things. I'm not sure if the upgrade from Ubuntu 18 to Ubuntu 20 broke it, or if some security configuration broke it. Or perhaps there is a missing dependency package on the broken systems? Could anyone out there please provide a little bit more guidance on how I might troubleshoot this and determine the root cause of the issue? I really hate to bother folks here but I'm feeling stuck. Thank you! Sean
