[systemd-devel] Container restart issue: Failed to attach 1 to compat systemd cgroup
Hi all, I've come across an issue when restarting a systemd container, which I'm seeing on a CentOS 8.2 VM but not able to reproduce on an Ubuntu 20.04 VM (both cgroups v1). The failure looks as follows, hitting the warning condition at https://github.com/systemd/systemd/blob/v245/src/shared/cgroup-setup.c#L279: [root@localhost ubuntu-systemd]# podman run -it --privileged --name ubuntu --detach ubuntu-systemd 5e4ab2a36681c092f4ef937cf03b25a8d3d7b2fa530559bf4dac4079c84d0313 [root@localhost ubuntu-systemd]# podman restart ubuntu 5e4ab2a36681c092f4ef937cf03b25a8d3d7b2fa530559bf4dac4079c84d0313 [root@localhost ubuntu-systemd]# podman logs ubuntu | grep -B6 -A2 'Set hostname' systemd 245.4-4ubuntu3.19 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid) Detected virtualization podman. Detected architecture x86-64. Welcome to Ubuntu 20.04.5 LTS! Set hostname to <5e4ab2a36681>. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-modprobe.slice. -- systemd 245.4-4ubuntu3.19 running in system mode. (+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=hybrid) Detected virtualization podman. Detected architecture x86-64. Welcome to Ubuntu 20.04.5 LTS! Set hostname to <5e4ab2a36681>. *Failed to attach 1 to compat systemd cgroup /machine.slice/libpod-5e4ab2a36681c092f4ef937cf03b25a8d3d7b2fa530559bf4dac4079c84d0313.scope/init.scope: No such file or directory*[ OK ] Created slice system-getty.slice. If using docker instead of podman (still on CentOS 8.2) the container actually exits after restart (when hitting the code at https://github.com/systemd/systemd/blob/v245/src/core/cgroup.c#L2972): [root@localhost ubuntu-systemd]# docker logs ubuntu | grep -C5 'Set hostname' Detected virtualization docker. Detected architecture x86-64. Welcome to Ubuntu 20.04.5 LTS! Set hostname to <523caa1f03e9>. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-modprobe.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. -- Detected virtualization docker. Detected architecture x86-64. Welcome to Ubuntu 20.04.5 LTS! Set hostname to <523caa1f03e9>. *Failed to attach 1 to compat systemd cgroup /system.slice/docker-523caa1f03e9c96a6a12a55fb07df995c6e4b3a27e18585cbeda869b943ae728.scope/init.scope: No such file or directoryFailed to open pin file: No such file or directoryFailed to allocate manager object: No such file or directory[!!] Failed to allocate manager object.Exiting PID 1...* Does anyone know what might be causing this? Is it a systemd bug? I can copy the info into a GitHub issue if that's helpful. Thanks, Lewis
Re: [systemd-devel] Please help: timeout waiting for /dev/tty* console device
On Mon, Jan 09, 2023 at 01:37:58PM +0100, Lennart Poettering wrote: > On Fr, 06.01.23 19:15, Gabriel L. Somlo (gso...@gmail.com) wrote: > > > Hi, > > > > I'm failing to get a login prompt on the serial console of my system, > > because a few steps earlier serial-getty@.service fails due to a > > dependency on the actual tty device, which times out: > > > > [ TIME ] Timed out waiting for device …ttyLXU0.device - /dev/ttyLXU0. > > [DEPEND] Dependency failed for seri…ice - Serial Getty on ttyLXU0. > > So you are saying that the device actually *does* pop up eventually, > but your systems is simply so awfully slow that the default time-outs > are hit? Yeah, at 50MHz and 512MB RAM, it's just about the slowest, most resource-constrained thing that I'm willing to call a "computer", by virtue of its actually being able to (barely) boot Fedora's riscv f37 port... :) > If so, you solve this locally for dev-ttyLXU0.device by adding a > JobTimeoutSec= drop-in file (for the [Unit]) section. > > Or if you want to increase the time-out globally, consider setting > DefaultTimeoutStartSec= in /etc/systemd/system.conf to any value you > like. I went with "systemd.default_timeout_start_sec=360s" (as also suggested by Martin elsewhere in this thread). That actually fixed the tty issue, and also a bunch of other services that were failing earlier... Thanks again for the quick replies! --Gabriel
Re: [systemd-devel] sd_bus_get_timeout returns absolute time despite what's documented
Hey Lennart, Yes, I was referring to man pages I read a few years ago when I used sd_bus_get_timeout() for the first time. And, before posting this mail, I also checked on my current Linux development machine, which has systemd v247. So I was not that far :-) The fixed wording in v250 is correct, understandable and explicit to me. All good now. Thanks a lot. Stanislav. --- Original Message --- On Monday, January 9th, 2023 at 1:02 PM, Lennart Poettering wrote: > On Mo, 09.01.23 12:53, Lennart Poettering (lenn...@poettering.net) wrote: > > > https://www.freedesktop.org/software/systemd/man/sd_bus_get_fd.html#Description > > > > Note that the returned time-value is absolute, based of > > CLOCK_MONOTONIC and specified in microseconds. When converting > > this value in order to pass it as third argument to poll() (which > > expects relative milliseconds), care should be taken to convert to > > a relative time and use a division that rounds up to ensure the > > I/O polling operation doesn't sleep for shorter than necessary, > > which might result in unintended busy looping (alternatively, use > > ppoll(2) instead of plain poll(), which understands timeouts with > > nano-second granularity). > > > > That's pretty explicit already, no? > > > This was fixed in 2021 btw, 25060a570c106cf5a14a3268bb0d38d9feb7fdab > i.e. systemd 250. Upgrade! > > > (I mean, you have half a point, the first sentence of the explanation > > might people think this was a relative timeout, but we all read the > > full documentation, no, before actually using this API, no? ;-)) > > > > Anyway, will prep a fix that rewords the first sentence to make this > > clearer right away. > > > https://github.com/systemd/systemd/pull/25985 > > Lennart > > -- > Lennart Poettering, Berlin
Re: [systemd-devel] Please help: timeout waiting for /dev/tty* console device
On Fr, 06.01.23 19:15, Gabriel L. Somlo (gso...@gmail.com) wrote: > Hi, > > I'm failing to get a login prompt on the serial console of my system, > because a few steps earlier serial-getty@.service fails due to a > dependency on the actual tty device, which times out: > > [ TIME ] Timed out waiting for device …ttyLXU0.device - /dev/ttyLXU0. > [DEPEND] Dependency failed for seri…ice - Serial Getty on ttyLXU0. So you are saying that the device actually *does* pop up eventually, but your systems is simply so awfully slow that the default time-outs are hit? If so, you solve this locally for dev-ttyLXU0.device by adding a JobTimeoutSec= drop-in file (for the [Unit]) section. Or if you want to increase the time-out globally, consider setting DefaultTimeoutStartSec= in /etc/systemd/system.conf to any value you like. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] sd_bus_get_timeout returns absolute time despite what's documented
On Mo, 09.01.23 12:53, Lennart Poettering (lenn...@poettering.net) wrote: > https://www.freedesktop.org/software/systemd/man/sd_bus_get_fd.html#Description > > Note that the returned time-value is absolute, based of > CLOCK_MONOTONIC and specified in microseconds. When converting > this value in order to pass it as third argument to poll() (which > expects relative milliseconds), care should be taken to convert to > a relative time and use a division that rounds up to ensure the > I/O polling operation doesn't sleep for shorter than necessary, > which might result in unintended busy looping (alternatively, use > ppoll(2) instead of plain poll(), which understands timeouts with > nano-second granularity). > > That's pretty explicit already, no? This was fixed in 2021 btw, 25060a570c106cf5a14a3268bb0d38d9feb7fdab i.e. systemd 250. Upgrade! > (I mean, you have half a point, the first sentence of the explanation > might people think this was a relative timeout, but we all read the > full documentation, no, before actually using this API, no? ;-)) > > Anyway, will prep a fix that rewords the first sentence to make this > clearer right away. https://github.com/systemd/systemd/pull/25985 Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] sd_bus_get_timeout returns absolute time despite what's documented
On Sa, 07.01.23 10:35, Stanislav Angelovič (stanislav.angelo...@protonmail.com) wrote: > Hi everyone, > > According to sd_bus_get_timeout(3) man page, sd_bus_get_timeout() > returns "timeout in us to pass to poll()" and it also states that > "the returned time-value is relative". > > However, in reality, this function returns an absolute time point > (a-ka relative to the monotonic clock epoch). Hence, the returned > value cannot be simply converted to milliseconds and given to > poll(). It must first be subtracted from the current monotonic clock > time. > > I followed the documentation and had a bug in my code for some time > :) Shall the documentation be updated to mention the necessity of > conversion from absolute to relative? Or does sd_bus_get_timeout() > have a bug and shall this function do the subtraction for the user > and return relative time? Hmm, the man page says this currently: https://www.freedesktop.org/software/systemd/man/sd_bus_get_fd.html#Description Note that the returned time-value is absolute, based of CLOCK_MONOTONIC and specified in microseconds. When converting this value in order to pass it as third argument to poll() (which expects relative milliseconds), care should be taken to convert to a relative time and use a division that rounds up to ensure the I/O polling operation doesn't sleep for shorter than necessary, which might result in unintended busy looping (alternatively, use ppoll(2) instead of plain poll(), which understands timeouts with nano-second granularity). That's pretty explicit already, no? (I mean, you have half a point, the first sentence of the explanation might people think this was a relative timeout, but we all read the full documentation, no, before actually using this API, no? ;-)) Anyway, will prep a fix that rewords the first sentence to make this clearer right away. Lennart -- Lennart Poettering, Berlin
Re: [systemd-devel] Please help: timeout waiting for /dev/tty* console device
On Fri, 2023-01-06 at 19:32 -0500, Gabriel L. Somlo wrote: > > I *can* run any tests y'all might suggest to further debug the state > of the system. But at this point I really do believe there is (or > should be :) a way to extend the timeout during initial boot to force > the system to wait for /dev/ttyLXU0 to become available (via udev?). > Have you tried systemd.default_timeout_start_sec= ? Martin
