[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
@Łukasz: Thanks for the fix. I've tried the packages systemd (and other packages from the source ball) in version "252.5-2ubuntu3.1". And the bug went away. Great. Thank you and the other hard-working people. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd: Fix Released Status in systemd package in Ubuntu: Fix Released Status in systemd source package in Jammy: Fix Committed Status in systemd source package in Lunar: Fix Committed Bug description: [Impact] Depending on the contents of /proc/cmdline, when systemd is re- executed with systemctl daemon-reexec, the --deserialize flag may be ignored because it was added after the other arguments. For example, if /proc/cmdline contains ---, then the re-exec cmdline might look like: $ cat /proc/1/cmdline | tr '\0' '\n' /lib/systemd/systemd --- splash --system --deserialize 54 This causes systemd not to process the --deserialize 54 argument, causing it to start with a fresh state. This can cause all kinds of problems, and one easy symptom to see is many lines in the journal like: "$service.service: Found left-over process $pid ($service) in control group while starting unit. Ignoring." [Test Plan] 1. (Only needed if your test system is not already affected) Edit the kernel command line to contain '---' at the end, which would trigger the bug. This can be done by appending '---' to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub, running update-grub, and then rebooting. 2. After enabling -proposed, install systemd: $ apt install systemd -y 3. Check that the systemd.postinst script skipped the daemon-reexec call, and instead indicated a reboot is required: $ grep -Fsx systemd /run/reboot-required.pkgs systemd 4. Reboot. 5. Try to re-exec systemd, and check that there are not tons of "left- over process" log messages: $ systemctl daemon-reexec $ journalctl --grep "Found left-over process" -b 0 6. Also confirm that the ordering of /proc/1/cmdline is correct, i.e. that --deserialize $fd comes before args from /proc/cmdline: $ cat /proc/1/cmdline | tr '\0' '\n' [Where problems could occur] There are two changes for this bug. First is the patch against systemd itself, which changes the ordering of arguments on the systemd commandline. This change simply makes it so that systemd's own arguments are always put first on it's re-exec commandline, and that anything from /proc/cmdline is appended after. Any regressions caused by this would also be seen in systemctl daemon-reexec invocations. The second change is in systemd.postinst, which skips the systemctl daemon-reexec call when upgrading from versions of systemd that could hit this bug. Regressions caused by this would be seen during package upgrades. [Original Description] # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usu
[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
Nike - Unfortunately, I don't have the resources to put a new image in our environment. But I've updated systemd (and all related) libraries to `252.5-2ubuntu3` and it's still "broken": `systemctl daemon-reexec` forgets all the old processes and restarts everything. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Jammy: Incomplete Bug description: # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found left-over process 164 (cron) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Regular background program processing daemon. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: dbus.service: Found left-over process 177 (dbus-daemon) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a
[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
@enr0n: Thanks for your help. I've deleted my old comment because I messed up the output of the container. Here are the outputs of your test script. Output of the host: hostsystem ~ # bash /tmp/test.sh cgroupsv1 Output of the container: container ~ $ bash /tmp/test.sh hybrid So, there is a mismatch of the cgroups versions. What does that mean? I've attached the output of the `journalctl -t systemd -b 0`. Do you need an additional output after I've executed the command `systemctl daemon-reexec`? Note: Executing the test scripts on a working(!) container with Ubuntu 20.04 and systemd 245.4-4ubuntu3.20 outputs the following: hostsystem: cgroupsv1 container: hybrid ** Attachment added: "Output of the container with `journalctl -t systemd -b 0` and LogLevel=debug" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2013543/+attachment/5665411/+files/logs.txt.gz -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Jammy: Incomplete Bug description: # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31
[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
@enr0n: Thanks for your help. Here are the outputs of your test script. Output of the host: hostsystem ~ # bash /tmp/test.sh cgroupsv1 Output of the container: container ~ $ bash /tmp/test.sh cgroupsv2 So, there is a mismatch of the cgroups versions. What does that mean? I've attached the output of the `journalctl -t systemd -b 0`. Do you need an additional output after I've executed the command `systemctl daemon-reexec`? Note: Executing the test scripts on a container with Ubuntu 20.04 and systemd 245.4-4ubuntu3.20 output the following: hostsystem: cgroupsv1 container: hybrid ** Attachment added: "Output of the container with `journalctl -t systemd -b 0` and LogLevel=debug" https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2013543/+attachment/5665158/+files/logs.txt.gz -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Jammy: Incomplete Bug description: # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found left-over process 164 (cron)
[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
Appendix to comment #4: The configuration Ubuntu 20.04 and systemd 245.4-4ubuntu3.20 is working fine. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Jammy: Incomplete Bug description: # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found left-over process 164 (cron) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Regular background program processing daemon. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: dbus.service: Found left-over process 177 (dbus-daemon) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started D-Bus System Message Bus. And... Mar 31 12:51:39 FQD
[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
Clarification: Yes, we've isolated the behavior down to the call of `system daemon-reexec` from the `/var/lib/dpkg/info/systemd.postinst` file. But every call of `system daemon-reexec` will trigger this "bug", if you have installed the packages in version `249.11-0ubuntu3.4`. The host machine and the container are running the same kernel. But you can fake the kernel version in the container. So, the host machine reports the real version of the kernel: > Linux FQDN_REDACTED 3.10.0-1127.18.2.vz7.163.46 #1 SMP Fri Nov 20 21:47:55 MSK 2020 x86_64 x86_64 x86_64 GNU/Linux And the container reports this: > Linux FQDN_REDACTED 5.4.0 #1 SMP Thu Apr 22 16:18:59 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux This was implemented because some applications in the container are checking for a newer kernel version, so you have to fake it. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd package in Ubuntu: Incomplete Status in systemd source package in Jammy: Incomplete Bug description: # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Fo
[Touch-packages] [Bug 2013543] Re: systemctl daemon-reexec forgets running services and starts everything new
** Package changed: sagemath (Ubuntu) => systemd (Ubuntu) -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to systemd in Ubuntu. https://bugs.launchpad.net/bugs/2013543 Title: systemctl daemon-reexec forgets running services and starts everything new Status in systemd package in Ubuntu: New Bug description: # Our problem # During a regular update of our container environment, `systemd` (and the related packages libpam-systemd, libsystemd0, libudev1, systemd- sysv and udev) were updated from `249.11-0ubuntu3.6` to `249.11-0ubuntu3.7`. We're talking only about Ubuntu 22.04. Our Ubuntu 20.04 is working fine with `systemctl daemon-reexec`. In my opinion, the update was not the problem because we've tried downgrading and tried these versions: (current) `249.11-0ubuntu3.7`, `249.11-0ubuntu3.6`, `249.11-0ubuntu3.4` and `249.11-0ubuntu3.3`. The symptoms were the same. # Symptoms # The `/var/lib/dpkg/info/systemd.postinst` executes a `systemctl daemon-reexec` and that ended in a disaster. It seems that `systemd` is forgetting all it started children and tries to start nearly every configured service again. Naturally, the old services are still running, and the ports can't be opened twice and `systemd` won't give up. Here are some(!) of the logfiles: Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Create Volatile Files and Directories... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 130 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31475 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: systemd-udevd.service: Found left-over process 31476 (systemd-udevd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target System Initialization. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt download activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily apt upgrade and clean activities. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily dpkg database backup timer. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Periodic ext4 Online Metadata Check for All Filesystems. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Condition check resulted in Discard unused blocks once a week being skipped. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily rotation of log files. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily man-db regeneration. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Message of the Day. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Clean PHP session files every 30 mins. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Update the plocate database daily. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Daily Cleanup of Temporary Directories. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Basic System. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: System is tainted: cgroupsv1 Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Reached target Timer Units. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: atd.service: Found left-over process 206 (atd) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Starting Deferred execution scheduler... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: cron.service: Found left-over process 164 (cron) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started Regular background program processing daemon. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: dbus.service: Found left-over process 177 (dbus-daemon) in control group while starting unit. Ignoring. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies. Mar 31 12:51:39 FQDN_REDACTED systemd[1]: Started D-Bus System Message Bus. And... Mar 31 12:51:39 FQDN_REDACTED systemd[1]: rsyslog.service: Found left-over process 204 (rsyslogd) in control group while star
