Public bug reported:
== Description ==
The initramfs udevd is sometimes still running after the switch to the
real root filesystem. It then aborts on an assertion in chase(), leaving a
core dump behind on every affected boot.
The udev initramfs hook (debian/extra/initramfs-tools/scripts/init-bottom/udev)
in noble only does:
udevadm control --exit
mount -n -o move /dev "${rootmnt}/dev" ...
nuke /dev (or rm -rf /dev)
"udevadm control --exit" does not wait for udevd to terminate. If udevd
still has queued events, it keeps running while the initramfs is torn down.
It is still running after run-init switches to the real root, alongside the
systemd-udevd.service instance started by systemd.
Debian fixed this hook in systemd 256~rc3-3 (Debian bug #1056564,
salsa commit f2590632 "initramfs: call udevadm settle before control --exit
to ensure workers have stopped"):
udevadm settle --timeout=60
udevadm control --exit
noble (255.4-1ubuntu8.17) does not have this change.
Release: Ubuntu 24.04.4 LTS (noble)
Package: systemd / udev 255.4-1ubuntu8.17 (amd64)
Kernel: 6.8.0-138-generic
== Steps to reproduce ==
The problem is random.
Start a VM with Ubuntu-24.04 and reboot it several times until a coredump is
generated.
1. Boot Ubuntu 24.04 with the default initramfs-tools initramfs (udev hook
unmodified).
2. After boot, check "coredumpctl list".
3. reboot with 'reboot -f' until a coredump happens
== Expected result ==
The initramfs udevd has exited before the switch to the real root. Only
systemd-udevd.service runs afterwards and nothing crashes.
== Actual result ==
The initramfs udevd (started by initramfs-tools: "--daemon
--resolve-names=never", cgroup /init.scope) is still running after the
switch. It runs alongside systemd-udevd.service and aborts with SIGABRT:
Assertion 'path_is_absolute(p)' failed at src/basic/chase.c:648,
function chase(). Aborting.
Journal of the affected boot:
10:37:36 systemd[1]: Starting systemd-udev-trigger.service - Coldplug All
udev Devices...
10:37:36 systemd[1]: Finished systemd-udev-trigger.service - Coldplug All
udev Devices.
10:37:36 systemd[1]: Starting systemd-udevd.service - Rule-based Manager for
Device Events and Files...
10:37:36 systemd-udevd[437]: Using default interface naming scheme 'v255'.
10:37:36 systemd[1]: Started systemd-udevd.service - Rule-based Manager for
Device Events and Files.
10:37:37 systemd-coredump[526]: Process 156 (systemd-udevd) of user 0 dumped
core.
coredumpctl info:
PID: 156 (systemd-udevd)
Command Line: /usr/lib/systemd/systemd-udevd --daemon --resolve-names=never
Executable: /usr/bin/udevadm
Control Group: /init.scope
Signal: 6 (ABRT)
build-id: 83d1596822cd6fdd8bda1fc3d2501add8f16e141
When the core was opened, gdb reported every file mapping of the process as
"(deleted)": /usr/bin/udevadm, libc, hwdb.bin, modules.*.bin and others.
These are the initramfs copies, removed by run-init.
The same crash, with the same backtrace and the same paths, was seen on a
second machine (initramfs udevd PID 239 aborted 2 s after
systemd-udevd.service PID 487 started).
== Backtrace (udev-dbgsym 255.4-1ubuntu8.17) ==
#5 log_assert_failed () at ../src/basic/log.c:967
#6 chase (path=<optimized out>, root=0x5622716a3194 "/", flags=<optimized out>,
ret_path=0x7ffe92f19400, ret_fd=0x7ffe92f193f4) at
../src/basic/chase.c:648
root_abs = 0x0
absolute = "/usr/lib/udev/rules.d"
p = "usr/lib/udev/rules.d"
fd = 44
pfd = 45
r = 1
#7 chase_and_opendir (chase_flags=CHASE_PREFIX_ROOT, root=0x0, ...) at
../src/basic/chase.c:810
#8 conf_files_list_strv (suffix=".rules", root=0x0, dirs=..., flags=0) at
../src/basic/conf-files.c:166
#9 config_get_stats_by_path (suffix=".rules", check_dropins=false, root=0x0)
at ../src/shared/conf-parser.c:747
#10 udev_rules_should_reload (rules=0x5622924272e0) at
../src/udev/udev-rules.c:1685
#11 manager_reload (manager=..., force=false) at ../src/udev/udev-manager.c:271
now_usec = 4157749
#12 event_queue_start (manager=...) at ../src/udev/udev-manager.c:579
#13 on_post (s=..., userdata=...) at ../src/udev/udev-manager.c:1147
#14 source_dispatch () at ../src/libsystemd/sd-event/sd-event.c:4227
#15 sd_event_dispatch () at ../src/libsystemd/sd-event/sd-event.c:4808
#16 sd_event_run () at ../src/libsystemd/sd-event/sd-event.c:4869
#17 sd_event_loop () at ../src/libsystemd/sd-event/sd-event.c:4891
#18 manager_main () at ../src/udev/udev-manager.c:1346
#19 run_udevd () at ../src/udev/udevd.c:407
#20 run () at ../src/udev/udevadm.c:124
#21 main () at ../src/udev/udevadm.c:140
(full "thread apply all bt full" attached)
== Analysis ==
The crash happens when udevd's event queue becomes idle: on_post ->
event_queue_start -> manager_reload(force=false) -> udev_rules_should_reload()
-> chase("/usr/lib/udev/rules.d", root="/", CHASE_PREFIX_ROOT).
chase() opens "/" and calls chaseat(fd, "usr/lib/udev/rules.d", ...).
chaseat() returns an absolute path only if chaseat_needs_absolute() ->
dir_fd_is_root(fd) is true. dir_fd_is_root() compares the inode and mount ID
of "." and "..". chase() assumes that the result is always absolute when
root is "/" (comment at chase.c:640-646) and asserts on it at line 648.
Here chaseat() returned a relative path, so dir_fd_is_root() returned false
for the process's own "/". This fits a process whose root is still the old
initramfs rootfs after run-init has moved the real root over "/". That last
step is inferred and has not been verified.
== Proposed fix ==
1. Backport Debian commit f2590632 (#1056564) to noble: run
"udevadm settle --timeout=60" before "udevadm control --exit" in
init-bottom/udev. This removes the event backlog that keeps the
initramfs udevd alive.
2. Optionally, wait in init-bottom/udev until udevd has actually exited
(bounded, with a kill fallback) before tearing down /dev. settle +
"control --exit" still leaves a small window if new uevents arrive.
3. Upstream robustness (separate, lower priority): chase() could handle a
relative chaseat() result when root is "/" instead of asserting.
== References ==
- Debian bug #1056564 "udev: run-init stops while nuking initramfs contents",
fixed in systemd 256~rc3-3: https://bugs.debian.org/1056564
- Debian commit: https://salsa.debian.org/systemd-team/systemd/-/commit/f2590632
== Attachments ==
- crash3 (coredumpctl info + gdb "thread apply all bt full")
- coredump:
core.systemd-udevd.0.8933582c82164652916d2961fb4c2a6e.156.1792139856000000.zst
** Affects: systemd (Ubuntu)
Importance: Undecided
Status: New
** Tags: noble
** Attachment added: "coredumpctl info"
https://bugs.launchpad.net/bugs/2172311/+attachment/6013010/+files/crash3
** Tags added: noble
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2172311
Title:
[noble] initramfs udevd survives switch to real root and crashes:
"Assertion 'path_is_absolute(p)' failed at src/basic/chase.c:648" —
please backport Debian fix for #1056564
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2172311/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
