Public bug reported: [Summary] On Ubuntu Server 24.04.5, after an unclean shutdown (power loss), roughly 30-60% of boots end with one or more ext4 filesystems from /etc/fstab not mounted (/home, /boot or a data partition), local-fs.target never reached, and no "Dependency failed" or timeout in the journal. systemd-fsck@<uuid>.service is "Stopped" (or killed with SIGTERM) instead of "Finished", and the dependent mount jobs are cancelled and never retried.
The trigger is the udev blkid builtin failing with a hard error while e2fsck is running. I believe the hard error comes from the O_DIRECT re-read added by the fix for LP #2090972 (lp2090972-libblkid-fix-spurious-ext-superblock-checksum-mismat.patch), which in 2.39.3 reads into a buffer that is not sector aligned. [System] Ubuntu 24.04.5 LTS (Server), kernel 6.8.0-149-generic util-linux / libblkid1 2.39.3-9ubuntu6.6 udev / systemd 255.4-1ubuntu8.18 e2fsprogs 1.47.0-2.4~exp1ubuntu4.1 SATA HDD (WDC WD5000LPLX), 512 logical / 4096 physical, /sys/block/sda/queue/dma_alignment = 511 GPT, sda1 vfat /boot/efi, sda2 ext4 /boot, sda3 ext4 /, sda4 ext4 /home, sda5 swap, sda6 ext4 data; fstab written by the installer: /dev/disk/by-uuid/... ext4 defaults 0 1 for all of them. Seen on 4 identical machines. [What happens] (udev_log=debug, systemd LogLevel=debug, monotonic time) 9.664 systemd-fsck: /dev/sda4: recovering journal 9.815 systemd-udevd: sda4: Received inotify event for /dev/sda4. 9.815 systemd-udevd: sda4: device is closed, synthesising 'change' on .../block/sda/sda4 9.830 (udev-worker): sda4: 60-persistent-storage.rules:133 Importing properties from results of builtin command 'blkid' 9.830 (udev-worker): sda4: Probe /dev/sda4 with raid and offset=0 9.946 systemd-fsck: /dev/sda4: clean, 144/1048576 files, 113080/4194304 blocks 9.947 (udev-worker): sda4: Failed to probe superblocks: Operation not permitted 9.947 (udev-worker): sda4: 60-persistent-storage.rules:133 Failed to run builtin 'blkid': Operation not permitted 9.950 (udev-worker): sda4: Removing/updating old device symlink '/dev/disk/by-partuuid/<PARTUUID>', which is no longer belonging to this device. 9.950 (udev-worker): sda4: Removing/updating old device symlink '/dev/disk/by-uuid/<UUID>', which is no longer belonging to this device. 9.953 systemd: dev-disk-by\x2dpartuuid-<PARTUUID>.device: Changed plugged -> dead 9.953 systemd: dev-disk-by\x2duuid-<UUID>.device: Changed plugged -> dead 9.953 systemd: systemd-fsck@dev-disk-by\x2duuid-<UUID>.service: Trying to enqueue job .../stop/replace 9.954 systemd: ...: Job [email protected]/start finished, result=canceled 10.033 systemd-udevd: sda4: Received inotify event for /dev/sda4. (second change event; the symlinks come back) 10.074 systemd: Stopped systemd-fsck@dev-disk-by\x2duuid-<UUID>.service home.mount and everything ordered after it stays cancelled until the next reboot. So: 1. e2fsck closes and reopens the device after journal replay (e2fsck/unix.c: ext2fs_close_free(); goto restart), which makes udev synthesise a 'change' event in the middle of the fsck run. 2. The blkid builtin then probes the partition while e2fsck is still writing. 3. The probe does not return "nothing found", it returns an error, so ALL blkid-derived properties are lost: ID_FS_UUID and also ID_PART_ENTRY_UUID. That is why by-partuuid disappears too, unlike in systemd issue #43695 where only by-uuid goes away. 4. [email protected] has BindsTo= on the by-uuid device unit, so it is stopped and the mount jobs are cancelled. [Why I think the probe error is the LP #2090972 patch] "Operation not permitted" is just blkid_do_safeprobe() returning -1, which udev-builtin-blkid.c passes to log_device_debug_errno(). In ext.c, after a checksum mismatch the superblock is re-read with blkid_probe_get_sb_buffer_direct(); if that read fails, ext_get_super() returns NULL with errno set and the callers do "return errno ? -errno : 1", which aborts the whole probe. In 2.39.3 the buffer for that read is bf = calloc(1, sizeof(struct blkid_bufinfo) + len); bf->data = ((unsigned char *) bf) + sizeof(struct blkid_bufinfo); i.e. the data pointer is offset by the header size from the malloc pointer and is not sector aligned (upstream at the time of commit 483c9f38 already used page-aligned buffers). On a device with dma_alignment 511 an O_DIRECT read into such a buffer fails with EINVAL. Verified on the affected machine with a small read-only test (O_DIRECT pread of 1024 bytes at offset 1024 of /dev/sda4): buffer aligned to 4096: OK same buffer shifted by 40 bytes: FAILED errno=22 (Invalid argument) same buffer shifted by 512: OK I did not capture the failing read inside libblkid itself (no LIBBLKID_DEBUG trace of the failing probe), so this last step is from reading the source. [How to reproduce] Machine as above with several ext4 filesystems in fstab with a non-zero fsck pass number. Hard reset it while the filesystems are mounted read-write (power cut, or "echo b > /proc/sysrq-trigger" after sync). On the next boot e2fsck replays the journals and one of the systemd-fsck@ units is stopped. Results: power cut 6 failures in 13 boots (4 machines); sysrq reset 10 failures in 21 boots (2 machines). Clean reboots never fail. [Workaround] Run systemd-fsck under the udev block device lock, so that udev requeues the event instead of probing during fsck: /etc/systemd/system/[email protected]/10-udev-lock.conf [Service] ExecStart= ExecStart=/usr/bin/udevadm lock --device=%f /usr/lib/systemd/systemd-fsck %f SyslogIdentifier=systemd-fsck With this: 0 failures in 60 sysrq resets and 6 power cuts on the same two machines; the udev log shows "Block device is currently locked, requeueing the event." in place of the failed probe. [Related] LP #2090972 (the fix that introduced the O_DIRECT re-read) https://github.com/systemd/systemd/issues/43695 (same race, e2fsck vs udev probe) https://github.com/systemd/systemd/pull/43696 (draft: lock the whole block device while fsck runs) ** Affects: util-linux (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/2174478 Title: libblkid: O_DIRECT superblock re-read from LP #2090972 fails with EINVAL in noble; after unclean shutdown udev drops by-uuid/by-partuuid during e2fsck journal replay and mounts are cancelled To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2174478/+subscriptions -- ubuntu-bugs mailing list [email protected] https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
