Public bug reported:

[Summary]
On Ubuntu Server 24.04.5, after an unclean shutdown (power loss), roughly 
30-60% of
boots end with one or more ext4 filesystems from /etc/fstab not mounted
(/home, /boot or a data partition), local-fs.target never reached, and no
"Dependency failed" or timeout in the journal. systemd-fsck@<uuid>.service is
"Stopped" (or killed with SIGTERM) instead of "Finished", and the dependent
mount jobs are cancelled and never retried.

The trigger is the udev blkid builtin failing with a hard error while e2fsck
is running. I believe the hard error comes from the O_DIRECT re-read added by
the fix for LP #2090972 
(lp2090972-libblkid-fix-spurious-ext-superblock-checksum-mismat.patch),
which in 2.39.3 reads into a buffer that is not sector aligned.

[System]
Ubuntu 24.04.5 LTS (Server), kernel 6.8.0-149-generic
util-linux / libblkid1 2.39.3-9ubuntu6.6
udev / systemd 255.4-1ubuntu8.18
e2fsprogs 1.47.0-2.4~exp1ubuntu4.1
SATA HDD (WDC WD5000LPLX), 512 logical / 4096 physical,
/sys/block/sda/queue/dma_alignment = 511
GPT, sda1 vfat /boot/efi, sda2 ext4 /boot, sda3 ext4 /, sda4 ext4 /home,
sda5 swap, sda6 ext4 data; fstab written by the installer:
/dev/disk/by-uuid/... ext4 defaults 0 1 for all of them.
Seen on 4 identical machines.

[What happens] (udev_log=debug, systemd LogLevel=debug, monotonic time)
  9.664 systemd-fsck: /dev/sda4: recovering journal
  9.815 systemd-udevd: sda4: Received inotify event for /dev/sda4.
  9.815 systemd-udevd: sda4: device is closed, synthesising 'change' on 
.../block/sda/sda4
  9.830 (udev-worker): sda4: 60-persistent-storage.rules:133 Importing 
properties from results of builtin command 'blkid'
  9.830 (udev-worker): sda4: Probe /dev/sda4 with raid and offset=0
  9.946 systemd-fsck: /dev/sda4: clean, 144/1048576 files, 113080/4194304 blocks
  9.947 (udev-worker): sda4: Failed to probe superblocks: Operation not 
permitted
  9.947 (udev-worker): sda4: 60-persistent-storage.rules:133 Failed to run 
builtin 'blkid': Operation not permitted
  9.950 (udev-worker): sda4: Removing/updating old device symlink 
'/dev/disk/by-partuuid/<PARTUUID>', which is no longer belonging to this device.
  9.950 (udev-worker): sda4: Removing/updating old device symlink 
'/dev/disk/by-uuid/<UUID>', which is no longer belonging to this device.
  9.953 systemd: dev-disk-by\x2dpartuuid-<PARTUUID>.device: Changed plugged -> 
dead
  9.953 systemd: dev-disk-by\x2duuid-<UUID>.device: Changed plugged -> dead
  9.953 systemd: systemd-fsck@dev-disk-by\x2duuid-<UUID>.service: Trying to 
enqueue job .../stop/replace
  9.954 systemd: ...: Job [email protected]/start finished, 
result=canceled
 10.033 systemd-udevd: sda4: Received inotify event for /dev/sda4.   (second 
change event; the symlinks come back)
 10.074 systemd: Stopped systemd-fsck@dev-disk-by\x2duuid-<UUID>.service
home.mount and everything ordered after it stays cancelled until the next 
reboot.

So:
1. e2fsck closes and reopens the device after journal replay (e2fsck/unix.c:
   ext2fs_close_free(); goto restart), which makes udev synthesise a 'change'
   event in the middle of the fsck run.
2. The blkid builtin then probes the partition while e2fsck is still writing.
3. The probe does not return "nothing found", it returns an error, so ALL
   blkid-derived properties are lost: ID_FS_UUID and also ID_PART_ENTRY_UUID.
   That is why by-partuuid disappears too, unlike in systemd issue #43695
   where only by-uuid goes away.
4. [email protected] has BindsTo= on the by-uuid device unit, so it is
   stopped and the mount jobs are cancelled.

[Why I think the probe error is the LP #2090972 patch]
"Operation not permitted" is just blkid_do_safeprobe() returning -1, which
udev-builtin-blkid.c passes to log_device_debug_errno().
In ext.c, after a checksum mismatch the superblock is re-read with
blkid_probe_get_sb_buffer_direct(); if that read fails, ext_get_super()
returns NULL with errno set and the callers do "return errno ? -errno : 1",
which aborts the whole probe.
In 2.39.3 the buffer for that read is
  bf = calloc(1, sizeof(struct blkid_bufinfo) + len);
  bf->data = ((unsigned char *) bf) + sizeof(struct blkid_bufinfo);
i.e. the data pointer is offset by the header size from the malloc pointer and
is not sector aligned (upstream at the time of commit 483c9f38 already used
page-aligned buffers). On a device with dma_alignment 511 an O_DIRECT read
into such a buffer fails with EINVAL. Verified on the affected machine with a
small read-only test (O_DIRECT pread of 1024 bytes at offset 1024 of /dev/sda4):
  buffer aligned to 4096:        OK
  same buffer shifted by 40 bytes: FAILED errno=22 (Invalid argument)
  same buffer shifted by 512:    OK
I did not capture the failing read inside libblkid itself (no LIBBLKID_DEBUG
trace of the failing probe), so this last step is from reading the source.

[How to reproduce]
Machine as above with several ext4 filesystems in fstab with a non-zero fsck
pass number. Hard reset it while the filesystems are mounted read-write
(power cut, or "echo b > /proc/sysrq-trigger" after sync). On the next boot
e2fsck replays the journals and one of the systemd-fsck@ units is stopped.
Results: power cut 6 failures in 13 boots (4 machines); sysrq reset
10 failures in 21 boots (2 machines). Clean reboots never fail.

[Workaround]
Run systemd-fsck under the udev block device lock, so that udev requeues the
event instead of probing during fsck:
  /etc/systemd/system/[email protected]/10-udev-lock.conf
  [Service]
  ExecStart=
  ExecStart=/usr/bin/udevadm lock --device=%f /usr/lib/systemd/systemd-fsck %f
  SyslogIdentifier=systemd-fsck
With this: 0 failures in 60 sysrq resets and 6 power cuts on the same two
machines; the udev log shows "Block device is currently locked, requeueing
the event." in place of the failed probe.

[Related]
LP #2090972 (the fix that introduced the O_DIRECT re-read)
https://github.com/systemd/systemd/issues/43695 (same race, e2fsck vs udev 
probe)
https://github.com/systemd/systemd/pull/43696 (draft: lock the whole block 
device while fsck runs)

** Affects: util-linux (Ubuntu)
     Importance: Undecided
         Status: New

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2174478

Title:
  libblkid: O_DIRECT superblock re-read from LP #2090972 fails with
  EINVAL in noble; after unclean shutdown udev drops by-uuid/by-partuuid
  during e2fsck journal replay and mounts are cancelled

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2174478/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to