** Description changed:

  This might instead be a bug in systemd's udev (and eudev?). See the
  claim by Karol Zak in this report I made against util-linux's fsck about
  its locking:
  
  https://github.com/util-linux/util-linux/issues/4477
  
  I disagree, but what do I know?  I'm coming to you because it needs a
  person who can consider how these programs work together as a system.
  I'd report to systemd upstream, but their bug reporting guidelines
  suggest to start with my distro, and you also have a relevant patch in
  play not accepted by upstream (lp2090972-libblkid-fix-spurious-ext-
  superblock-checksum-mismat.patch in util-linux 2.39.3-9ubuntu6.4). In
  passing, I've mentioned the change in locking in util-linux's fsck and
  why I think that is a problem, possibly, to eudev developers on their
  irc channel, but no one present seemed to take an interest.
  
  At my work, we've been seeing mount failures, very intermittently. I
  have a reproducible test case in my local vm where I can cause the mount
  failure within 1000 reboots or so in a certain configuration of Noble
  (with some patches of our own applied, but nothing I think is relevant
  to this). Debugging into it some I've narrowed it down to happen in a
  2nd run of udevd against an add uevent for an ext4 filesystem. The
  initramfs phase run of udevd over this event seems to be okay, adding
  the by-uuid symlink and udev database entry. I've added debugging to
  show processing of the following 60-persistent-storage.rules rule
  returning from in udev_rule_apply_line_to_event() in udev-rules.c
  without assigning a value to ENV{ID_FS_UUID_ENC}.
  
  KERNEL!="sr*|mmcblk[0-9]boot[0-9]", IMPORT{builtin}="blkid"
  
  Then when that worker processes the following rule, it fails the match
  on that variable and never adds the corresponding devlink object to the
  current event's sd_device object's devlink collection.
  
  ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*",
  SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
  
  With that missing, at the end of rule processing in
  udev_event_execute_rules() the call to udev_node_update() calls
  udev_node_update() with a sd_device dev argument that lacks the devlink
  for the by-uuid symlink for the filesystem that will fail to mount.
  udev_node_update() looks for devlinks present in event->dev_db_clone (a
  representation of the udev database before processing this event?) to
  see which are present under dev's devlink collection. It finds it
  missing in dev's collection and calls link_update which removes the
  corresponding /dev/disk/by-uuid symlink on the actual drive. When that
  gets removed the systemd device unit for that symlink gets deactivated
  (state to "inactive (dead)" as shown by systemctl status).  systemd-
  [email protected] has a binds to dependency so also gets deactivated to the
  same inactive (dead) state. The mount unit has a requires/after
  dependency on the systemd-fsck service unit for that uuid (my fstab uses
  uuids to identify filesystems). So that never gets activated, and the
  mount never happens.
  
  I reproduce this by adding the following unit to a VM running Noble
  (with our company's patches):
  
          [Unit]
          Description=Reboot if no mounts fail
          SuccessAction=reboot
          DefaultDependencies=no
          Requires=local-fs.target
          After=local-fs.target
  
          [Install]
          WantedBy=sysinit.target
  
  I then set the grub.cfg timeout value to zero (instead of 10 seconds).
  This way I get the problem within three hours on my QEMU VM, or a little
  over 1000 reboots.
  
  Some other info showing the state of the failure. sc below is an eshell
  alias for systemctl. Log messages not corresponding to messages in your
  source would be from "printf debugging" statements I added myself:
  
  /scp:qvm:/tmp # lsblk -f
  NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% 
MOUNTPOINTS
  sda
  ├─sda1 ext4   1.0         75d03876-a9c1-4506-938a-934acf9d72bf    8.2G    12% 
/
  ├─sda2 ext4   1.0         1cc3b564-f846-4ad4-9c8b-eae6988f8416    5.2G     5% 
/var
  ├─sda3 swap   1           5e7bfad3-00ed-4fb2-b021-70645395daa1                
[SWAP]
  ├─sda4 ext4   1.0         bdc8fdc1-ce85-4d3c-b2be-c2ca96cbf4e7   27.4G     0% 
/cache
  └─sda5 vfat   FAT32       2F9C-D226                             501.4M     0% 
/boot/efi
  sdb
  ├─sdb1 ext4   1.0         738a91e7-dbd7-4b3d-bff8-846272eb91af   22.2G     0% 
/usr/local/acme
  ├─sdb2 ext4   1.0         d08ef5fe-6e72-4035-bd9d-ce85238142bd    2.7G     0% 
/tmp
  └─sdb3 ext4   1.0         4e53b0ae-1b9d-4731-80aa-9eeb03ee9c3f   21.3G     0% 
/cache2
  sdc
  ├─sdc1
  └─sdc2 ext4   1.0         601f92dd-f6f8-411e-8d81-6f3e1f61e530    5.5G     0% 
/usr/local/acme/coredumps
  sdd
  ├─sdd1 ext4   1.0         329989e2-e86d-4114-b8b7-4376f7ea13e5   29.7G     0% 
/cache4
  └─sdd2 ext4   1.0         2eed8f14-506d-42e0-a866-2c08f6e9c462   16.6G     0% 
/usr/local/acme/logs
  sde
  └─sde1 ext4   1.0         211a269b-c0ae-4b5a-a440-ed36cc8c9352   46.4G     0% 
/cache5
  
  /scp:qvm:/tmp # sc list-dependencies --all cache3.mount
  cache3.mount
  ● ├─-.mount
  ● │ └─-.slice
  ○ 
├─dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device
  ● ├─system.slice
  ● │ └─-.slice
  ○ 
└─systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service
  ○   
├─dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device
  ●   ├─system-systemd\x2dfsck.slice
  ●   │ └─system.slice
  ●   │   └─-.slice
  ●   └─systemd-fsckd.socket
  ●     ├─-.mount
  ●     │ └─-.slice
  ●     └─system.slice
  ●       └─-.slice
  
  /scp:qvm:/tmp # sc status -l 
'dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device'
  ○ dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device 
- QEMU_HARDDISK primary
       Loaded: loaded
       Active: inactive (dead) since Tue 2026-06-16 20:46:24 UTC; 3min 53s ago
     Duration: 493ms
  
  Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
  Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
  Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
  Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
  Jun 16 20:50:17 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
  Jun 16 20:50:17 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.device: 
unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
  Notice: journal has been rotated since unit was started, output may be 
incomplete.
  /scp:qvm:/tmp [3] # sc status -l 
'systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service'
  ○ 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service
 - File System Check on /dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb
       Loaded: loaded (/usr/lib/systemd/system/[email protected]; static)
       Active: inactive (dead) since Tue 2026-06-16 20:46:24 UTC; 4min 6s ago
     Duration: 204ms
         Docs: man:[email protected](8)
      Process: 1633 ExecStart=/usr/lib/systemd/systemd-fsck 
/dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb (code=exited, 
status=0/SUCCESS)
     Main PID: 1633 (code=exited, status=0/SUCCESS)
          CPU: 7ms
  
  Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: service_enter_stop_post() final else, calling service_enter_signal()
  Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: SERVICE_FINAL_... recursive call to service_enter_signal()
  Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: service_enter_signal() r == 0  final else not fail label, calling 
service_enter_dead()
  Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 service.c: service_enter_dead(),  s->result == SERVICE_SUCCESS, calling 
unit_log_success()
  Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 Deactivated successfully.
  Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
Stopped 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service
 - File System Check on /dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb.
  Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
  Jun 16 20:49:42 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
  Jun 16 20:50:30 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop_refuse_manual:../src/core/unit.c:6213: can refuse manual
  Jun 16 20:50:30 a192-168-100-2.deploy.acmetechnologies.com systemd[1]: 
systemd-fsck@dev-disk-by\x2duuid-bde994ca\x2d3285\x2d47d8\x2d9780\x2dfdddc29b81eb.service:
 unit_can_stop:../src/core/unit.c:2054: return !!stop() through vtable
  /scp:qvm:/tmp [3] # *cat /etc/fstab
  UUID=75d03876-a9c1-4506-938a-934acf9d72bf     /       ext4    defaults 1 1
  UUID=2F9C-D226        /boot/efi       vfat    noatime 0 0
  UUID=bdc8fdc1-ce85-4d3c-b2be-c2ca96cbf4e7     /cache  ext4    
inode_readahead_blks=0,noatime 1 4
  UUID=4e53b0ae-1b9d-4731-80aa-9eeb03ee9c3f     /cache2 ext4    
inode_readahead_blks=0,noatime 1 4
  UUID=bde994ca-3285-47d8-9780-fdddc29b81eb     /cache3 ext4    
inode_readahead_blks=0,noatime 1 4
  UUID=329989e2-e86d-4114-b8b7-4376f7ea13e5     /cache4 ext4    
inode_readahead_blks=0,noatime 1 4
  UUID=211a269b-c0ae-4b5a-a440-ed36cc8c9352     /cache5 ext4    
inode_readahead_blks=0,noatime 1 4
  proc        /proc             proc     defaults 0 0
  UUID=d08ef5fe-6e72-4035-bd9d-ce85238142bd     /tmp    ext4    defaults 1 1
  UUID=738a91e7-dbd7-4b3d-bff8-846272eb91af     /usr/local/acme ext4    
noatime,inode_readahead_blks=0 1 2
  UUID=601f92dd-f6f8-411e-8d81-6f3e1f61e530     /usr/local/acme/coredumps       
ext4    noatime,inode_readahead_blks=0 1 2
  UUID=2eed8f14-506d-42e0-a866-2c08f6e9c462     /usr/local/acme/logs    ext4    
noatime,inode_readahead_blks=0 1 2
  UUID=1cc3b564-f846-4ad4-9c8b-eae6988f8416     /var    ext4    defaults 1 2
  UUID=5e7bfad3-00ed-4fb2-b021-70645395daa1     swap    swap    sw 0 0
  /scp:qvm:/tmp # *ls -l /dev/disk/by-uuid/
  total 0
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 1cc3b564-f846-4ad4-9c8b-eae6988f8416 
-> ../../sda2
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 211a269b-c0ae-4b5a-a440-ed36cc8c9352 
-> ../../sde1
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 2F9C-D226 -> ../../sda5
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 2eed8f14-506d-42e0-a866-2c08f6e9c462 
-> ../../sdd2
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 329989e2-e86d-4114-b8b7-4376f7ea13e5 
-> ../../sdd1
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 4e53b0ae-1b9d-4731-80aa-9eeb03ee9c3f 
-> ../../sdb3
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 5e7bfad3-00ed-4fb2-b021-70645395daa1 
-> ../../sda3
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 601f92dd-f6f8-411e-8d81-6f3e1f61e530 
-> ../../sdc2
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 738a91e7-dbd7-4b3d-bff8-846272eb91af 
-> ../../sdb1
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 75d03876-a9c1-4506-938a-934acf9d72bf 
-> ../../sda1
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 bdc8fdc1-ce85-4d3c-b2be-c2ca96cbf4e7 
-> ../../sda4
  lrwxrwxrwx 1 root root 10 Jun 16 20:46 d08ef5fe-6e72-4035-bd9d-ce85238142bd 
-> ../../sdb2
  /scp:qvm:/tmp #
  
  journal entries (I've added logging statements of my own):
  
  45 matches in 33 lines for "sdc1" in buffer: 
debug-udev-update_devnode-tue.jctl-b.out.lz
      438:Jun 16 20:46:23 a192-168-100-2.deploy.acmetechnologies.com kernel:  
sdc: sdc1 sdc2
      980:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: The log level is changed to 'debug' while processing 
device (SEQNUM=2470, ACTION=add)
      982:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: /usr/lib/udev/rules.d/60-persistent-storage.rules:80 
Added SYMLINK 'disk/by-id/scsi-0QEMU_QEMU_HARDDISK_3-part1'
      983:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:116 Added SYMLINK 
'disk/by-path/pci-0000:00:03.0-scsi-0:0:2:0-part1'
      984:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:135 Importing properties from 
results of builtin command 'blkid'
      985:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Probe /dev/sdc1 with raid and offset=0
     1001:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
systemd-fsck[1700]: /dev/sdc1: clean, 11/2883584 files, 259873/11533824 blocks
     1004:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:150 Added SYMLINK 
'disk/by-partuuid/437f26c8-e793-4f9d-9d3a-94117b15a8f5'
     1005:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:151 Added SYMLINK 
'disk/by-partlabel/primary'
     1006:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
/usr/lib/udev/rules.d/60-persistent-storage.rules:156 Added SYMLINK 
'disk/by-diskseq/27-part1'
     1007:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: /usr/lib/udev/rules.d/69-vdev.rules:6 Importing 
properties from results of '/lib/udev/vdev_id -d sdc1'
     1008:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Starting '/lib/udev/vdev_id -d sdc1'
     1083:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: '/lib/udev/vdev_id -d sdc1'(out) 'Error: Config file 
"/etc/zfs/vdev_id.conf" not found'
     1085:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Process '/lib/udev/vdev_id -d sdc1' failed with exit 
code 1.
     1086:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: /usr/lib/udev/rules.d/69-vdev.rules:6 Command 
"/lib/udev/vdev_id -d sdc1" returned 1 (error), ignoring
     1087:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Setting permissions /dev/sdc1, uid=0, gid=6, 
mode=0660
     1088:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Removing/updating old device symlink 
'/dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb', which is no longer 
belonging to this device.
     1089:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Links that do belong to this device...
     1090:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
------------------------------------------------------------------------
     1091:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 0: 
/dev/disk/by-partuuid/437f26c8-e793-4f9d-9d3a-94117b15a8f5
     1092:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 1: /dev/disk/by-diskseq/27-part1
     1093:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 2: 
/dev/disk/by-path/pci-0000:00:03.0-scsi-0:0:2:0-part1
     1094:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 3: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_3-part1
     1095:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 4: /dev/disk/by-partlabel/primary
     1096:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: 
------------------------------------------------------------------------
     1097:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: No reference left for 
'/dev/disk/by-uuid/bde994ca-3285-47d8-9780-fdddc29b81eb', removing
     1098:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Successfully created symlink 
'/dev/disk/by-diskseq/27-part1' to '/dev/sdc1'
     1099:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Successfully created symlink 
'/dev/disk/by-partlabel/primary' to '/dev/sdc1'
     1100:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Successfully created symlink '/dev/block/8:33' to 
'/dev/sdc1'
     1101:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: sd-device: Created db file '/run/udev/data/b8:33' 
for 
'/devices/pci0000:00/0000:00:03.0/virtio0/host2/target2:0:2/2:0:2:0/block/sdc/sdc1'
     1102:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Adding watch on '/dev/sdc1'
     1103:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: Device processed (SEQNUM=2470, ACTION=add)
     1104:Jun 16 20:46:24 a192-168-100-2.deploy.acmetechnologies.com 
(udev-worker)[1601]: sdc1: sd-device-monitor(worker): Passed 1268 byte to 
netlink monitor.
  
  I also see how fsck (or the program it forks) has changed the ext4 fs
  superblock and has a last written timestamp with the same second as the
  journal entry where the builtin blkid (libblkid) rule processing starts.
  
  Thinking that the problem comes from udevd and util-linux fsck since
  2014 no longer both using /dev/sdX as the  file for their uses of flock,
  I tried adding code to udevd to take a shared lock on
  /run/fsck/sdX.lock, the lockfile fsck now uses. When I used a blocking
  call (with some narrowing by udev rule line number to avoid the bug
  going away simply by perturbing the timing) the problem went away. When
  I changed that to a non-blocking call, logged the errno, and allowed
  udevd to proceed normally, I got the problem back with errno after flock
  being 11 (EAGAIN or EWOULDBLOCK), which convinced me that fsck was
  running and holding an exclusive lock when the problem happened. With
  those patches reverted, I instead reverted fsck to its pre-2014 locking
  behaviour, where it took its exclusive lock on /dev/sdX again. That also
  prevented the problem from happening. "Preventing" here means running
  for 7+ hours without seeing it, or roughly more than 3000 boots.
  
  Note that I am running with lp2090972-libblkid-fix-spurious-ext-
  superblock-checksum-mismat.patch from ubuntu bug report
  https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2090972 but
  that does not avoid this problem.
  
  Here's the patch I revert from Karol Zak:
  
https://github.com/util-linux/util-linux/commit/3bbdae633f4a1dda5f95ee6c61f18a1c8ef12250
- But he doesn't view his lock as one to take the other side of udevd's read 
lock only to protect fsck from other instances of itself (see my util-linux bug 
report above). Still, I was thinking I'd suggest reverting that a local patch 
if my bosses like that idea. Yet, I'm worried about the systemd bug report 
leading up to Karol Zak's change:
+ But he doesn't view his lock as one to take the other side of udevd's read 
lock only to protect fsck from other instances of itself (see my util-linux bug 
report above). Still, I was thinking I'd suggest reverting that in a local 
company patch if my bosses like that idea. Yet, I'm worried about the systemd 
bug report leading up to Karol Zak's change:
  
  https://bugs.freedesktop.org/show_bug.cgi?id=79576
  
- Is there some other solution? Do you acknowledge this as a real bug?
+ Is there some other solution? Do you acknowledge that this as a real
+ bug?
  
  Version info:
  
  /scp:qvm:~ # *cat /etc/os-release
  PRETTY_NAME="Ubuntu 24.04.3 LTS"
  NAME="Ubuntu"
  VERSION_ID="24.04"
  VERSION="24.04.3 LTS (Noble Numbat)"
  VERSION_CODENAME=noble
  ID=ubuntu
  ID_LIKE=debian
  HOME_URL="https://www.ubuntu.com/";
  SUPPORT_URL="https://help.ubuntu.com/";
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/";
  
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy";
  UBUNTU_CODENAME=noble
  LOGO=ubuntu-logo
  
  /scp:qvm:~ # dpkg -s systemd | grep ^Ver
  (standard input):9:Version: 255.4-1ubuntu8.12acme24.0.2
  
  /scp:qvm:~ # dpkg -s util-linux | grep ^Ver
  (standard input):10:Version: 2.39.3-9ubuntu6.4acme24.0.1
  
  Expected to happen: all auto mounts happen every boot
  What happened: a mount, for me a non-root, non-boot one but a varying one, 
failed every 1000 or so boots.

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2160558

Title:
  udevd and fsck contend causing the prior to miss fs uuids

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/2160558/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to