Public bug reported:

Regression from the SRU for LP: #2100252.

Summary

Since 249.11-0ubuntu3.17, systemd can stop a mounted fstab entry when a
different block device carrying the same filesystem UUID goes away. Two
devices with one filesystem UUID are common on cloud and storage hosts:
a loop or device-mapper device stacked over a copy of the filesystem, a
cloned or restored volume, or a mirror leg.

What happens:

1. A second device with the same UUID appears.
2. udev points /dev/disk/by-uuid/<UUID> at it, and PID1 moves the 
dev-disk-by\x2duuid-<UUID>.device unit onto it.
3. When that device is removed, udev points the symlink back at the original 
disk.
4. PID1 marks the by-uuid device unit dead, though the original disk still has 
the link.
5. The unit going dead propagates a stop to the fstab-generated mount unit, so 
the filesystem is unmounted while the disk is still present and healthy.
6. With "nofail", nothing mounts it again.

The jammy release build (249.11-0ubuntu3) does not do this on the same
host. Neither does systemd 245 on 20.04 or systemd 255 on 24.04. Run the
same way, the unit moves to the second device and falls back to the
original disk on removal.

Impact

A filesystem the admin mounted through fstab disappears from its
mountpoint with no error from the disk. Applications then write into the
empty directory on the root filesystem, or fail. It needs a manual
"mount" (or a reboot) to recover.

A mount made with plain "mount" (not through an fstab unit) is not
affected. Only mounts that systemd binds to the by-uuid device unit are
stopped.

Steps to reproduce

Needs root and a spare, empty block device that the script reformats
(here a 2 GiB EBS volume on an EC2 instance, /dev/nvme1n1). The script
is attached as repro.sh:

  sudo bash repro.sh /dev/nvme1n1 20

It formats the disk and adds "UUID=<U> /data20 ext4 defaults,nofail 0 2"
to /etc/fstab. It starts data20.mount and copies the first 64 MiB of the
disk to an image file. Then it repeats:

1. Attach the image as a loop device, which has the same filesystem UUID, and 
run "udevadm settle".
2. Record where the by-uuid symlink points and which device the systemd unit 
follows.
3. Detach the loop device, run "udevadm settle", and record the by-uuid unit's 
and data20.mount's states.

Expected: data20.mount stays active after every loop device removal. The
by-uuid unit stays (or becomes) active, following /dev/nvme1n1.

Actual, systemd 249.11-0ubuntu3.23, kernel 6.8.0-1069-aws: /data20
unmounted in 14/20 trials. Every trial in which the symlink and unit had
moved to the loop device failed. Every trial in which they stayed on
nvme1n1 passed.

  systemd 249 (249.11-0ubuntu3.23)
  6.8.0-1069-aws
  trial=1 loop=loop4 link_while_up=../../loop4 unit_followed=loop4 
after_remove: by-uuid_unit=inactive data20.mount=inactive
  trial=2 loop=loop4 link_while_up=../../nvme1n1 unit_followed=nvme1n1 
after_remove: by-uuid_unit=active data20.mount=active
  trial=3 loop=loop4 link_while_up=../../loop4 unit_followed=loop4 
after_remove: by-uuid_unit=inactive data20.mount=inactive
  ...
  RESULT: /data20 unmounted in 14/20 trials

Same host and disk, after "apt-get install --allow-downgrades" of
systemd, udev and libs 249.11-0ubuntu3 (PID1 re-executed;
Manager.Version "249.11-0ubuntu3"): 0/20. The symlink still moves to the
loop device, but the unit stays on nvme1n1.

  systemd 249 (249.11-0ubuntu3)
  6.8.0-1069-aws
  trial=1 loop=loop4 link_while_up=../../loop4 unit_followed=nvme1n1 
after_remove: by-uuid_unit=active data20.mount=active
  ...
  RESULT: /data20 unmounted in 0/20 trials

An equivalent loop-device script gave 0/10 on each of these (EC2):
- Ubuntu 20.04 (systemd 245)
- Ubuntu 24.04 (systemd 255)
- Amazon Linux 2023 (systemd 252)

On 24.04 and AL2023 the unit followed the loop device and moved back to
the disk on removal.

We see the same failure with a device-mapper device in place of the loop
device. In that setup the device's udev rules first hide it
(ENV{SYSTEMD_READY}="0", SYMLINK=""), then un-hide it with "udevadm
trigger --action=change" before removal. In that variant the outcome
depended on the device-mapper device's name: one name failed 5/5,
another passed 5/5, and random names failed 4/12. That fits an
iteration-order dependence, described below.

What the journal shows on a failure, in the same tick as the second device's 
removal:
- the second device's own units go "plugged -> dead";
- "dev-disk-by\x2duuid-<UUID>.device: Changed plugged -> dead";
- a stop job for data20.mount (StopPropagatedFrom), and the unmount.

Afterwards, /dev/disk/by-uuid/<UUID> points at the disk and
/run/udev/links has the disk's claim, but "systemctl show <by-uuid unit>
-p SysFSPath" is empty and the unit is inactive. "udevadm trigger
--action=change /dev/nvme1n1" plugs the unit again, but the mount stays
stopped.

Probable cause (from reading the patched jammy source; we have not built
a test package)

The SRU for LP: #2100252 added lp2100252-0003-core-device-always-accept-
syspath-change.patch, a backport of upstream 4a1a1caf2192 ("core/device:
always accept syspath change"). With it, a devlink unit follows
whichever device its symlink currently points at, so it joins the second
device's same_sysfs list.

On that device's removal, device_setup_devlink_units_on_remove() walks
the removed device's same_sysfs list with LIST_FOREACH and re-resolves
each /dev path. A unit whose path still resolves to another ready device
is moved there, so it escapes the "unset all found bits" that follows
for the removed syspath.

The loop body can unlink the current element. A devlink that no longer
exists goes DEAD, and device_unset_sysfs() runs LIST_REMOVE. A unit that
moves to the other device has device_set_sysfs() run LIST_REMOVE plus
LIST_PREPEND. In jammy, LIST_FOREACH (src/basic/list.h) reads
i->same_sysfs_next after the body has run, and LIST_REMOVE has set it to
NULL or pointed it into another list. So the walk can end before it
reaches the by-uuid unit. The unit is then cleared along with the
removed device and goes dead.

Upstream, 4a1a1caf was written on top of 80a226b26b5e ("list: make
LIST_FOREACH() and LIST_FOREACH_BACKWARDS() safer", v251), which saves
the next pointer before running the body. That commit is not in the
jammy tree, and the backport's header doesn't list it among the commits
it worked around.

Whether the walk reaches the by-uuid unit depends on its position in the
list. That position follows the order in which PID1 iterated the
device's DEVLINKS set, which is hash order, so it depends on the link
names. That matches the name-dependence above.

Suggested fix: backport 80a226b26b5e, or make the LIST_FOREACH in
device_setup_devlink_units_on_remove() save the next element before
running its body.

Workarounds (each 0/10 on a host and boot that failed 10/10)
- Give the second device a lower link priority in its udev rule 
(OPTIONS+="link_priority=-100"), so /dev/disk/by-uuid stays on the original 
disk.
- Or, after the second device appears, run "udevadm trigger --action=change 
<original disk>; udevadm settle". That points the link and the unit back at the 
original disk before the second device is removed.

Environment
- Ubuntu 22.04.5 LTS on Amazon EC2 (x86_64), kernel 6.8.0-1069-aws
- systemd/udev 249.11-0ubuntu3.23 (bad); 249.11-0ubuntu3 (good). The changelog 
puts the LP: #2100252 patches in 249.11-0ubuntu3.17. We have not tested the 
versions between .17 and .23 individually.

** Affects: systemd (Ubuntu)
     Importance: Undecided
         Status: New


** Tags: jammy regression-update

-- 
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2174993

Title:
  [jammy] fstab mount stopped when a second device with the same
  filesystem UUID is removed (regression from LP: #2100252 in
  249.11-0ubuntu3.17)

To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/2174993/+subscriptions


-- 
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs

Reply via email to