Public bug reported:
Hello, I noticed that my Ubuntu VM would fully boot despite a local mount being
unavailable during
boot, instead of dropping into emergency mode. The below report and analysis
was done by AI, but the
explanation does sound plausible to me and I've confirmed that the proposed
drop-in fix does
cause emergency mode/emergency shell to trigger again when a local mount fails
during boot.
I tested this on a Ubuntu 24.04.3 LTS VM
(ubuntu-24.04-server-cloudimg-amd64.img) which has the following
grub packages installed:
grub-common/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
grub-efi-amd64-bin/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
grub-efi-amd64-signed/noble-updates,now 1.202.5+2.12-1ubuntu7.3 amd64
[installed,automatic]
grub-gfxpayload-lists/noble,now 0.7build2 amd64 [installed,automatic]
grub-pc-bin/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
grub-pc/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed]
grub2-common/noble-updates,now 2.12-1ubuntu7.3 amd64 [installed,automatic]
Best regards
Samuel
Summary
-------
When a required fstab mount fails at boot, systemd fails local-fs.target and
triggers emergency.target. On Ubuntu the boot continues anyway: sysinit.target,
basic.target and multi-user.target are reached, emergency.service starts late
and in parallel, and the result is a fully booted system with sshd running,
an empty mountpoint, and "systemctl is-system-running" reporting "maintenance".
Cause: grub-initrd-fallback.service is wanted by emergency.target but keeps
systemd's default dependencies, so it requires sysinit.target. Starting
emergency.target therefore also starts sysinit.target, which emergency mode
is supposed to prevent.
Reproduce
---------
1. Add a required mount (no "nofail") for a device to /etc/fstab, e.g. a
virtiofs share:
test_2 /data/test_2 virtiofs noatime 0 0
2. Remove the device from the VM and reboot.
Expected: boot stops at the emergency shell, sysinit.target is never
reached.
Actual (systemd journal, one boot):
12:36:25 data-test_2.mount: Failed with result 'exit-code'.
12:36:25 Dependency failed for local-fs.target - Local File Systems.
12:36:25 local-fs.target: Triggering OnFailure= dependencies.
... (no network configured so systemd-networkd-wait-online.service took a
while to fail)
12:38:26 Reached target sysinit.target - System Initialization.
12:38:26 Started emergency.service - Emergency Shell.
12:38:26 Reached target basic.target - Basic System.
12:38:27 Reached target emergency.target - Emergency Mode.
12:38:27 Reached target multi-user.target - Multi-User System.
Analysis
--------
$ systemctl show grub-initrd-fallback.service -p Requires,WantedBy
Requires=system.slice sysinit.target
WantedBy=emergency.target rescue.target sleep.target multi-user.target
The emergency transaction contains two jobs for sysinit.target: a stop
(sysinit.target has Conflicts=emergency.target) and a start (emergency.target
wants grub-initrd-fallback.service, which requires sysinit.target). Neither
is essential to the anchor job, and systemd's tie-break for such pairs
(src/core/transaction.c, delete_one_unmergeable_job) keeps the stop only if
it was pulled in by a Conflicts= on the unit being started. Here it was not,
so the stop is dropped and the pending sysinit.target start job survives.
Because emergency.service is ordered after sysinit.target, the emergency
shell then waits for the normal boot instead of blocking it.
systemd's own units wanted by rescue.target (e.g.
systemd-update-utmp-runlevel.service) set DefaultDependencies=no for this
reason.
Fix
---
In grub-initrd-fallback.service:
[Unit]
DefaultDependencies=no
Conflicts=shutdown.target
Before=shutdown.target
The unit already has After=local-fs.target and
ConditionPathExists=/boot/grub/grub.cfg, so normal boots are unaffected.
Verified as a drop-in: with the same missing device the boot now halts at the
emergency shell and sysinit.target is never reached.
Alternative: drop emergency.target from WantedBy= if an emergency boot should
not count as a successful boot for the initrd fallback logic.
References
----------
LP: #1823391 added WantedBy=emergency.target in grub2 2.04-1ubuntu1, so all
releases since eoan are likely affected.
** Affects: grub2 (Ubuntu)
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/2168866
Title:
grub-initrd-fallback.service defeats emergency mode: boot continues to
multi-user.target after local-fs.target fails
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/grub2/+bug/2168866/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
