I've encountered the same issue, and my server stopped working. I have a
RAID1 array with LVM on top of it.
Here is the root cause and the proper way to fix it. I would also like to
ask Daniel Baumann for advice on how to proceed with a bug fix that I can
work on.
Root cause:
1. Debian's "mdadm" package contains initramfs hooks, initramfs scripts,
and udev rules. You can discover them with "dpkg -L mdadm".
2. The files we are interested in are
"/usr/share/initramfs-tools/scripts/local-block/mdadm" and
"/lib/udev/rules.d/64-md-raid-assembly.rules".
3. The initramfs hooks pack them into the initramfs image after the "mdadm"
package is installed, or after any kernel/initramfs updates.
4. By default, during the boot phase when initramfs is running, udev is
started as well.
5. udev receives events from the kernel and discovers block devices.
6. The "mdadm" udev rule in "/lib/udev/rules.d/64-md-raid-assembly.rules"
tries to assemble the array when block partitions with the
"linux_raid_member" filesystem type are discovered.
7. This rule calls "/sbin/mdadm --incremental --export $devnode --offroot
$env{DEVLINKS}". Incremental mode allows "mdadm" to start building the
array with the first disk and then add more disks as they are discovered.
The working array in my case had 2 disks. When 1 of them is missing,
"--incremental" builds the array, but does not make it active until the
second disk is present. This will never happen on a degraded array.
8. However, "mdadm" contains a "poor man's last-resort" approach that works
in combination with scripts from the "initramfs-tools-core" package. You
can inspect them with "dpkg -L initramfs-tools-core". This package installs
"/usr/share/initramfs-tools/init" and
"/usr/share/initramfs-tools/scripts/local".
9. "init" executes "local_device_setup", and "local_device_setup" is
defined in "scripts/local".
10. "local_device_setup" executes "local_block "${dev_id}"" in a while
loop, and "local_block" executes "scripts/local-block/mdadm". The loop has
a counter. Only after 2/3 of the "ROOTDELAY" time has elapsed, it executes
"mdadm -q --assemble --scan --no-degraded || true". Obviously, with
"--no-degraded", it fails if the array is in a degraded state.
11. After 2/3 of the "ROOTDELAY" time has elapsed, it executes "mdadm -q
--run /dev/md?*" once, and the array becomes active even in a degraded
state.
12. It does not help in my case because I get an active array, but LVM is
not assembled. To assemble it, we need either to do this manually via a
custom script or to trigger udev to discover block devices again. One of
those devices will be our RAID device, so udev will trigger the LVM udev
rule to activate it.
13. Unfortunately, "mdadm" does not handle this well and does not trigger
udev.
Quick fix:
1. Do not edit any content in "/lib/udev/rules.d/" or
"/usr/share/initramfs-tools/". These files will be overwritten by updates.
2. Custom scripts should be placed in "/etc/udev/rules.d/" and
"/etc/initramfs-tools/", respectively.
3. Copy the original "mdadm" script to "/etc/initramfs-tools/": "sudo cp
/usr/share/initramfs-tools/scripts/local-block/mdadm
/etc/initramfs-tools/scripts/local-block/mdadm"
4. Add the following 2 lines after "mdadm -q --assemble --scan --run":
udevadm trigger --action=add -s block || true
wait_for_udev 10
5. Rebuild initramfs: "sudo update-initramfs -u -k all"
6. Optionally, inspect its content with "unmkinitramfs".
Permanent fix. Daniel Baumann, please comment on this.
1. It looks like the logic in the "mdadm" initramfs scripts conflicts with
the "initramfs-tools-core" scripts.
2. "scripts/local-block/mdadm" contains code that removes the
"/run/count.mdadm.initrd" counter. Other lines of code trigger udev when
"/run/count.mdadm.initrd" is missing. This resets the counter to 0, so
technically, with this logic, arrays should be activated. However, this
never happens because "scripts/local" detects the removal and recreates the
file. That is why I have no idea why this logic with removal and udev
triggering exists.
3. Maybe it was just a mistake, or maybe there is some logic that I am not
familiar with. So I am asking for your advice. I can submit a merge request
with the fix described above. I have tested it.
Thank you in advance.
