Hi everybody,
I am using a MDADM software RAID1 as a dedicated MariaDB database file system.
The devices used for the RAID1 are two partitions of identical size which are
LUKS encrypted. The devices are decrypted via entries in /etc/crypttab. The
resulting RAID1 is called /dev/md0, formatted as XFS. (For completeness' sake:
md0 is then forwarded to a database VM which stores the database on the device,
but that shouldn't play a role for my questions, IIUC.)
Some time ago, I noticed that the database content changed after a reboot.
Recent changes to the databases were seemingly lost. I couldn't pinpoint the
cause for this, but attributed it to an unclean shutdown of the database prior
to reboot of the database VM. Data loss in a database of course is not ideal,
so I kept on looking. It seems that I have now identified the root cause for
the data loss in the RAID1.
I checked the RAID1:
root@xxx:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 dm-30[2]
1073593280 blocks super 1.2 [2/1] [_U]
bitmap: 7/8 pages [28KB], 65536KB chunk
The [_U] seems to indicate that the RAID1 is currently degraded and using just
one of the two partitions is currently used for the RAID1.
Checking the partitions the RAID1 is based on gives the following output:
root@xxx:~# mdadm --examine /dev/dm-31
/dev/dm-31:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 6834a10d:edb03a51:cef24158:f9abc812
Name : xxx:0 (local to host xxx)
Creation Time : Fri Nov 4 16:05:45 2022
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 2147186655 sectors (1023.86 GiB 1099.36 GB)
Array Size : 1073593280 KiB (1023.86 GiB 1099.36 GB)
Used Dev Size : 2147186560 sectors (1023.86 GiB 1099.36 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=95 sectors
State : clean
Device UUID : 01c96166:ee782cc7:57bcf889:2ee53b43
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jun 17 13:17:46 2026
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : d46fa108 - correct
Events : 5397997
Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing, 'R' == replacing)
and
root@xxx:~# mdadm --examine /dev/dm-30
/dev/dm-30:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 6834a10d:edb03a51:cef24158:f9abc812
Name : xxx:0 (local to host xxx)
Creation Time : Fri Nov 4 16:05:45 2022
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 2147186655 sectors (1023.86 GiB 1099.36 GB)
Array Size : 1073593280 KiB (1023.86 GiB 1099.36 GB)
Used Dev Size : 2147186560 sectors (1023.86 GiB 1099.36 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=95 sectors
State : clean
Device UUID : 637fc155:8fb21b7c:fff27b71:c7ea1094
Internal Bitmap : 8 sectors from superblock
Update Time : Fri Jun 19 21:34:19 2026
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : b110fe9d - correct
Events : 4814810
Device Role : Active device 1
Array State : .A ('A' == active, '.' == missing, 'R' == replacing)
It can be seen that update time and number of events differ between both
partitions, which seems to indicate different data. I am assuming that due to
some circumstance (wild guess: a race condition when unlocking the LUKS
encryption), the RAID1 is more or less randomly using only one of the
partitions, which then results in differing database versions, depending on
which of the two partitions is currently used.
I also think that I found a possible cause for this misbehaviour. My
/etc/mdadm/mdadm.conf contains just the default settings:
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan, using
# wildcards if desired.
#DEVICE partitions containers
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
# This configuration was auto-generated on Fri, 04 Nov 2022 15:52:55 +0100 by
mkconf
Somehow, I missed to include the RAID1 information for md0 to the configuration file
(e. g. by entering root@localhost:~# mdadm --detail --scan /dev/md0 >>
/etc/mdadm/mdadm.conf). I am not sure if this actually is the cause and adding that
information would solve the issue.
My questions are the following:
1. Is my analysis valid in principle? Especially: Could the root cause for this
issue be that mdadm.conf is missing the information for md0, and could adding
that information prevent data loss or inconsistencies in the future?
2. Can I (re)create the RAID1 md0 or (re-)add the missing partition in an easy
way that no or at least not all information is lost? If yes, how?
I assume that it might not be possible to sync the data from two different
database versions without data loss. If this assumption is correct, I am
willing to use one data set (e. g. the one on dm-31) and discard the other data
set (e. g. the one on dm-30). Guides I found so far describe how to set up a
new RAID1 and copy the data from a partition to the new RAID1. However, perhaps
I am wondering if it is possible to (re-)create a RAID1 using just one existing
partition (e. g. dm-31) without losing the data on this partition, and then add
the other partition to the RAID1?
The databases are backed up regularly. However, the backup is incremental, and
it seems that the different database versions are messing up the incremental
backup, therefore my last valid backup doesn't include the most recent changes
to the database. If it is not possible to salvage the data on one or both of
the partitions, I could swallow the bitter pill and go back to a previous
database state without unacceptable consequences. However, I would like to try
to salvage as much data as possible.
Thank you in advance,
Paul
