On 6/20/26 09:01, Paul Leiber wrote:
Hi everybody,
I am using a MDADM software RAID1 as a dedicated MariaDB database file
system. The devices used for the RAID1 are two partitions of identical
size which are LUKS encrypted. The devices are decrypted via entries
in /etc/crypttab. The resulting RAID1 is called /dev/md0, formatted as
XFS. (For completeness' sake: md0 is then forwarded to a database VM
which stores the database on the device, but that shouldn't play a role
for my questions, IIUC.)
Some time ago, I noticed that the database content changed after a
reboot. Recent changes to the databases were seemingly lost. I couldn't
pinpoint the cause for this, but attributed it to an unclean shutdown of
the database prior to reboot of the database VM. Data loss in a database
of course is not ideal, so I kept on looking. It seems that I have now
identified the root cause for the data loss in the RAID1.
I checked the RAID1:
root@xxx:~# cat /proc/mdstat
Personalities : [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 dm-30[2]
1073593280 blocks super 1.2 [2/1] [_U]
bitmap: 7/8 pages [28KB], 65536KB chunk
The [_U] seems to indicate that the RAID1 is currently degraded and
using just one of the two partitions is currently used for the RAID1.
Checking the partitions the RAID1 is based on gives the following output:
root@xxx:~# mdadm --examine /dev/dm-31
/dev/dm-31:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 6834a10d:edb03a51:cef24158:f9abc812
Name : xxx:0 (local to host xxx)
Creation Time : Fri Nov 4 16:05:45 2022
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 2147186655 sectors (1023.86 GiB 1099.36 GB)
Array Size : 1073593280 KiB (1023.86 GiB 1099.36 GB)
Used Dev Size : 2147186560 sectors (1023.86 GiB 1099.36 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=95 sectors
State : clean
Device UUID : 01c96166:ee782cc7:57bcf889:2ee53b43
Internal Bitmap : 8 sectors from superblock
Update Time : Wed Jun 17 13:17:46 2026
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : d46fa108 - correct
Events : 5397997
Device Role : Active device 0
Array State : A. ('A' == active, '.' == missing, 'R' == replacing)
and
root@xxx:~# mdadm --examine /dev/dm-30
/dev/dm-30:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 6834a10d:edb03a51:cef24158:f9abc812
Name : xxx:0 (local to host xxx)
Creation Time : Fri Nov 4 16:05:45 2022
Raid Level : raid1
Raid Devices : 2
Avail Dev Size : 2147186655 sectors (1023.86 GiB 1099.36 GB)
Array Size : 1073593280 KiB (1023.86 GiB 1099.36 GB)
Used Dev Size : 2147186560 sectors (1023.86 GiB 1099.36 GB)
Data Offset : 264192 sectors
Super Offset : 8 sectors
Unused Space : before=264112 sectors, after=95 sectors
State : clean
Device UUID : 637fc155:8fb21b7c:fff27b71:c7ea1094
Internal Bitmap : 8 sectors from superblock
Update Time : Fri Jun 19 21:34:19 2026
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : b110fe9d - correct
Events : 4814810
Device Role : Active device 1
Array State : .A ('A' == active, '.' == missing, 'R' == replacing)
It can be seen that update time and number of events differ between both
partitions, which seems to indicate different data. I am assuming that
due to some circumstance (wild guess: a race condition when unlocking
the LUKS encryption), the RAID1 is more or less randomly using only one
of the partitions, which then results in differing database versions,
depending on which of the two partitions is currently used.
I also think that I found a possible cause for this misbehaviour. My /
etc/mdadm/mdadm.conf contains just the default settings:
# mdadm.conf
#
# !NB! Run update-initramfs -u after updating this file.
# !NB! This will ensure that initramfs has an uptodate copy.
#
# Please refer to mdadm.conf(5) for information about this file.
#
# by default (built-in), scan all partitions (/proc/partitions) and all
# containers for MD superblocks. alternatively, specify devices to scan,
using
# wildcards if desired.
#DEVICE partitions containers
# automatically tag new arrays as belonging to the local system
HOMEHOST <system>
# instruct the monitoring daemon where to send mail alerts
MAILADDR root
# definitions of existing MD arrays
# This configuration was auto-generated on Fri, 04 Nov 2022 15:52:55
+0100 by mkconf
Somehow, I missed to include the RAID1 information for md0 to the
configuration file (e. g. by entering root@localhost:~# mdadm --detail
--scan /dev/md0 >> /etc/mdadm/mdadm.conf). I am not sure if this
actually is the cause and adding that information would solve the issue.
My questions are the following:
1. Is my analysis valid in principle? Especially: Could the root cause
for this issue be that mdadm.conf is missing the information for md0,
and could adding that information prevent data loss or inconsistencies
in the future?
2. Can I (re)create the RAID1 md0 or (re-)add the missing partition in
an easy way that no or at least not all information is lost? If yes, how?
I assume that it might not be possible to sync the data from two
different database versions without data loss. If this assumption is
correct, I am willing to use one data set (e. g. the one on dm-31) and
discard the other data set (e. g. the one on dm-30). Guides I found so
far describe how to set up a new RAID1 and copy the data from a
partition to the new RAID1. However, perhaps I am wondering if it is
possible to (re-)create a RAID1 using just one existing partition (e. g.
dm-31) without losing the data on this partition, and then add the other
partition to the RAID1?
The databases are backed up regularly. However, the backup is
incremental, and it seems that the different database versions are
messing up the incremental backup, therefore my last valid backup
doesn't include the most recent changes to the database. If it is not
possible to salvage the data on one or both of the partitions, I could
swallow the bitter pill and go back to a previous database state without
unacceptable consequences. However, I would like to try to salvage as
much data as possible.
Thank you in advance,
Paul
I will interpret the above as MariaDB is storing data on files on an XFS
file system on an mdadm RAID1 block device built from two LUKS
containers on two partitions of the same size on two hard disk drives.
AIUI, mdadm RAID protects you when the disk controller is unable to read
a block on one disk. When that happens, mdadm will read other disk(s)
in the array, compute the requested block, and return the requested
information to the calling application. (I assume mdadm will also write
the computed block back to the original disk, write a log entry, and
take other actions as designed and configured.)
AIUI neither XFS nor mdadm compute, store, or verify checksums of data
or metadata on disk. So, if a bit, byte, block, etc., changes on disk
unexpectedly, neither XFS nor mdadm will know; they will simply use the
information on disk. Whatever is looking at that information (e.g.
MariaDB or XFS) may or may not notice the corruption. The user may or
may not notice the corruption.
To protect against corruption in memory, you need error correction code
memory. This requires hardware support on the motherboard and memory
modules.
To protect against data corruption on storage, you need a checksumming
storage system. AIUI btrfs and ZFS are the obvious choices on Debian
GNU/Linux. Unfortunately, ZFS is not supported OOTB due to licensing
conflicts; you must install ZFS. If you choose to do so, it is wise to
also install ZFS on your maintenance/ rescue media.
Over the past several years, I migrated my file storage services and CVS
repository from a desktop computer with non-ECC memory, Debian
GNU/Linux, SATA HDD's, LUKS, mdadm RAID1, and ext4 to an entry-level
server with ECC memory, FreeBSD, SAS/SATA HBA's, new SAS HDD's, new SAS
and SATA cables, GELI, and ZFS mirror. The cost was moderate, the
learning curve was non-trivial, and I caused some non-critical data loss
along the way, but now everything is accurate and reliable. I suggest
that you migrate your MariaDB storage similarly.
David
