Am 21.06.26 um 06:47 schrieb David Christensen:
On 6/20/26 13:01, [email protected] wrote:
David Christensen <[email protected]> wrote:
AIUI neither XFS nor mdadm compute, store, or verify checksums of data or
metadata on disk.
XFS checksums metadata for many years now (12?), but it doesn't checksum user
data.
Thank you for the clarification:
https://wiki.archlinux.org/title/XFS#Checksumming
On 6/20/26 13:26, Robert Heller wrote:
I've always thought that the hardware controller checksums raw disk blocks (sectors) as
part of the low-level I/O processing in the controller hardware's "firmware"
and that this is how the controller
knows it has a bad block.
That is also my understanding. The HDD's I own are proprietary, so the
engineering documentation is unavailable. But, I seem to recall reading an
article on the WWW stating that HDD's store additional data in hidden blocks on
the media to allow detecting and/or correcting several flipped bits.
On 6/20/26 16:23, Paul Leiber wrote:
I just noticed that I didn't manage to make clear that (1) I don't think that
there is one specific failed partition, but that both partitions containing
databases seem to work, but not at the same time, and that (2) I want to keep
the data on the seemingly failed device.
Let me explain with an example: I am using KODI to access my video data from
different devices. A couple of months ago, I switched KODI to using a
centralized database (containing metadata information on movies, watch status,
etc.) in order to maintain only one database instead of a database on each
device running KODI. The data is stored on the database VM, running MariaDB,
which stores the data on the md software RAID1 (at least that's what was
supposed to happen). I spent some time configuring the metadata, e.g.
correcting mistakes in the movie titles etc. I then noticed that I mistakenly
selected English language to display the movie descriptions. Because of family
members not fluent in English, I redid the metadata configuration in German.
(It was an annoying work, therefore I remember it well.) Then, some time later,
after a reboot of the hypervisor (and the database VM) due to kernel updates,
the
language of the movie descriptions was displayed in English again. I attributed
this to a corrupt database after the database VM reboot and loaded a database
backup from some time ago, where the movie description was still in English. So
I did the metadata configuration a third time, again in German. (I guess you
can imagine the fun I had.) Then, a couple of days ago, after a reboot of the
hypervisor and the database VM, the KODI movie description was displayed in
English again. That's when I really started digging, because now it was clear
that there were actually two intact, but differing databases. (To be clear:
There were some other changes to other databases that also were affected in a
similar manner which I don't mention in this example, so this issue is not
restricted to the KODI database).
Based on the available data, I attribute this issue to the RAID1 which seems to select one of two
partitions at random when booting the hypervisor. Indications: - The last update time in the
description of the (seemingly) failed device given by mdadm -- examine match the point in time of
the switch from one database version ("German") to the other ("English"),
therefore I assume that the switch happens at the software RAID level. - A failure at hardware
level doesn't seem likely, because how could there suddenly be an older version of a database
available in a RAID1 if one device fails and the RAID1 is degraded, and this after entirely
rebuilding the database from a backup? And, mind you, this switch to an older version of the
database didn't happen just once, but at least two times. The data (in English) simply shouldn't
have been available anymore at this point if the RAID1 had been working as intended.
The most likely explanation to me is that the RAID1 has been running in a
degraded state for some time (unnoticed by me), the database changes (e. g.
from English to German) were stored to just one of the two partitions, and at
some point the RAID1 switched to the other partition after a reboot, containing
intact, but older (e. g. English) data. As a defective hardware doesn't seem
likely, I assume that something in my setup causes this behaviour by md. But of
course, I might be wrong and I am open to other explanations. For example, what
my assumption fails to explain is why the switch only happens from time to
time, and not more often, e.g. after each reboot.
The example you kindly give is for removing a seemingly failed partition (currently dm-30, "German"
database) from a md RAID1, keeping the data on an intact partition (currently dm-31, "English"
database) and than re-fgadding a partition to the RAID1. This is pretty straightforward: the data is kept and
replicated from the valid partition to the freshly added one. However, in my case, the dataset I want to keep
is on the seemingly failed partition not used in the RAID (currently dm-30, "German").
Options I see (besides recreating the RAID1 from scratch and using an available
backup to restore the data, losing some data):
1. I could fail the seemingly intact partition or remove the RAID1 entirely, somehow use
the seemingly failed partition (dm-30, "German") to create a new RAID without
losing the data on it, then add the other partition (dm-31) as a new drive and have the
data replicated. I am not sure if this is possible, therefore my question to this list.
2. Another option is to reboot the hypervisor and hope for a switch of the RAID to the
partition containing the more recent version of the database, then follow your guide. But
I am not really confident that such a "strategy" is the best choice I have at
the moment. Also, I just tried a reboot three times, each time the data in the database
is the wrong, old one.
3. I could also backup the database from the seemingly failed partition in
order to not lose data and then use this backup to recreate the RAID1, but I
would need to mount that partition, which ended in an error when I tried it.
And, of course, I don't want this to happen again, therefore I want to find the
root cause for this situation and fix it. If it is not the missing information
in /etc/mdadm/mdadm.conf, what else could it be?
Sorry for the lengthy posts, I don't know how to describe this situation
clearly in a shorter way.
Do you own a power supply tester? If not, I suggest buying one. When you have
one, test your power supply. It is possible for one rail to fail (e.g. +12
VDC), the computer to boot, and all or part of the computer to operate
incorrectly. Without a power supply tester, you will be chasing seemingly
random errors until the power supply goes to 100% failure and/or you damage/
destroy other hardware.
There are two identical hard drives (manufacturer, type, size) in my system
used for data storage. Both have an identical layout: One large partition used
as BTRFS RAID1 (duplicated data and metadata), one small partition which is
used for the md software RAID1. BTRFS is working fine, it's just the md
software RAID that is not working correctly. I can't rule out an issue with
power supply, but it seems unlikely, considering that BTRFS is not having any
issue. But I will put a test of the power supply on the list of things to try.
Does your computer have ECC memory? If not, I suggest getting a computer with
ECC memory. In any case, I suggest testing your memory with memtest86+ for 24
hours.
Yes, it does have ECC memory. I will put the memory test on the list as well.
Have you tested your hard disks? If not, I suggest running smartctl(8) "--test long". When testing is done, view the results with "--xall".
Both disks are monitored via smartctl. Automated short and long tests are being
done regularly. There are no indications for hardware failure in the smart data.
Have you validated the filesystem with fsck.xfs(8)? If not, I suggest doing so.
I just did a check of the file system (using xfs_repair -n), with no errors
reported.
Do you have streams of database transactions since the last known good backups? If so, can they be replayed?
Not exactly knowing what such a stream is, I guess I don't have one. But I am
not sure. Will check.
Can you switch the databases to read-only, shutdown, disconnect the first disk, boot, backup the database(s), shutdown, connect the first disk, disconnect the second disk, boot, backup the database(s), shutdown, and connect the second disk? If so, you could then restore those backups, and the last known good backups, (with different names, read-only) and trouble-shoot. It may or may not be possible to identify the newest data and implement queries/ scripts to do the three-way merge.
That's a good suggestion. I will need to check what booting with just one disk
could do to the BTRFS filesystem, but this might be a way to force md to use
the disk which is currently indicated as failed.
