On 6/20/26 13:01, [email protected] wrote:
David Christensen <[email protected]> wrote:
AIUI neither XFS nor mdadm compute, store, or verify checksums of
data or metadata on disk.
XFS checksums metadata for many years now (12?), but it doesn't
checksum user data.
Thank you for the clarification:
https://wiki.archlinux.org/title/XFS#Checksumming
On 6/20/26 13:26, Robert Heller wrote:
I've always thought that the hardware controller checksums raw disk
blocks (sectors) as part of the low-level I/O processing in the
controller hardware's "firmware" and that this is how the controller
knows it has a bad block.
That is also my understanding. The HDD's I own are proprietary, so the
engineering documentation is unavailable. But, I seem to recall reading
an article on the WWW stating that HDD's store additional data in hidden
blocks on the media to allow detecting and/or correcting several flipped
bits.
On 6/20/26 16:23, Paul Leiber wrote:
I just noticed that I didn't manage to make clear that (1) I don't
think that there is one specific failed partition, but that both
partitions containing databases seem to work, but not at the same
time, and that (2) I want to keep the data on the seemingly failed
device.
Let me explain with an example: I am using KODI to access my video
data from different devices. A couple of months ago, I switched KODI
to using a centralized database (containing metadata information on
movies, watch status, etc.) in order to maintain only one database
instead of a database on each device running KODI. The data is
stored on the database VM, running MariaDB, which stores the data on
the md software RAID1 (at least that's what was supposed to happen).
I spent some time configuring the metadata, e.g. correcting mistakes
in the movie titles etc. I then noticed that I mistakenly selected
English language to display the movie descriptions. Because of
family members not fluent in English, I redid the metadata
configuration in German. (It was an annoying work, therefore I
remember it well.) Then, some time later, after a reboot of the
hypervisor (and the database VM) due to kernel updates, the
language of the movie descriptions was displayed in English again. I
attributed this to a corrupt database after the database VM reboot
and loaded a database backup from some time ago, where the movie
description was still in English. So I did the metadata
configuration a third time, again in German. (I guess you can
imagine the fun I had.) Then, a couple of days ago, after a reboot
of the hypervisor and the database VM, the KODI movie description
was displayed in English again. That's when I really started
digging, because now it was clear that there were actually two
intact, but differing databases. (To be clear: There were some other
changes to other databases that also were affected in a similar
manner which I don't mention in this example, so this issue is not
restricted to the KODI database).
Based on the available data, I attribute this issue to the RAID1
which seems to select one of two partitions at random when booting
the hypervisor. Indications: - The last update time in the
description of the (seemingly) failed device given by mdadm --
examine match the point in time of the switch from one database
version ("German") to the other ("English"), therefore I assume that
the switch happens at the software RAID level. - A failure at
hardware level doesn't seem likely, because how could there suddenly
be an older version of a database available in a RAID1 if one device
fails and the RAID1 is degraded, and this after entirely rebuilding
the database from a backup? And, mind you, this switch to an older
version of the database didn't happen just once, but at least two
times. The data (in English) simply shouldn't have been available
anymore at this point if the RAID1 had been working as intended.
The most likely explanation to me is that the RAID1 has been running
in a degraded state for some time (unnoticed by me), the database
changes (e. g. from English to German) were stored to just one of
the two partitions, and at some point the RAID1 switched to the
other partition after a reboot, containing intact, but older (e. g.
English) data. As a defective hardware doesn't seem likely, I assume
that something in my setup causes this behaviour by md. But of
course, I might be wrong and I am open to other explanations. For
example, what my assumption fails to explain is why the switch only
happens from time to time, and not more often, e.g. after each
reboot.
The example you kindly give is for removing a seemingly failed
partition (currently dm-30, "German" database) from a md RAID1,
keeping the data on an intact partition (currently dm-31, "English"
database) and than re-fgadding a partition to the RAID1. This is
pretty straightforward: the data is kept and replicated from the
valid partition to the freshly added one. However, in my case, the
dataset I want to keep is on the seemingly failed partition not used
in the RAID (currently dm-30, "German").
Options I see (besides recreating the RAID1 from scratch and using
an available backup to restore the data, losing some data):
1. I could fail the seemingly intact partition or remove the RAID1
entirely, somehow use the seemingly failed partition (dm-30,
"German") to create a new RAID without losing the data on it, then
add the other partition (dm-31) as a new drive and have the data
replicated. I am not sure if this is possible, therefore my question
to this list.
2. Another option is to reboot the hypervisor and hope for a switch
of the RAID to the partition containing the more recent version of
the database, then follow your guide. But I am not really confident
that such a "strategy" is the best choice I have at the moment.
Also, I just tried a reboot three times, each time the data in the
database is the wrong, old one.
3. I could also backup the database from the seemingly failed
partition in order to not lose data and then use this backup to
recreate the RAID1, but I would need to mount that partition, which
ended in an error when I tried it.
And, of course, I don't want this to happen again, therefore I want
to find the root cause for this situation and fix it. If it is not
the missing information in /etc/mdadm/mdadm.conf, what else could it
be?
Sorry for the lengthy posts, I don't know how to describe this
situation clearly in a shorter way.
Do you own a power supply tester? If not, I suggest buying one. When
you have one, test your power supply. It is possible for one rail to
fail (e.g. +12 VDC), the computer to boot, and all or part of the
computer to operate incorrectly. Without a power supply tester, you
will be chasing seemingly random errors until the power supply goes to
100% failure and/or you damage/ destroy other hardware.
Does your computer have ECC memory? If not, I suggest getting a
computer with ECC memory. In any case, I suggest testing your memory
with memtest86+ for 24 hours.
Have you tested your hard disks? If not, I suggest running smartctl(8)
"--test long". When testing is done, view the results with "--xall".
Have you validated the filesystem with fsck.xfs(8)? If not, I suggest
doing so.
Do you have streams of database transactions since the last known good
backups? If so, can they be replayed?
Can you switch the databases to read-only, shutdown, disconnect the
first disk, boot, backup the database(s), shutdown, connect the first
disk, disconnect the second disk, boot, backup the database(s),
shutdown, and connect the second disk? If so, you could then restore
those backups, and the last known good backups, (with different names,
read-only) and trouble-shoot. It may or may not be possible to identify
the newest data and implement queries/ scripts to do the three-way merge.
David
