On Mon, Feb 22, 2010 at 10:06 AM, Michael ODonnell <[email protected]> wrote: > Anybody else running CentOS5.x ...
liberty.gnhlug.org is running CentOS 5.4 with kernel 2.6.18-92.1.22.el5 (still haven't rebooted it). liberty$ fgrep -i sync /var/log/kernel* | fgrep -i raid /var/log/kernel:Feb 21 04:22:01 liberty kernel: md: syncing RAID array md0 /var/log/kernel.1:Feb 14 04:22:01 liberty kernel: md: syncing RAID array md0 /var/log/kernel.2:Feb 7 04:22:01 liberty kernel: md: syncing RAID array md0 /var/log/kernel.3:Jan 31 04:22:01 liberty kernel: md: syncing RAID array md0 /var/log/kernel.4:Jan 24 04:22:01 liberty kernel: md: syncing RAID array md0 liberty$ It appears to be doing that at 4:22 AM (US Eastern) every Saturday, for device md0. 4:22 AM is also the timestamp on those email messages about mismatch_cnt I keep getting. Interestingly, the "md1" name does not appear in the log files. (To review: md0 is the /boot/ partition, which is small and quiet. md1 is the everything-else LVM PE partition, which hosts swap space and all other filesystems.) > It looks like the RAIDs on at least seven of our (mostly stock) CentOS5.4 > systems are routinely getting broken and going through a resync operation > on a weekly basis at 4:22am which is when that /etc/cron.weekly script > runs that's generating the mismatch_cnt warnings in question. I believe the script is </etc/cron.weekly/99-raid-check>. It apparently uses a config file </etc/sysconfig/raid-check>, which is well-commented. The control flow of the script seems to be: The operation is only run if the array is in a clean and idle state. If the array is degraded or rebuilding, the operation is skipped for that array. The default operation is "check", not "repair". The default operation can be either a "check" or "repair", as specified by "CHECK="; on liberty, it is "check". Both operation types presumably scan all blocks on all members. In the world of RAID, this is often called "patrol read". It's a good thing. Disks tend to be full of files which are rarely read. If one of those files develops a bad sector, the disk won't notice until you try to read it. Then when a different disk dies and the RAID subsystem tries to rebuild from the remaining member(s), you discover your redundant disks weren't as redundant as you would have liked. Assuming the comments in the config file are accurate, the "check" operation will only attempt to repair "bad sectors". Exactly how "bad sectors" are detected isn't explained, but I presume it means "could not read the block from one member device". Exactly what "repair" means isn't explained, but I presume it means "write the block from the good device to the other device". (This is a good thing -- any hard disk manufactured within the past 20 years or so will remap a bad block once it is written to. And modern hard disks are virtually guaranteed to have bad blocks.) Again assuming the comments are accurate, "check" will *not* attempt to repair mismatches. Mismatches are when all member devices could be read, but the data is not consistent across devices. This is what "mismatch_cnt" reportedly reflects. "repair", on the other hand, will attempt to make the array consistent. How the "repair" will choose which data to keep is not explained, but the phase "luck of the draw" is used. You can set "ENABLED=no" in the config file to disable the whole thing, but before you do that, see above about "patrol read". If you think patrol read is a bad idea, you're probabbly wrong. So! Having done what I should have done in the first place (RTFM and RTFS), I know now why the problem was detected, and what my options for remediation are. That leaves "How did the mismatch occur in the first place?" as my remaining question. Based on what I'm seeing (in particular, the mismatch *only* being in the GRUB stage2 file), I'm going to conclude liberty's mismatch is due to GRUB being installed on both physical hard disks independently (booting from floppy). Whether or not that's the right away to install GRUB is an open question. Assuming that is a valid way to install GRUB, the system should be fine, including for kernel updates, until and unless one mirror member fails. Any updates to kernel files will write to blocks other than the GRUB stage2 file, and be properly mirrored. But if a mirror member dies, then once that bad disk is replaced, the system will copy the good mirror to the new disk, including the "wrong copy" of GRUB. -- Ben _______________________________________________ gnhlug-discuss mailing list [email protected] http://mail.gnhlug.org/mailman/listinfo/gnhlug-discuss/
