On Wed, Mar 26, 2008 at 7:34 PM, Michael Hannon <[EMAIL PROTECTED]> wrote: > Greetings. We have a lately had a lot of trouble with relatively large > (order of 1TB) file systems mounted on RAID 5 or RAID 6 volumes. The > file systems in question are based on ext3. > > In a typical scenario, we have a drive go bad in a RAID array. We then > remove it from the array, if it isn't already, add a new hard drive > (i.e., by hand, not from a hot spare), and add it back to the RAID > array. The RAID operations are all done using mdadm. > > After the RAID array has completed its rebuild, we run fsck on the RAID > device. When we do that, fsck seems to run forever, i.e., for days at a > time, occasionally spitting out messages about files with recognizable > names, but never completing satisfactorily. >
fsck of 1TB is going to take days due to the linear nature of it checking the disk. [ I think the disks for mirrors.kernel.org take many weeks to fsck.] The bigger question is what kind of data are you writing to these disks, and is the ext3 journal large enough for those writes? > The systems in question are typically running SL 4.x. We've read that > the version of fsck that is standard in SL 4 has some known bugs, > especially wrt large file systems. > > Hence, we've attempted to repeat the exercise with fsck.ext3 taken from > the Fedora 8 distribution. This gives us improved, but still not > satisfactory, results. > Did you recompile the binary from source, or did you use it straight? I am just wondering if fsck is dependant on some kernel particulars... To tell you the truth, I have not done anything with Linux Raid in the Terabyte range.. Usually I go with a hardware solution at that point (usually for business reasons.. that much storage usually comes with a box with hardware raid). I did run into a similar issue though trying to help someone last week on a SuSE box with ext3. They also had a long fsck and weird file names coming up. I think they went with the same solution ( restore from backups). -- Stephen J Smoogen. -- CSIRT/Linux System Administrator How far that little candle throws his beams! So shines a good deed in a naughty world. = Shakespeare. "The Merchant of Venice"
