On 05/04/17 05:44, Konstantin Olchanski wrote: > Moving to ZFS because of issues like this. RAID6 rebuild with 4-6-8-10TB disks > has become too scary. If there is any transient error during the rebuild, > the md driver starts kicking disks out, getting into funny states with many > "missing" disks, recovery is only via "mdadm --assemble --force" and without > per-file checksums in ext4/xfs there is no confidence whatsoever that data > was not subtly corrupted. > > ZFS is also scary, but seems to behave well, even in the presence > of "bad disks". ZFS scrub seems to overcome/rewrite bad sectors, > bad data on disk (if I poop on a disk using "dd"), all without corrupting > data files (I compute and check my own sha-512 checksums for each file).
Heh - another soon to be victim of ZFS on linux :) You'll quickly realise that the majority of major features you'd expect to work - don't. You can't grow a ZFS 'raid'. You're stuck with the number of disks you first start with. You'll find out more as you go down this rabbit hole. > BTRFS is even better (on paper), but not usable in el7.3 because it has no > concept of "failed disk". If you pull a disk on btrfs, it will fill /var/log > with disk error messages, will not take any mitigation/recovery action > (drop disk from array, rerun the data balancer, etc). DO NOT USE RAID5/6 WITHIN BTRFS. I have tried this before and have the many Gb of lost data when it goes wrong. In fact, I discovered several new bugs that I lodged with the BTRFS guys - which led to warnings of DO NOT USE PARITY BASED RAID LEVELS IN BTRFS becoming the official line. However, BTRFS is very stable if you use it as a simple filesystem. You will get more flexible results in using mdadm with btrfs on top of it. mdadm can be a pain to tweak - but almost all problems are well known and documented - and unless you really lose all your parity, you'll be able to recover with much less data loss than most other concoctions. > > K.O. > > > On Tue, Apr 04, 2017 at 04:17:22PM +0200, David Sommerseth wrote: >> Hi, >> >> I just need some help to understand what might be the issue on a SL7.3 >> server which today decided to disconnect two drives from a RAID 6 setup. >> >> First some gory details >> >> - smartctl + mdadm output >> <https://paste.fedoraproject.org/paste/wLyz44nipkJ7FgKxWk-1mV5M1UNdIGYhyRLivL9gydE=> >> >> - kernel log messages >> https://paste.fedoraproject.org/paste/mkyjZINKnkD4SQcXTSxyt15M1UNdIGYhyRLivL9gydE= >> >> >> The server is setup with 2x WD RE4 harddrives and 2x Seagate >> Constellation ES.3 drives. All 4TB, all was bought brand new. They're >> installed in a mixed pattern (sda: RE4, sdb: ES3, sdc: RE4, sdd: ES3) >> ... and the curious devil in the detail ... there are no /dev/sde >> installed on this system - never have been even, at least not on that >> controller. (Later today, I attached a USB drive to make some backups - >> which got designated /dev/sde) >> >> This morning *both* ES.3 drives (sdb, sdd) got disconnected and removed >> from the mdraid setup. With just minutes in between. On drives which >> have been in production for less than 240 days or so. >> >> lspci details: >> 00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset >> Family SATA AHCI Controller (rev 05) >> >> Server: HP ProLiant MicroServer Gen8 (F9A40A) >> >> <https://www.hpe.com/us/en/product-catalog/servers/proliant-servers/pip.specifications.hpe-proliant-microserver-gen8.5379860.html> >> >> >> Have any one else experienced such issues? Several places on the net, >> the ata kernel error messages have been resolved by checking SATA cables >> and their seating. It just sounds a bit too incredible that two >> harddrives of the same brand and type in different HDD slots have the >> same issues but not at the exact same time (but close, though). And I >> struggle to believe two identical drives just failing so close in time. >> >> What am I missing? :) Going to shut down the server soon (after last >> backup round) and will double check all the HDD seating and cabling. >> But I'm not convinced that's all just yet. >> >> >> -- >> kind regards, >> >> David Sommerseth > -- Steven Haigh Email: net...@crc.id.au Web: https://www.crc.id.au Phone: (03) 9001 6090 - 0412 935 897
signature.asc
Description: OpenPGP digital signature