On Tue 22 Feb 2011, Dave Howorth wrote: > > Some basic parameters for the two filesystems and two others that are > involved in my tests are: > > data 2.0 TB, xfs, part of 4 disk md RAID10 > backup 1.2 TB, reiserfs, part of 3 disk 3ware 9650SE RAID5
3ware, there's your problem... At work we had a 24-disk storage system, with 2 x 3ware 9550 RAID. We had had trepidations about using hardware raid, as all our earlier tests had shown that hardware raid basically sucks (OK, this beginning of the century that we had done the tests). However there was not a lot of choice when it comes to attaching that amount of disks. It was terribly slow. After struggling for a couple of months, I wiped it all, reconfigured the 3ware controllers to just present 24 JBOD disks, and configured the raid under linux with md. That performed a hell of a lot better. Initial tests with the 3ware controllers didn't show it to be too bad, but as soon as we hit the system with the normal load (about 3 rsyncs in parallel) the performance really hit rock bottom. Apparently the 3wares can't really handle concurrent stuff well. Recently the cabinet underwent an upgrade, and the motherboard and 3ware controllers were replaced. We now went with an Areca controller. That does work quite well; well enough that we haven't bothered with linux md raid this time. > I measured the disk speeds with hdparm: > > # hdparm -tT /dev/mapper/vg--backup-lv--storage 207.82 MB/s Of course, the question is whether the 3ware controller is really syncing to the disks when it reports that it does. I suspect it's not, so I don't trust this number. Also we've given up on reiserfs in the last couple of years, it's never really been stable with 2.6 kernels (on 2.4 kernels we never had problems). We're now happy with ext4, although I'm experimenting with btrfs and besides a number of crashes when I was experimenting with snapshots (and not really understanding what I was doing ;-) that works quite well as well. In short, what I'd do: - don't use the 3ware raid features, turn it into a JBOD and do raid with linux. - replace reiserfs with ext4 or btrfs. Perhaps xfs if you mainly deal with large files. Paul _______________________________________________ Dirvish mailing list [email protected] http://www.dirvish.org/mailman/listinfo/dirvish
