On Sat, 11 Dec 2010, Stephan Budach wrote:
At first I disabled all write cache and read ahead options for each raid
group on the raids, since I wanted to provide ZFS as much control over the
drives as possible, but the performance was quite worse. I am running this
zpool on a Sun Fire X4170M2 with 32 GB of RAM so I ran bonnie++ with -s 63356
-n 128 and got these results:
Sequential Output
char: 51819
block: 50602
rewrite: 28090
I am not very familiar with bonnie++ output. Does 51819 mean
51MB/second? If so, that is perhaps 1 disk's worth of performance.
Random seeks: 510 <- this seems really low to me, isn't it?
It does seem a bit low. Everything depends on if the "random seek"
was satisfied from ARC cache or from the underlying disk. You should
be able to obtain at least the number of physical seeks available from
1/2 your total disks. For example, with 16 pair and if each disk
could do 100 seeks per second, then you should expect at least 8*100
random seeks per second. With zfs mirroring and doing only
read-seeks, you should expect to get up to 75% of the seek capability
of all 16 disks combined.
Since I was curious, what would happen, if I'd enable WriteCache and
ReadAhead on the raid groups, I turned them on for all 32 devices and re-ran
bonnie++. To my great dismay, this time zfs had a lot of random troubles with
the drives, where zfs would remove drives arbitrarily from the pool since
they exceeded the error thresholds. On one run, this only happend to 4 drives
from one fc raid on the next run 3 drives from the other raid got removed
from the pool.
Ungood. Note that with this many disks, you should be able to swamp
your fiber channel link and that the fiber channel should be the
sequential I/O bottleneck. It may also be that your RAID array
firmware/CPUs become severely overloaded.
I know, that I'd better disable all "optimizations" on the raid side, but the
performance seems just too bad with these settings. Maybe running 16 mirrors
in a zpool is not a good idea - but that seems more than unlikely to me.
16 mirrors in a zpool is a very good idea. Just keep in mind that
this is a lot of I/O power and you might swamp your FC link and
adaptor card.
Is there anything else I can check?
Check the output of
iostat -xn 30
while bonnie++ is running. This may reveal an issue.
Bob
--
Bob Friesenhahn
bfrie...@simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/
GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
