[zfs-discuss] Help understanding some benchmark results

2007-01-11 Thread Chris Smith 
G'day, all,

So, I've decided to migrate my home server from Linux+swRAID+LVM to 
Solaris+ZFS, because it seems to hold much better promise for data integrity, 
which is my primary concern.

However, naturally, I decided to do some benchmarks in the process, and I don't 
understand why the results are what they are.  I though I had a reasonable 
understanding of ZFS, but now I'm not so sure.

I've used bonnie++ and a variety of Linux RAID configs below to approximate 
equivalent ZFS configurations and compare.  I do realise they're not exactly 
the same thing, but it seems to me they're reasonable comparisons and should 
return at least somewhat similar performance.  I also realise bonnie++ is not 
an especially comprehensive or complex benchmark, but ultimately I don't really 
care about performance and this was only done out of curiosity.


The executive summary is that ZFS write performance appears to be relatively 
awful (all the time), and it's read performance is relatively good most of the 
time (with "striping", "mirroring" and raidz[2]'s with fewer numbers of disks).


Examples:
* 8-disk RAID0 on Linux returns about 190MB/s write and 245MB/sec read, while a 
ZFS raidz using the same disks returns about 120MB/sec write, but 420MB/sec 
read.
* 16-disk RAID10 on Linux returns 165MB/sec and 440MB/sec write and read, while 
a ZFS pool with 8 mirrored disks returns 140MB/sec write and 410MB/sec read.
* 16-disk RAID6 on Linux returns 126MB/sec write, 162MB/sec read, while a 
16-disk raidz2 returns 80MB/sec write and 142MB/sec read.

The biggest problem I am having understanding "why is it so", is because I was 
under the impression with ZFS's CoW, etc, that writing (*especially* writes 
like this, to a raidz array) should be much faster than a regular old-fashioned 
RAID6 array.

I certainly can't complain about the read speed, however - 400-odd MB/sec out 
of this old beastie is pretty impressive :).

Help ?  Have I missed something obvious or done something silly ?
(Additionally, from the Linux perspective, why are reads so slow ?)


PS: Hardware specs are
Dual 2.8Ghz Xeon, 533Mhz bus, 2G RAM.
Two SuperMicro 8-port SATA controllers - AOC-SAT2-MV8 - on separate PCI-X buses
(Marvell  88SX6081 based, 
http://www.supermicro.com/products/accessories/addon/AoC-SAT2-MV8.cfm)
16 Seagate 250G drives, 8M cache, ST3250820AS
Motherboard is a pull from an old Gateway dual Xeon server.


PPS: Benchmark data (this will look awful with a proportional font):
(With the Linux "RAID50" and "RAID60", I created two RAID5 and RAID6 arrays, 
respectively, then create a RAID0 array over the top of them.  For "RAID10" I 
used the 'builtin' RAID10.)

Single drive
bonnie++ -d /mnt/ -s 8g -u0 -f
Version  1.03   --Sequential Output-- --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G   42807  21 25278   7   65191   6 118.4   0
--Sequential Create-- Random Create
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 + +++ + +++ + +++ + +++ + +++ + +++
csmith-desktop,8G,,,42807,21,25278,7,,,65191,6,118.4,0,16,+,+++,+,+++,+,+++,+,+++,+,+++,+,+++

zpool create internal c0t0d0
./bonnie++ -d /internal/ -s 8g -u0 -f

Version  1.03   --Sequential Output-- --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
nitrogen 8G   46916  18 27814  11   62103   8 212.5   0
--Sequential Create-- Random Create
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP /sec %CP
 16 15522  99 + +++ 24489  99 10140  99 + +++ 5196  99
nitrogen,8G,,,46916,18,27814,11,,,62103,8,212.5,0,16,15522,99,+,+++,24489,99,10140,99,+,+++,5196,99



8 disk RAID0, one controller, 128k stripe
Version  1.03   --Sequential Output-- --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
csmith-desktop   8G   190853  95 119364  42   245770  28 421.4  
 1
--Sequential Create-- Random Create
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
  files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
 16 + +++ + +++ + +++ + +++ + +++ + +++
csmith-desktop,8G,

Re: [zfs-discuss] Help understanding some benchmark results

2007-01-11 Thread Bart Smaalders 

Chris Smith wrote:

G'day, all,

So, I've decided to migrate my home server from Linux+swRAID+LVM to 
Solaris+ZFS, because it seems to hold much better promise for data integrity, 
which is my primary concern.

However, naturally, I decided to do some benchmarks in the process, and I don't 
understand why the results are what they are.  I though I had a reasonable 
understanding of ZFS, but now I'm not so sure.

I've used bonnie++ and a variety of Linux RAID configs below to approximate 
equivalent ZFS configurations and compare.  I do realise they're not exactly 
the same thing, but it seems to me they're reasonable comparisons and should 
return at least somewhat similar performance.  I also realise bonnie++ is not 
an especially comprehensive or complex benchmark, but ultimately I don't really 
care about performance and this was only done out of curiosity.


The executive summary is that ZFS write performance appears to be relatively awful (all the time), 
and it's read performance is relatively good most of the time (with "striping", 
"mirroring" and raidz[2]'s with fewer numbers of disks).


Examples:
* 8-disk RAID0 on Linux returns about 190MB/s write and 245MB/sec read, while a 
ZFS raidz using the same disks returns about 120MB/sec write, but 420MB/sec 
read.
* 16-disk RAID10 on Linux returns 165MB/sec and 440MB/sec write and read, while 
a ZFS pool with 8 mirrored disks returns 140MB/sec write and 410MB/sec read.
* 16-disk RAID6 on Linux returns 126MB/sec write, 162MB/sec read, while a 
16-disk raidz2 returns 80MB/sec write and 142MB/sec read.

The biggest problem I am having understanding "why is it so", is because I was 
under the impression with ZFS's CoW, etc, that writing (*especially* writes like this, to 
a raidz array) should be much faster than a regular old-fashioned RAID6 array.

I certainly can't complain about the read speed, however - 400-odd MB/sec out 
of this old beastie is pretty impressive :).

Help ?  Have I missed something obvious or done something silly ?
(Additionally, from the Linux perspective, why are reads so slow ?)




What build/version of Solaris/ZFS are you using?

What block size are you using for writes in bonnie++?  I
find performance on streaming writes is better w/ larger writes.

What happens when you run two threads at once?  Does write performance
improve?

Does zpool iostat -v 1 report anything interesting during the benchmark?
What about iostat -x 1?  Is one disk significantly more busy than the
others?

I have a 4x 500GB disk raidz config w/ a 2.6 GHz dual core at home;
this config sustains approx 120 MB/sec on reads and writes on single
or multiple streams.  I'm running build 55; the box has a SI controller
running in PATA compat. mode.

One of the challenging aspects of performance work on these sorts of
things is separating out drivers vs cpus vs memory bandwidth vs disk
behavior vs intrinsic FS behavior.


- Bart



--
Bart Smaalders  Solaris Kernel Performance
[EMAIL PROTECTED]   http://blogs.sun.com/barts
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss