:On Mon, Jul 11, 2011 at 12:37:08PM -0700, Matthew Dillon wrote:
:>
:> There's something very basic wrong. How much memory does the machine
:> have?
:
:32GB. I didn't think too much memory could be a liability.
It will mostly be cached, at least until the system hits its vnode
limit and starts throwing things away (which will occur long before
it actually runs out of memory). You might need to set iterations
to 400 or 500 to blow out 32GB worth of cache.
To tune DragonFly's HAMMER to work around the vnode limits just turn on
double buffering. sysctl vfs.hammer.double_buffer=1
:> Otherwise you aren't testing the disk drives at all.
:
:There was definitely some disk activity. And there was more at the start of
:the benchmark than at the end.
:I've kept a log of "iostat da0 1" output. It's as if something was limiting
:the number of iops
It should be nearly all write activity at the beginning, then shift into
read + write activity as the test progresses and either the vnode cache
gets blown out or (for DFly with double buffering turned on) actual ram
gets blown out.
:> Serialized disk reads will show up at around ~150-200 tps. Multi-tagged
:> disk reads will show up in the 400-500 tps range.
:
:If I interpret the results correctly, I get 1000-1300 iops:
:
:randread 1.000s 1287 loops = 777.047uS/loop
:randread 1.000s 1288 loops = 776.438uS/loop
:randread 1.000s 1282 loops = 780.059uS/loop
:randread 1.000s 1314 loops = 761.074uS/loop
:randread 1.000s 1043 loops = 958.813uS/loop
:randread 1.000s 1282 loops = 780.075uS/loop
:randread 1.000s 1326 loops = 754.184uS/loop
:randread 1.000s 1195 loops = 836.863uS/loop
:randread 1.000s 1179 loops = 848.221uS/loop
:
:With 16KB requests, the numbers are in the 1400-1600 tps range:
:
:randread 1.000s 1509 loops = 662.727uS/loop
:randread 1.000s 1531 loops = 653.212uS/loop
:randread 1.000s 1479 loops = 676.168uS/loop
:randread 1.000s 1575 loops = 634.956uS/loop
:randread 1.000s 1418 loops = 705.241uS/loop
:randread 1.000s 1536 loops = 651.066uS/loop
:randread 1.000s 1645 loops = 607.927uS/loop
:randread 1.000s 1472 loops = 679.379uS/loop
:randread 1.000s 1457 loops = 686.371uS/loop
:
:--
:Francois Tigeot
This sounds reasonable for a RAID system. Each disk should be able to
do ~400 iops, but RAID won't scale 1:1 on the disk count. It would be
worth testing without the CAM fix to see if it in fact fixed anything
(then I can commit it).
Turning on double buffering mode should be interesting. You will get
slightly lower initial read performance but it should not drop off
virtually at all until the data set exceeds available ram.
In FreeBSD and DragonFly observe the kern.maxvnodes sysctl value, the
defaults are going to be set differently. They have to be set the same
if you want the test to be relatively compatible (without HAMMER double
buffering turned on, that is).
In a real life situation HAMMER double buffering would probably want to
be turned on in order to prevent the vnode limit from causing
unnecessary I/O.
In anycase, you need to tune the blogbench to blow out the caches and/or
artificially reduce physical memory if you want to test actual disk I/O.
I'm not sure where the concurrency is breaking down on the RAID subsystem,
it could be due to that particular driver still being under the big
giant lock (the AHCI and SILI drivers don't use the bgl, but arcmsr
driver probably still does).
-Matt
Matthew Dillon
<[email protected]>
