see below...
On Mar 2, 2010, at 12:38 AM, Abdullah Al-Dahlawi wrote:
> Greeting All
>
> I am using Filebench benchmark in an "Interactive mode" to test ZFS
> performance with randomread wordload.
> My Filebench setting & run results are as follwos
> ------------------------------------------------------------------------------------------
> filebench> set $filesize=5g
> filebench> set $dir=/hdd/fs32k
> filebench> set $iosize=32k
> filebench> set $workingset=10m
> filebench> set $function=generic
> filebench> set $filesystem=zfs
> filebench> run 100
> 1062: 106.866: Creating/pre-allocating files and filesets
> 1062: 106.867: File largefile1: mbytes=5120
> 1062: 106.867: Re-using file largefile1.
> 1062: 106.867: Creating file largefile1...
> 1062: 108.612: Preallocated 1 of 1 of file largefile1 in 2 seconds
> 1062: 108.612: waiting for fileset pre-allocation to finish
> 1062: 108.612: Starting 1 rand-read instances
> 1063: 109.617: Starting 1 rand-thread threads
> 1062: 112.627: Running...
> 1062: 213.627: Run took 100 seconds...
> 1062: 213.628: Per-Operation Breakdown
> rand-rate 0ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
> rand-read1 41845ops/s 1307.7mb/s 0.0ms/op 20us/op-cpu
>
> 1062: 213.628:
> IO Summary: 4226337 ops, 41845.0 ops/s, (41845/0 r/w) 1307.7mb/s,
> 21us cpu/op, 0.0ms latency
> 1062: 213.628: Shutting down processes
> ---------------------------------------------------------------------------------------------
> The output looks GREAT so far .... notice the 1307.7 mb/s
>
> **** HOWEVER *****
>
> When I run the SAME workload using Filebench "config file" in batch mode, the
> performance dropped significantly !!!!!!!!
>
> Here is my config file & filebench results.
>
>
> # Copyright 2007 Sun Microsystems, Inc. All rights reserved.
> # Use is subject to license terms.
> #
> # ident "%Z%%M% %I% %E% SMI"
>
> DEFAULTS {
> runtime = 30;
> dir = /hdd/fs32k;
> $statsdir=/export/home/abdullah/bench.stat/woow87;
> stats = /export/home/abdullah/bench.stat;
> filesystem = zfs;
> description = "ZFS-RR-WS-10M";
> }
>
> CONFIG rr32k {
> function = generic;
> personality = randomread;
> filesize = 5g;
> iosize = 32k;
> nthreads = 1;
> workingset=10m;
> }
>
> And the Run result ....
>
> abdul...@hp_hdx_16:/usr/benchmarks/filebench/config# filebench rrws10m
> parsing profile for config: rr32k
> Creating Client Script
> /export/home/abdullah/bench.stat/HP_HDX_16-zfs-rrws10m-Mar_2_2010-03h_10m_46s/rr32k/thisrun.f
> Running
> /export/home/abdullah/bench.stat/HP_HDX_16-zfs-rrws10m-Mar_2_2010-03h_10m_46s/rr32k/thisrun.f
> FileBench Version 1.4.4
> 1147: 0.004: Random Read Version 2.0 IO personality successfully loaded
> 1147: 0.004: Creating/pre-allocating files and filesets
> 1147: 0.005: File largefile1: mbytes=5120
> 1147: 0.005: Re-using file largefile1.
> 1147: 0.005: Creating file largefile1...
> 1147: 1.837: Preallocated 1 of 1 of file largefile1 in 2 seconds
> 1147: 1.837: waiting for fileset pre-allocation to finish
> 1147: 1.837: Running '/usr/benchmarks/filebench/scripts/fs_flush zfs
> /hdd/fs32k'
This step flushes the cache.
> 1147: 1.845: Change dir to
> /export/home/abdullah/bench.stat/HP_HDX_16-zfs-rrws10m-Mar_2_2010-03h_10m_46s/rr32k
> 1147: 1.845: Starting 1 rand-read instances
> 1149: 2.850: Starting 1 rand-thread threads
> 1147: 5.860: Running...
> 1147: 36.159: Run took 30 seconds...
> 1147: 36.160: Per-Operation Breakdown
> rand-rate 0ops/s 0.0mb/s 0.0ms/op 0us/op-cpu
> rand-read1 88ops/s 2.7mb/s 11.4ms/op 35us/op-cpu
This is right on spec for a single drive: seek + rotate = 11.3 ms
-- richard
>
> 1147: 36.160:
> IO Summary: 2660 ops, 87.8 ops/s, (88/0 r/w) 2.7mb/s, 443us
> cpu/op, 11.4ms latency
> 1147: 36.160: Stats dump to file 'stats.rr32k.out'
> 1147: 36.160: in statsdump stats.rr32k.out
> 1147: 36.415: Shutting down processes
> Generating html for
> /export/home/abdullah/bench.stat/HP_HDX_16-zfs-rrws10m-Mar_2_2010-03h_10m_46s
> file =
> /export/home/abdullah/bench.stat/HP_HDX_16-zfs-rrws10m-Mar_2_2010-03h_10m_46s/rr32k/stats.rr32k.out
> ------------------------------------------------------------------------------------------------
>
> The output for the same workload is disappointing , notice that the
> throughput dropped from 1307.7 mb/s to 2.7 mb/s !!!!!!!!!!!!!!!!!!!!!!1
>
> My ARC_max is 3G
>
> Here is a snapshot of my arcstat output in case of high throughput ---
> notice the 100% hits ratio
>
>
> arcsz,read,hits,Hit%,miss,miss%,dhit,dh%,dmis,dm%,phit,ph%,pmis,pm%,mhit,mh%,mmis,mm%,mfug,mrug,
> 1G, 31M, 31M, 99,111K, 0, 28M, 99, 99K, 0, 2M, 99, 12K, 0, 1M, 98,
> 13K, 1, 43, 43,
> 1G,147K,145K, 99, 1K, 0, 14K, 99, 2, 0,131K, 99, 1K, 0, 0, 0,
> 0, 0, 0, 0,
> 1G,166K,166K, 100, 0, 0, 37K,100, 0, 0,128K,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 41K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 41K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 10,100,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 41K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 41K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 42K, 42K, 100, 0, 0, 42K,100, 0, 0, 256,100, 0, 0, 0, 0,
> 0, 0, 0, 0,
>
> and a snapshot in case of low throughput
> notice the low hit ratio !!
> arcsz,read,hits,Hit%,miss,miss%,dhit,dh%,dmis,dm%,phit,ph%,pmis,pm%,mhit,mh%,mmis,mm%,mfug,mrug,
> 1G, 3, 3, 100, 0, 0, 3,100, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 40, 3, 7, 37, 92, 3, 7, 37, 92, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 113, 12, 10, 101, 89, 12, 10, 101, 89, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 105, 14, 13, 91, 86, 14, 13, 91, 86, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 108, 15, 13, 93, 86, 15, 13, 93, 86, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 99, 11, 11, 88, 88, 11, 11, 88, 88, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 103, 11, 10, 92, 89, 11, 10, 92, 89, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 101, 13, 12, 88, 87, 13, 12, 88, 87, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 107, 12, 11, 95, 88, 12, 11, 95, 88, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 99, 12, 12, 87, 87, 12, 12, 87, 87, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 100, 5, 5, 95, 95, 5, 5, 95, 95, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 114, 17, 14, 97, 85, 17, 14, 97, 85, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 106, 17, 16, 89, 83, 17, 16, 89, 83, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 107, 7, 6, 100, 93, 7, 6, 100, 93, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 100, 11, 11, 89, 89, 11, 11, 89, 89, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 99, 8, 8, 91, 91, 8, 8, 91, 91, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 101, 9, 8, 92, 91, 9, 8, 92, 91, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
> 1G, 101, 9, 8, 92, 91, 9, 8, 92, 91, 0, 0, 0, 0, 0, 0,
> 0, 0, 0, 0,
>
>
> Any Feed Back !!!!!!!!!!!!
>
> --
> Abdullah Al-Dahlawi
> PhD Candidate
> George Washington University
> Department. Of Electrical & Computer Engineering
> ----
> Check The Fastest 500 Super Computers Worldwide
> http://www.top500.org/list/2009/11/100
> _______________________________________________
> zfs-discuss mailing list
> [email protected]
> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
ZFS storage and performance consulting at http://www.RichardElling.com
ZFS training on deduplication, NexentaStor, and NAS performance
http://nexenta-atlanta.eventbrite.com (March 16-18, 2010)
_______________________________________________
perf-discuss mailing list
[email protected]
