On Wed, Jan 06, 2010 at 11:59:37PM -0800, Jack Z wrote: > Hi Pasi, > > Thank you very much for your help. I really appreciate it! > > On Jan 5, 12:58 pm, Pasi Kärkkäinen <pa...@iki.fi> wrote: > > On Tue, Jan 05, 2010 at 02:05:03AM -0800, Jack Z wrote: > > > > > > > > Try using some benchmarking tool that can do multiple outstanding IOs.. > > > > for example ltp disktest. > > > > > And I tried ltp disktest, too. But I'm not sure whether I used it > > > right because the result was a little surprising... > > > > > I did > > > > > disktest -w -S0:1k -B 1024 /dev/sdb > > > > > (/dev/sdb is the iSCSI device file, no partition or file system on it) > > > > > And the result was: > > > > > | 2010/01/05-02:58:26 | START | 27293 | v1.4.2 | /dev/sdb | Start > > > args: -w -S0:1024k -B 1024 -PA (-I b) (-N 8385867) (-K 4) (-c) (-p R) > > > (-L 1048577) (-D 0:100) (-t 0:2m) (-o 0) > > > | 2010/01/05-02:58:26 | INFO | 27293 | v1.4.2 | /dev/sdb | Starting > > > pass > > > ^C| 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > > bytes written in 85578 transfers: 87631872 > > > | 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > > write throughput: 701055.0B/s (0.67MB/s), IOPS 684.6/s. > > > | 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > > Write Time: 125 seconds (0d0h2m5s) > > > | 2010/01/05-03:00:58 | STAT | 27293 | v1.4.2 | /dev/sdb | Total > > > overall runtime: 152 seconds (0d0h2m32s) > > > | 2010/01/05-03:00:58 | END | 27293 | v1.4.2 | /dev/sdb | User > > > Interrupt: Test Done (Passed) > > > > > As you can see, the throughput was only 0.67MB/s and only 85578 > > > written in 87631872 transfers... > > > I also tweaked the options with "-p l" and/or "-I bd" (change seek > > > pattern to linear and/or speficy IO type as block and direct IO) but > > > no improvement happened... > > > > Hmm.. so it does 684 IO operations per second (IOPS), and each IO was 1k > > in size, so it makes 684 kB/sec of throughput. > > > > 1000 milliseconds (1 second) divided by 684 IOPS is 1.46 milliseconds per > > IO.. > > > > Are you sure you had 16ms of rtt? > > Actually that was probably the output from 0.2 ms rtt instead of 16 > ms... I'm sorry for the mistake. I tried again the same command on a > 16ms RTT, and the IOPS was mostly around 180. >
1000ms divided by 16ms rtt gives you 62,5 synchronous IOPS max. So that means you had about 3 outstanding IOs running, since you got 180 IOPS. If I'm still following everything correctly :) > > > Try to play and experiment with these options: > > > > -B 64k (blocksize 64k, try also 4k) > > -I BD (block device, direct IO (O_DIRECT)) > > -K 16 (16 threads, aka 16 outstanding IOs. -K 1 should be the same as dd) > > > > Examples: > > > > Sequential (linear) reads using blocksize 4k and 4 simultaneous threads, > > for 60 seconds: > > disktest -B 4k -h 1 -I BD -K 4 -p l -P T -T 60 -r /dev/sdX > > > > Random writes: > > > > disktest -B 4k -h 1 -I BD -K 4 -p r -P T -T 60 -w /dev/sdX > > > > 30% random reads, 70% random writes: > > disktest -r -w -D30:70 -K2 -E32 -B 8k -T 60 -pR -Ibd -PA /dev/md4 > > > > Hopefully that helps.. > > That did help. I tried the following combinations of -B -K and -p at > 20 ms RTT and the other options were -h 30 -I BD -P T -S0:(1 GB size) > > -B 4k/64k -K 4/64 -p l > > It seems that when I put -p l there the performance goes down > drastically... > That's really weird.. linear/sequential (-p l) should always be faster than random. > -B 4k -K 4/64 -p r > > The disk throughput is similar to the one I used in the previous post > "disktest -w -S0:1k -B 1024 /dev/sdb " and it's much lower than dd > could get. > like said, weird. > -B 64k -K 4 -p r > > The disk throughput is higher than the last one but still not as high > as dd could get. > > -B 64k -K 64 -p r > > The disk throughput was boosted to 8.06 MB/s and the IOPS was 129.0. > At the link layer, the traffic rate was 70.536 Mbps (the TCP baseline > was 96.202 Mbps). At the same time, dd ( bs=64K count=(1 GB size)) got > a throughput of 6.7 MB/s and the traffic rate on the link layer was > 57.749 Mbps. > Ok. 129 IOPS * 64kB = 8256 kB/sec, which pretty much matches the 8 MB/sec you measured. this still means there was only 1 outstanding IO.. and definitely not 64 (-K 64). > Although not much, it was still an improvement and it was the first > improvement I have ever seen since I started my experiments! Thank you > very much! > > As for > > > Oh, also make sure you have 'oflag=direct' for dd. > > The result was surprisingly low again... Do you think the reason might > be that I was running dd on a device file (/dev/sdb), which did not > have any partitions/file systems on it? > > Thanks a lot! > oflag=direct makes dd use O_DIRECT, aka bypass all kernel/initiator caches for writing. iflag=direct would bypass all caches for reading. It shouldn't matter if you write or read from /dev/sda1 instead of /dev/sda. As long as it's a raw block device, it shouldn't matter. If you write/read to/from a filesystem, that obviously matters. What kind of target you are using for this benchmark? -- Pasi
-- You received this message because you are subscribed to the Google Groups "open-iscsi" group. To post to this group, send email to open-is...@googlegroups.com. To unsubscribe from this group, send email to open-iscsi+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/open-iscsi?hl=en.
