Some test data for OFED1.1. I am going to run OFED1.2 and IBGOLD.
* 1 Lun per Tier(8+1) with block size of 4096KB
* Using xdd writing to 4 luns.
* Using 1 IB host port to 1 IB DDN Port.
The 700MB/sec appear to be a host limit. Because using 2 IB Host ports and 2
DDN IB ports IO still max out at 700MB/sec
Note that write level off at 512KB Request Size. I did verified IO Request
Lengths on DDN
= WRITE DirectIO==
IO
Size Throughput
KB MB/sec
--- ---------
16 46.711
32 93.293
64 185.098
128 348.522
256 547.118
512 671.227
1024 697.149
2048 680.645
4096 692.067
8192 710.564
= READ DirectIO==
IO
Size Throughput
KB MB/sec
--- ---------
16 54.856
32 104.526
64 191.592
128 312.586
256 462.460
512 471.877
1024 509.806
2048 535.050
4096 543.130
8192 565.176
Martin W. Schlining III wrote:
Wonder why it halves the value? I'll have to try that myself.
If you are using OFED 1.2, you can also load the ib_srp module with an
option to increase the size of the scatter gather lists. The default size is
12 which is way too small. I don't think this option exists for OFED 1.1. In
1.1, you have to modify ib_srp.h and recompile the module ib_srp.o.
modprobe ib_srp srp_sg_tablesize=256
Fiber channel drivers also set the max_sect field in their drivers to 65535
(0xffff) to eliminate any restrictions. Perhaps the same value for SRP will
help?
- Martin
-----Original Message-----
From: MAHMOUD HANAFI [mailto:[EMAIL PROTECTED]
Sent: Friday, June 01, 2007 2:56 PM
To: Martin W. Schlining III
Cc: 'MAHMOUD HANAFI'
Subject: Re: [ofa-general] Need OFED1.1 ib_srp max_hw_sectors_kb help!
I didn't get a answer on the email list but I figured it out.
You can pass "max_sect=xxx" option during initialization of the srp traget.
If you have upgraded to
OFED1.2 you can set a line in /etc/srp_daemon.conf "A max_sect=2096" (you
will need the "A") and then run the srp_daemon.sh. Only thing I have noticed
is what ever you set in the max_sect value it always takes 1/2 of the value.
-Mahmoud
Martin W. Schlining III wrote:
Hello,
Did you ever get an answer to this? I'm a bit curious myself what the
response was.
Regards,
Martin Schlining
Senior Software Engineer
[EMAIL PROTECTED]
-----Original Message-----
From: [EMAIL PROTECTED]
[mailto:[EMAIL PROTECTED] On Behalf Of MAHMOUD
HANAFI
Sent: Tuesday, May 29, 2007 6:58 PM
To: [email protected]
Subject: [ofa-general] Need OFED1.1 ib_srp max_hw_sectors_kb help!
All,
I am using OFED1.1 with CISCO HCA/switch and DDN Storage. I am able to
load and perform IO to the DDN via srp driver. But, the
max_hw_sectors_kb for the device is getting set to 64kb. Any one else
seen this issue? Same host and storage with fiber channel doesn't have
this problem. It set max_hw_sectors_kb correctly to 4096KB.
Thanks,
--
Mahmoud Hanafi
Senior System Administrator
ASC/MSRC
www.asc.hpc.mil
2435 5th Street
WPAFB, OHIO 45433
(937) 255-1536
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
--
Mahmoud Hanafi
Senior System Administrator
ASC/MSRC
www.asc.hpc.mil
2435 5th Street
WPAFB, OHIO 45433
(937) 255-1536
--
Mahmoud Hanafi
Senior System Administrator
ASC/MSRC
www.asc.hpc.mil
2435 5th Street
WPAFB, OHIO 45433
(937) 255-1536
_______________________________________________
general mailing list
[email protected]
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
