increase the srp_sg_tablesize isnt going to help much. you want to make the rdma segments bigger, and with the default sg tablesize is big enough for large i/o if you get the segment size up. increasing the srp_sg_tablesize leads to a bigger iu message size and some devices have some limitations in this area.
checking with our analyzer here on sles10, i cannot get dd to issue a read with an rdma segment larger than 512k. however: echo 256 > /sys/block/<device>/queue/max_phys_segments echo 256 > /sys/block/<device>/queue/max_hw_segments echo 1 > /proc/scsi/sg/allow_dio sg_dd if=/dev/<device> of=/tmp/file dio=1 bpt=2048 count=10240 blk_sgio=1 does produce 5 scsi reads with a single rdma segment of 1MB. In message <[EMAIL PROTECTED]>,"Batw ara, Ashish" writes: >This is what I did as suggested by Vu and it seems to be working. >However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP as >seen on analyzer. I am just wondering what the logic is? On the other >side, when we increase the srp_sg_tablesize beyond 256, we are seeing >following message in /var/log/messages "Nov 29 21:17:50 p50 kernel: >REJ reason 0x3" which indicates "IB_CM_REJ_NO_RESOURCES", so not sure >how to get around to this problem to send larger IO than 1MB in one >shot. > >=20 > >=20 > >modprobe ib_srp srp_sg_tablesize=3D256 > >echo >id_ext=3D200600A0B81138C9,max_sect=3D4096,ioc_guid=3D00a0b81112da0003,dgi= >d=3Dfe8 >000000000000000a0b81112da0001,pkey=3Dffff,service_id=3D200600a0b81138c9> >/sys/class/infiniband_srp/srp-mthca0-1/add_target > >=20 > >-----Original Message----- >From: chas williams - CONTRACTOR [mailto:[EMAIL PROTECTED] >Sent: Friday, November 30, 2007 11:43 AM >To: Kevin Harms >Cc: Vu Pham; [EMAIL PROTECTED]; Batwara, Ashish >Subject: Re: [ofa-general] IO Size more than 48K=20 > >=20 > >addtionally, you might need to echo 'blocks' > > >/sys/block/<device/queue/max_hw_segments to increase the size of the > >rdma segments. > >=20 > >max_hw_segments doesnt exist on all kernels i think. > >=20 > >In message <[EMAIL PROTECTED]>,Kevin >Harms writ > >es: > >>=20 > >> you may also have to go to /sys/block/sdX/queue and echo 1024 > =20 > >>max_sectors_kb > >> if you use the srp_daemon you can also add: > >> a max_sect=3D2048 to /etc/srp_daemon.conf > >>=20 > >>kevin > >>=20 > >>On Nov 29, 2007, at 11:08 AM, Vu Pham wrote: > >>=20 > >>>=20 > >>>> Hi, > >>>> We are using OFED-1.2, and using xdd and some other tools, and =20 > >>>> trying to > >>>> send 1/2MB IOs, but what we are seeing in analyzer traces, that =20 > >>>> memory > >>>> descriptor in SRP command shows max. 48K which means 1MB I/Os has =20 > >>>> broken > >>>> into smaller SRP request from initiator. > >>>> How can I have this I/O directly going to target? What parameter I =20 > >>>> need > >>>> to change? > >>>>=20 > >>>>=20 > >>>=20 > >>> module param srp_sg_tablesize (default is 12 ie. 12 x 4K =3D 48K) > >>> and/or > >>> max_sect=3Dyyy in echo id_ext=3Dxxx,...,max_sect=3D1024,service_id=3D = >> /sys/ > > >>> class/infiniband_srp/... > >>>=20 > >>> -vu > >>>=20 > >>>> Thanks > >>>> Ashish > >>>> _______________________________________________ > >>>> general mailing list > >>>> [email protected] > >>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >>>>=20 > >>>> To unsubscribe, please visit >http://openib.org/mailman/listinfo/openib-gene > >>ral > >>>>=20 > >>>=20 > >>> _______________________________________________ > >>> general mailing list > >>> [email protected] > >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >>>=20 > >>> To unsubscribe, please visit >http://openib.org/mailman/listinfo/openib-gener > >>al > >>>=20 > >>=20 > >>_______________________________________________ > >>general mailing list > >>[email protected] > >>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general > >>=20 > >>To unsubscribe, please visit >http://openib.org/mailman/listinfo/openib-general > >>=20 > > >------_=_NextPart_001_01C83379.3F66B657 >Content-Type: text/html; > charset="us-ascii" >Content-Transfer-Encoding: quoted-printable > ><html xmlns:o=3D"urn:schemas-microsoft-com:office:office" = >xmlns:w=3D"urn:schemas-microsoft-com:office:word" = >xmlns:st1=3D"urn:schemas-microsoft-com:office:smarttags" = >xmlns=3D"http://www.w3.org/TR/REC-html40";> > ><head> ><META HTTP-EQUIV=3D"Content-Type" CONTENT=3D"text/html; = >charset=3Dus-ascii"> ><meta name=3DGenerator content=3D"Microsoft Word 11 (filtered medium)"> ><o:SmartTagType = >namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" > name=3D"PersonName"/> ><!--[if !mso]> ><style> >st1\:*{behavior:url(#default#ieooui) } ></style> ><![endif]--> ><style> ><!-- > /* Style Definitions */ > p.MsoNormal, li.MsoNormal, div.MsoNormal > {margin:0in; > margin-bottom:.0001pt; > font-size:12.0pt; > font-family:"Times New Roman";} >a:link, span.MsoHyperlink > {color:blue; > text-decoration:underline;} >a:visited, span.MsoHyperlinkFollowed > {color:purple; > text-decoration:underline;} >p.MsoPlainText, li.MsoPlainText, div.MsoPlainText > {margin:0in; > margin-bottom:.0001pt; > font-size:10.0pt; > font-family:"Courier New";} >@page Section1 > {size:8.5in 11.0in; > margin:1.0in 77.95pt 1.0in 77.95pt;} >div.Section1 > {page:Section1;} >--> ></style> > ></head> > ><body lang=3DEN-US link=3Dblue vlink=3Dpurple> > ><div class=3DSection1> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>This is what I did as suggested by Vu and it seems to be = >working. >However, when I send 2MB IO, it gets broken into 512K+1MB+512K by SRP as = >seen >on analyzer. I am just wondering what the logic is? On the other side, = >when we >increase the srp_sg_tablesize beyond 256, we are seeing following = >message in >/var/log/messages “Nov 29 21:17:50 p50 kernel: REJ = >reason 0x3” >which indicates “IB_CM_REJ_NO_RESOURCES”, so not sure how to = >get >around to this problem to send larger IO than 1MB in one = >shot.<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>modprobe ib_srp <b><font color=3Dred><span = >style=3D'color:red;font-weight: >bold'>srp_sg_tablesize=3D256</span></font></b><o:p></o:p></span></font></= >p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>echo id_ext=3D200600A0B81138C9,<b><font color=3Dred><span = >style=3D'color: >red;font-weight:bold'>max_sect=3D4096</span></font></b>,ioc_guid=3D00a0b8= >1112da0003,dgid=3Dfe8000000000000000a0b81112da0001,pkey=3Dffff,service_id= >=3D200600a0b81138c9> >/sys/class/infiniband_srp/srp-mthca0-1/add_target<o:p></o:p></span></font= >></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>-----Original Message-----<br> >From: chas williams - CONTRACTOR [mailto:[EMAIL PROTECTED] <br> >Sent: Friday, November 30, 2007 11:43 AM<br> >To: Kevin Harms<br> >Cc: Vu Pham; [EMAIL PROTECTED]; <st1:PersonName = >w:st=3D"on">Batwara, > Ashish</st1:PersonName><br> >Subject: Re: [ofa-general] IO Size more than 48K </span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>addtionally, you might need to echo 'blocks' = >><o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>/sys/block/<device/queue/max_hw_segments to increase the size = >of the<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>rdma segments.<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>max_hw_segments doesnt exist on all kernels i = >think.<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>In message ><[EMAIL PROTECTED]>,Kevin Harms = >writ<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>es:<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>> you may also have to go to = >/sys/block/sdX/queue and echo 1024 >> <o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>max_sectors_kb<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>> if you use the srp_daemon you can = >also add:<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>> a max_sect=3D2048 to = >/etc/srp_daemon.conf<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>kevin<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>On Nov 29, 2007, at 11:08 AM, Vu Pham = >wrote:<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> Hi,<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> We are using OFED-1.2, and using xdd and some other = >tools, >and <o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> trying to<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> send 1/2MB IOs, but what we are seeing in analyzer = >traces, >that <o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> memory<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> descriptor in SRP command shows max. 48K which = >means 1MB >I/Os has <o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> broken<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> into smaller SRP request from = >initiator.<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> How can I have this I/O directly going to target? = >What >parameter I <o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> need<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> to change?<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> module param srp_sg_tablesize (default is 12 ie. 12 x = >4K =3D >48K)<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> and/or<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> max_sect=3Dyyy in echo = >id_ext=3Dxxx,...,max_sect=3D1024,service_id=3D >> /sys/ <o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> class/infiniband_srp/...<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> -vu<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> Thanks<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> Ashish<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> = >_______________________________________________<o:p></o:p></span></font><= >/p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> general mailing list<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> = >[email protected]<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general<o:p></o:p><= >/span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>> To unsubscribe, please visit >http://openib.org/mailman/listinfo/openib-gene<o:p></o:p></span></font></= >p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>ral<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> = >_______________________________________________<o:p></o:p></span></font><= >/p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> general mailing list<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> = >[email protected]<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> = >http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general<o:p></o:p><= >/span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>> To unsubscribe, please visit = >http://openib.org/mailman/listinfo/openib-gener<o:p></o:p></span></font><= >/p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>al<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>_______________________________________________<o:p></o:p></s= >pan></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>general mailing list<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>[email protected]<o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general= ><o:p></o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>>To unsubscribe, please visit = >http://openib.org/mailman/listinfo/openib-general<o:p></o:p></span></font= >></p> > ><p class=3DMsoPlainText><font size=3D2 face=3D"Courier New"><span = >style=3D'font-size: >10.0pt'>><o:p> </o:p></span></font></p> > ></div> > ></body> > ></html> > >------_=_NextPart_001_01C83379.3F66B657-- > _______________________________________________ general mailing list [email protected] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
