ibv_rc_pingpong runs over verbs and not RDMA_CM. As far as I understood, RoCE requires the use of the RDMA CM (in open mpi), and it is not possible to set explicitly the SL in RDMA_CM. We should check if RDMA_CM is must for RoCE.
Thanks, Doron On Wed, Mar 2, 2011 at 8:01 PM, Jeff Squyres <jsquy...@cisco.com> wrote: > Mellanox...? > > On Mar 2, 2011, at 12:51 PM, Michael Shuey wrote: > > > Honestly, I don't know - I haven't looked into the source. OFED 1.5.2 > > has a version of ibv_rc_pingpong that's been modified to work with > > RoCE; you can pass the gid_index and SL as command-line arguments. > > I'm not sure how that's handled at the IB layer, but the source may be > > a good place to start. > > > > -- > > Mike Shuey > > > > > > > > On Tue, Mar 1, 2011 at 9:14 AM, Jeff Squyres <jsquy...@cisco.com> wrote: > >> I thought you mentioned in a prior email that you had gotten one or two > other OFED sample applications to work properly. How are they setting the > SL? Are they not using the RDMA CM? > >> > >> > >> On Mar 1, 2011, at 7:35 AM, Michael Shuey wrote: > >> > >>> So, since RoCE has no SM, and setting an SL is required to get > >>> lossless ethernet on Cisco switches (and possibly others), does this > >>> mean that RoCE will never work correctly with OpenMPI on Cisco > >>> hardware? > >>> > >>> -- > >>> Mike Shuey > >>> > >>> > >>> > >>> On Tue, Mar 1, 2011 at 3:42 AM, Doron Shoham <dor...@mellanox.com> > wrote: > >>>> Hi, > >>>> > >>>> Regarding to using a specific SL with RDMA CM, I've checked in the > code and it seems that RDMA_CM uses the SL from the SA. > >>>> So if you want to configure a specific SL, you need to do it via the > SM. > >>>> > >>>> Doron > >>>> > >>>> -----Original Message----- > >>>> From: Jeff Squyres [mailto:jsquy...@cisco.com] > >>>> Sent: Thursday, February 24, 2011 3:45 PM > >>>> To: Michael Shuey > >>>> Cc: Open MPI Users , Mike Dubman > >>>> Subject: Re: [OMPI users] RoCE (IBoE) & OpenMPI > >>>> > >>>> On Feb 24, 2011, at 8:00 AM, Michael Shuey wrote: > >>>> > >>>>> Late yesterday I did have a chance to test the patch Jeff provided > >>>>> (against 1.4.3 - testing 1.5.x is on the docket for today). While it > >>>>> works, in that I can specify a gid_index, > >>>> > >>>> Great! I'll commit that to the trunk and start the process of moving > it to the v1.5.x series (I know you haven't tested it yet, but it's > essentially the same patch, just slightly adjusted for each of the 3 > branches). > >>>> > >>>>> it doesn't do everything > >>>>> required - my traffic won't match a lossless CoS on the ethernet > >>>>> switch. Specifying a GID is only half of it; I really need to also > >>>>> specify a service level. > >>>> > >>>> RoCE requires the use of the RDMA CM (I think?), and I didn't think > there was a way to request a specific SL via the RDMA CM...? (I could > certainly be wrong here) > >>>> > >>>> I think Mellanox will need to follow up with these questions... > >>>> > >>>> -- > >>>> Jeff Squyres > >>>> jsquy...@cisco.com > >>>> For corporate legal information go to: > >>>> http://www.cisco.com/web/about/doing_business/legal/cri/ > >>>> > >>>> > >>>> _______________________________________________ > >>>> users mailing list > >>>> us...@open-mpi.org > >>>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >>>> > >>> > >>> _______________________________________________ > >>> users mailing list > >>> us...@open-mpi.org > >>> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > >> > >> -- > >> Jeff Squyres > >> jsquy...@cisco.com > >> For corporate legal information go to: > >> http://www.cisco.com/web/about/doing_business/legal/cri/ > >> > >> > >> _______________________________________________ > >> users mailing list > >> us...@open-mpi.org > >> http://www.open-mpi.org/mailman/listinfo.cgi/users > >> > > > -- > Jeff Squyres > jsquy...@cisco.com > For corporate legal information go to: > http://www.cisco.com/web/about/doing_business/legal/cri/ > > > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >