RE: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
Richard, All, > Has anyone else done any evaluation / testing with RDMAoE / RoCEE ? We are testing the RDMAoE version of OFED all the time (mainly on IB but also on Eth.) and currently we did not fined stability issue related only to RDMAoE in our IB testing (all opened bugs/issues are the same as in OFED that does not have RDMAoE patches). After saying the above I will still prefer to see a lot other that will test it so we can have fast turn-around with OFED 1.5.1 including RDMAoE... our IB install-base is big and we do not want to risk it with stability, quality and future compatibility issues. As Or, and other stated, we are still see missing (or not so good) support in RDMAoE for important features like multicast traffic, VLAN, QoS, ... I'm sure Mellanox are working to provide/fix them. Regards, Yiftah > -Original Message- > From: ewg-boun...@lists.openfabrics.org [mailto:ewg- > boun...@lists.openfabrics.org] On Behalf Of Richard Frank > Sent: Monday, November 23, 2009 4:59 PM > To: Jeff Squyres > Cc: o...@lists.openfabrics.org; Roland Dreier (rdreier); OpenFabrics EWG; > Liran Liss > Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > > Is this code new ? We've been evaluating versions of it since before > June/2009. > > We are currently testing with OFED-RDMAoE-1.5-20091116-0620.tgz. > > Our plans are to move from OFED 1.4.2 to OFED 1.5.x in June/2010.. > > It takes us this long to complete internal testing. > > Has anyone else done any evaluation / testing with RDMAoE / RoCEE ? > > > Jeff Squyres wrote: > > FWIW: the dealbreaker for me is that we're already at 1.5rc2. By > > OFED's own rules, new features are not to be allowed. Or you can > > reset the release clock and target Jan/Feb. > > > > Mellanox already has their own OFED distribution -- since there > > appears to be strong desire to get this stuff released ASAP, is there > > an issue with releasing it through Mellanox OFED. Then later release > > it through community OFED in the next go-round? > > > > > > > > On Nov 23, 2009, at 4:18 AM, Liran Liss wrote: > > > >> In the past few months of review, the responsibility for rdmaoe > >> addressing was moved to the rdmacm. > >> So, any future addressing enhancements can be confined to the rdmacm > >> module without breaking existing APIs. > >> > >> RFC 3041 deals with static global IP addresses on the Internet, > >> especially for portable devices. > >> rmdaoe allows using link-local GIDs for applications residing on the > >> same subnet, so I don't see the relevance. > >> Note that for rdmacm apps, the intention is to map the IP addresses > that > >> were assigned to the host's interfaces. > >> Please see http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf. > >> > >> Regarding multicast, current switches will flood the traffic just as > any > >> other non-IP multicast traffic (e.g., fcoe). > >> Using switches that support multicast pruning for additional > ethertypes, > >> you can optimize the traffic and achieve the same link utilization as > >> normal IP multicast. > >> In any case, this is not a correctness issue that prohibits > >> experimentation with rdmaoe multicast on any network today. > >> --Liran > >> > >> > >> -Original Message- > >> From: ewg-boun...@lists.openfabrics.org > >> [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier > >> Sent: Thursday, November 19, 2009 9:35 PM > >> To: Richard Frank > >> Cc: o...@lists.openfabrics.org; OpenFabrics EWG > >> Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > >> > >> > >> > Having lots of testing exposure can help in validating that all the > >> > edge cases are handled.. > >> > >> To some extent -- but there also needs to be some thinking involved to > >> make sure that the interface can actually handle future cases. > >> > >> > Are there a set of cases that you have in mind ? > >> > >> For example -- how is multicast going to interact with IGMP on ethernet > >> switches? How is address resolution going to be done (current patches > >> seem to assume that stateless IPv6 link-local addresses contain the > >> ethernet address, which is not valid if RFC 3041 is used)? etc > >> > >> - R. > >> ___ > >> ewg mailing list > >> ewg@lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > >> ___ > >> ewg mailing list > >> ewg@lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > >> > > > > > ___ > ewg mailing list > ewg@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] [GIT PULL ofed-1.5] - cxgb3 bug fix
Vlad, Please pull from: ssh://v...@sofa.openfabrics.org/~swise/scm/ofed_kernel.git ofed_1_5 This pulls in an important upstream bug fix and adjusts the backport files accordingly. Thanks, Steve. --- [sw...@build linux-2.6]$ git show --stat c5dfd17460415cfd982821c097c1a3385e65e36e commit c5dfd17460415cfd982821c097c1a3385e65e36e Author: Steve Wise Date: Mon Nov 23 14:41:00 2009 -0600 cxgb3: pull in page unmap fix. Signed-off-by: Steve Wise .../2.6.16_sles10_sp2/cxgb3_0100_remove_lro.patch | 30 ++-- .../2.6.16_sles10_sp3/cxgb3_0100_remove_lro.patch | 30 ++-- .../2.6.18-EL5.2/cxgb3_0100_remove_lro.patch | 30 ++-- .../2.6.18-EL5.3/cxgb3_0100_remove_lro.patch | 30 ++-- .../2.6.18-EL5.4/cxgb3_0100_remove_lro.patch | 30 ++-- .../backport/2.6.18/cxgb3_0100_remove_lro.patch| 30 ++-- .../backport/2.6.19/cxgb3_0100_remove_lro.patch| 30 ++-- .../backport/2.6.20/cxgb3_0100_remove_lro.patch| 30 ++-- .../backport/2.6.21/cxgb3_0100_remove_lro.patch| 30 ++-- .../backport/2.6.22/cxgb3_0100_remove_lro.patch| 30 ++-- .../backport/2.6.23/cxgb3_0100_remove_lro.patch| 30 ++-- .../backport/2.6.24/cxgb3_0100_remove_lro.patch| 28 +- .../backport/2.6.25/cxgb3_0100_remove_lro.patch| 28 +- .../backport/2.6.26/cxgb3_0100_remove_lro.patch| 28 +- .../backport/2.6.27/cxgb3_0100_remove_lro.patch| 28 +- .../2.6.27_sles11/cxgb3_0100_remove_lro.patch | 28 +- .../cxgb3_0100_remove_lro.patch| 28 +- .../backport/2.6.28/cxgb3_0100_remove_lro.patch| 28 +- .../backport/2.6.9_U6/cxgb3_0100_remove_lro.patch | 30 ++-- .../backport/2.6.9_U7/cxgb3_0100_remove_lro.patch | 30 ++-- .../backport/2.6.9_U8/cxgb3_0100_remove_lro.patch | 30 ++-- .../fixes/cxgb3_00600_fixprematurepageunmap.patch | 28 ++ 22 files changed, 336 insertions(+), 308 deletions(-) [sw...@build linux-2.6]$ ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ofw] Re: [ewg] RE: SC'09 BOF - Meeting notes and Final Slides
> Why should clients have to emulate asynchronous behavior when the > (Mellanox) HCA command interface is already asynchronous? That's one HCA command model... others may not be (eg I believe ehca makes synchronous hypercalls to do things like modify QP, and ipath is pure software with no other asynchronous entity to talk to). > I would prefer any threading be added to the HCA driver in the > kernel, and not as threads in each process. That would enable a > seamless transition to a better driver model in the future. Of course Windows can (and should!) design the best API for Windows; however for Linux I would probably argue in favor of conservatism (follow the model we've had for years when it's not totally broken) and simplicity (it's hard for developers to get callback-based async APIs right, and I'd rather not have an API that tries to allow both async and sync ways of doing the same thing). - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
> In any case, this is not a correctness issue that prohibits > experimentation with rdmaoe multicast on any network today. I agree -- nothing prevents experimentation. I am just leery about making invasive changes to the core stack in the absence of any documented design for IBoE (that I've seen at least). - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
> RFC 3041 deals with static global IP addresses on the Internet, > especially for portable devices. > rmdaoe allows using link-local GIDs for applications residing on the > same subnet, so I don't see the relevance. I guess you're right -- I was confused about when random addresses are used for generating stateless autoconfig addresses, and I guess even with RFC3041 they are not for link-local scope. However, do you know of anything in the IPv6 RFCs that guarantees that link-local IPv6 addresses are generated using ethernet addresses? - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] EWG meeting -unable to join
On Mon, Nov 23, 2009 at 09:06:38AM -0800, Pradeep Satyanarayana wrote: > > I am unable to join the meeting. Here is what I have from an earlier > invite. I no longer have invites for the last 3 weeks or so. Meeting Number: 209 862 690 > > > > > > > > > > Meeting status: > Not started > Starting date: > Monday, October > 19, 2009 > Starting time: > 9:00 am,Pacific > Standard Time > (San Francisco, > GMT-08:00) > Duration: > 1 hour > Meeting number: > 200 791 042 > Meeting password: > OFED > Audio conference: > To receive a > call back, > provide your > phone number > when you join > the meeting, or > call the number > below and enter > the access > code. > > > > Call-in > toll-free > number > (US/Canada): > +1-866-43 > 2-9903 > Call-in toll > number > (US/Canada): > +1-408-52 > 5-6800 > > > > > Show all global > call-in numbers > > Show toll-free > dialing > restrictions > > > > Access > code: 200 791 > 042
[ewg] EWG meeting -unable to join
I am unable to join the meeting. Here is what I have from an earlier invite. I no longer have invites for the last 3 weeks or so. Meeting status: Not started Starting date: Monday, October 19, 2009 Starting time: 9:00 am,Pacific Standard Time (San Francisco, GMT-08:00) Duration: 1 hour Meeting number: 200 791 042 Meeting password: OFED Audio conference: To receive a call back, provide your phone number when you join the meeting, or call the number below and enter the access code. Call-in toll-free number (US/Canada): +1-866-43 2-9903 Call-in toll number (US/Canada): +1-408-52 5-6800 Show all global call-in numbers Show toll-free dialing restrictions Access code: 200 791 042 Host's name: Jeff Squyres Host's Email:
RE: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
See below. --Liran I understand that this is your assessment of the situation, looking on the series present at the ofed1.5 rdmaoe branch in a black box manner yields that many many files are touched, see below. Coming and saying that changes in your HW LL driver are out of the scope for other companies to discuss is not acceptable, since we provide enterprise ready stack based on your HW driver. LL: Any comments on our low-level driver are more than welcome. That being said, we have been running extensive testing on this code base for several months now and see no stability issues. all the rdmaoe materials saying the lossless traffic class is a must, are you saying that this works well also without it? then why from architect point of view you have posed this requirement? LL: lossless traffic can be achieved today using global pause, for example. PFC is still important; we will submit initial patches that support it next week. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] EWG meeting agenda for today (Nov 23)
Agenda for the EWG meeting today 1. RDMAoE and OFED 1.5 2. Decide on RC3 date - I suggest Wed (Nov 25) 3. Bugs status review - see attachment Note: we have 5 blocker, 8 critical and 13 major bugs. 4. Open discussion Tziporet bugs-2009-11-23.csv Description: bugs-2009-11-23.csv ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
We at OSU have done testing of MVAPICH2 1.4 against the OFED-RDMAoE branch mentioned below. Everything works well. In fact, we made a formal release of MVAPICH2 1.4 with RDMAoE support last month. Thanks, DK > Is this code new ? We've been evaluating versions of it since before > June/2009. > > We are currently testing with OFED-RDMAoE-1.5-20091116-0620.tgz. > > Our plans are to move from OFED 1.4.2 to OFED 1.5.x in June/2010.. > > It takes us this long to complete internal testing. > > Has anyone else done any evaluation / testing with RDMAoE / RoCEE ? > > > Jeff Squyres wrote: > > FWIW: the dealbreaker for me is that we're already at 1.5rc2. By > > OFED's own rules, new features are not to be allowed. Or you can > > reset the release clock and target Jan/Feb. > > > > Mellanox already has their own OFED distribution -- since there > > appears to be strong desire to get this stuff released ASAP, is there > > an issue with releasing it through Mellanox OFED. Then later release > > it through community OFED in the next go-round? > > > > > > > > On Nov 23, 2009, at 4:18 AM, Liran Liss wrote: > > > >> In the past few months of review, the responsibility for rdmaoe > >> addressing was moved to the rdmacm. > >> So, any future addressing enhancements can be confined to the rdmacm > >> module without breaking existing APIs. > >> > >> RFC 3041 deals with static global IP addresses on the Internet, > >> especially for portable devices. > >> rmdaoe allows using link-local GIDs for applications residing on the > >> same subnet, so I don't see the relevance. > >> Note that for rdmacm apps, the intention is to map the IP addresses that > >> were assigned to the host's interfaces. > >> Please see http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf. > >> > >> Regarding multicast, current switches will flood the traffic just as any > >> other non-IP multicast traffic (e.g., fcoe). > >> Using switches that support multicast pruning for additional ethertypes, > >> you can optimize the traffic and achieve the same link utilization as > >> normal IP multicast. > >> In any case, this is not a correctness issue that prohibits > >> experimentation with rdmaoe multicast on any network today. > >> --Liran > >> > >> > >> -Original Message- > >> From: ewg-boun...@lists.openfabrics.org > >> [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier > >> Sent: Thursday, November 19, 2009 9:35 PM > >> To: Richard Frank > >> Cc: o...@lists.openfabrics.org; OpenFabrics EWG > >> Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > >> > >> > >> > Having lots of testing exposure can help in validating that all the > >> > edge cases are handled.. > >> > >> To some extent -- but there also needs to be some thinking involved to > >> make sure that the interface can actually handle future cases. > >> > >> > Are there a set of cases that you have in mind ? > >> > >> For example -- how is multicast going to interact with IGMP on ethernet > >> switches? How is address resolution going to be done (current patches > >> seem to assume that stateless IPv6 link-local addresses contain the > >> ethernet address, which is not valid if RFC 3041 is used)? etc > >> > >> - R. > >> ___ > >> ewg mailing list > >> ewg@lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > >> ___ > >> ewg mailing list > >> ewg@lists.openfabrics.org > >> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > >> > > > > > ___ > ewg mailing list > ewg@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg > ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [PATCH 7/9] rdma/cm: fix loopback address support
ewg-boun...@lists.openfabrics.org wrote on 11/22/2009 02:36:32 AM: > Pradeep Satyanarayana wrote: > > Roland Dreier wrote: > > > >> Thanks... in any case I applied all 9 of the patches in this series. > >> Thanks for pulling all this together. > >> > > Sean, Thanks a lot for pulling it all together. Can we consider > including this > > into OFED-1.5 too? > > > > Pradeep > > > > > > > Pradeep > If someone will send a patch to Vlad we can add it to OFED 1.5 Tziporet, Sure, we will do it. Thanks Pradeep prad...@us.ibm.com___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
Is this code new ? We've been evaluating versions of it since before June/2009. We are currently testing with OFED-RDMAoE-1.5-20091116-0620.tgz. Our plans are to move from OFED 1.4.2 to OFED 1.5.x in June/2010.. It takes us this long to complete internal testing. Has anyone else done any evaluation / testing with RDMAoE / RoCEE ? Jeff Squyres wrote: FWIW: the dealbreaker for me is that we're already at 1.5rc2. By OFED's own rules, new features are not to be allowed. Or you can reset the release clock and target Jan/Feb. Mellanox already has their own OFED distribution -- since there appears to be strong desire to get this stuff released ASAP, is there an issue with releasing it through Mellanox OFED. Then later release it through community OFED in the next go-round? On Nov 23, 2009, at 4:18 AM, Liran Liss wrote: In the past few months of review, the responsibility for rdmaoe addressing was moved to the rdmacm. So, any future addressing enhancements can be confined to the rdmacm module without breaking existing APIs. RFC 3041 deals with static global IP addresses on the Internet, especially for portable devices. rmdaoe allows using link-local GIDs for applications residing on the same subnet, so I don't see the relevance. Note that for rdmacm apps, the intention is to map the IP addresses that were assigned to the host's interfaces. Please see http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf. Regarding multicast, current switches will flood the traffic just as any other non-IP multicast traffic (e.g., fcoe). Using switches that support multicast pruning for additional ethertypes, you can optimize the traffic and achieve the same link utilization as normal IP multicast. In any case, this is not a correctness issue that prohibits experimentation with rdmaoe multicast on any network today. --Liran -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier Sent: Thursday, November 19, 2009 9:35 PM To: Richard Frank Cc: o...@lists.openfabrics.org; OpenFabrics EWG Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > Having lots of testing exposure can help in validating that all the > edge cases are handled.. To some extent -- but there also needs to be some thinking involved to make sure that the interface can actually handle future cases. > Are there a set of cases that you have in mind ? For example -- how is multicast going to interact with IGMP on ethernet switches? How is address resolution going to be done (current patches seem to assume that stateless IPv6 link-local addresses contain the ethernet address, which is not valid if RFC 3041 is used)? etc - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
Is this code new ? We've been evaluating versions of it since before June/2009. We are currently testing with OFED-RDMAoE-1.5-20091116-0620.tgz. Our plans are to move from OFED 1.4.2 to OFED 1.5.x in June/2010.. It takes us this long to complete internal testing. Has anyone else done any evaluation / testing with RDMAoE / RoCEE ? Jeff Squyres wrote: FWIW: the dealbreaker for me is that we're already at 1.5rc2. By OFED's own rules, new features are not to be allowed. Or you can reset the release clock and target Jan/Feb. Mellanox already has their own OFED distribution -- since there appears to be strong desire to get this stuff released ASAP, is there an issue with releasing it through Mellanox OFED. Then later release it through community OFED in the next go-round? On Nov 23, 2009, at 4:18 AM, Liran Liss wrote: In the past few months of review, the responsibility for rdmaoe addressing was moved to the rdmacm. So, any future addressing enhancements can be confined to the rdmacm module without breaking existing APIs. RFC 3041 deals with static global IP addresses on the Internet, especially for portable devices. rmdaoe allows using link-local GIDs for applications residing on the same subnet, so I don't see the relevance. Note that for rdmacm apps, the intention is to map the IP addresses that were assigned to the host's interfaces. Please see http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf. Regarding multicast, current switches will flood the traffic just as any other non-IP multicast traffic (e.g., fcoe). Using switches that support multicast pruning for additional ethertypes, you can optimize the traffic and achieve the same link utilization as normal IP multicast. In any case, this is not a correctness issue that prohibits experimentation with rdmaoe multicast on any network today. --Liran -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier Sent: Thursday, November 19, 2009 9:35 PM To: Richard Frank Cc: o...@lists.openfabrics.org; OpenFabrics EWG Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > Having lots of testing exposure can help in validating that all the > edge cases are handled.. To some extent -- but there also needs to be some thinking involved to make sure that the interface can actually handle future cases. > Are there a set of cases that you have in mind ? For example -- how is multicast going to interact with IGMP on ethernet switches? How is address resolution going to be done (current patches seem to assume that stateless IPv6 link-local addresses contain the ethernet address, which is not valid if RFC 3041 is used)? etc - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
Liran Liss wrote: > The patches don't change the logic of existing flows at all, so we are > not risking *anything* in terms of the stability of the current stack. I understand that this is your assessment of the situation, looking on the series present at the ofed1.5 rdmaoe branch in a black box manner yields that many many files are touched, see below. Coming and saying that changes in your HW LL driver are out of the scope for other companies to discuss is not acceptable, since we provide enterprise ready stack based on your HW driver. > As for vlan id and priorities - we are fully aware to the importance of > exposing vlan ids and priorities to the user, but thanks for pointing this > out. sure, I am saying this since my first look on the patches, couple of months ago, good that someone listens now. > There are deployments today that work fine with the current patches all the rdmaoe materials saying the lossless traffic class is a must, are you saying that this works well also without it? then why from architect point of view you have posed this requirement? Or. $ cat rdmaoe_patches core_0300_refine_device_personality.patch core_0310-Add-RDMAoE-transport-protocol.patch core_0320_rdmaoe_support_qp1.patch core_0330_umad-Enable-support-only-for-IB-ports.patch core_0340_Enable-CM-support-for-RDMAoE.patch core_0350-CMA-device-binding.patch core_0360_RDMAoE-UD-packet-packing-support.patch core_0370-support-RDMAoE-from-userspace.patch core_0380_mcast-support-to-rdmaoe.patch core_0390_cma-move-netdev-mac.patch mlx4_2000_RDMAoE-address-resolution.patch mlx4_2010_RDMAoE-support-allow-interfaces-to-correspond.patch mlx4_2020_handle_mcast_mac.patch mlx4_2030_fix_port_num.patch mlx4_2040_Fix-multicast-handling.patch xxx_rdmaoe_port_notice.patch $ cat rdmaoe_patches | xargs diffstat b/drivers/infiniband/core/cm.c | 25 - b/drivers/infiniband/core/cma.c | 54 +- b/drivers/infiniband/core/mad.c | 41 +- b/drivers/infiniband/core/multicast.c |4 b/drivers/infiniband/core/sa_query.c| 39 +- b/drivers/infiniband/core/ucm.c |8 b/drivers/infiniband/core/ucma.c|2 b/drivers/infiniband/core/user_mad.c|6 b/drivers/infiniband/core/verbs.c | 25 + b/drivers/infiniband/hw/mlx4/main.c | 56 ++ b/drivers/infiniband/ulp/ipoib/ipoib_main.c | 12 b/include/rdma/ib_addr.h| 93 b/include/rdma/ib_verbs.h | 11 b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c |3 b/net/sunrpc/xprtrdma/svc_rdma_transport.c |3 drivers/infiniband/core/cm.c|5 drivers/infiniband/core/cma.c | 254 +++-- drivers/infiniband/core/ucm.c | 13 drivers/infiniband/core/ucma.c | 31 + drivers/infiniband/core/user_mad.c | 16 drivers/infiniband/hw/mlx4/ah.c | 22 - drivers/infiniband/hw/mlx4/main.c | 110 - drivers/infiniband/hw/mlx4/mlx4_ib.h| 13 drivers/infiniband/hw/mlx4/qp.c | 29 + include/rdma/ib_verbs.h |4 ofed_kernel-fixes/drivers/infiniband/core/agent.c | 39 +- ofed_kernel-fixes/drivers/infiniband/core/local_sa.c| 22 - ofed_kernel-fixes/drivers/infiniband/core/mad.c | 45 +- ofed_kernel-fixes/drivers/infiniband/core/notice.c |4 ofed_kernel-fixes/drivers/infiniband/core/ud_header.c | 111 + ofed_kernel-fixes/drivers/infiniband/core/uverbs.h |1 ofed_kernel-fixes/drivers/infiniband/core/uverbs_cmd.c | 32 + ofed_kernel-fixes/drivers/infiniband/core/uverbs_main.c |1 ofed_kernel-fixes/drivers/infiniband/core/verbs.c |9 ofed_kernel-fixes/drivers/infiniband/hw/mlx4/ah.c | 187 +++-- ofed_kernel-fixes/drivers/infiniband/hw/mlx4/mad.c | 32 + ofed_kernel-fixes/drivers/infiniband/hw/mlx4/main.c | 309 ++-- ofed_kernel-fixes/drivers/infiniband/hw/mlx4/mlx4_ib.h | 19 ofed_kernel-fixes/drivers/infiniband/hw/mlx4/qp.c | 169 ++-- ofed_kernel-fixes/drivers/net/mlx4/en_main.c| 15 ofed_kernel-fixes/drivers/net/mlx4/en_port.c|4 ofed_kernel-fixes/drivers/net/mlx4/en_port.h|3 ofed_kernel-fixes/drivers/net/mlx4/fw.c |3 ofed_kernel-fixes/drivers/net/mlx4/intf.c | 20 + ofed_kernel-fixes/drivers/net/mlx4/main.c |6 ofed_kernel-fixes/drivers/net/mlx4/mlx4.h |1 ofed_kernel-fixes/include/linux/mlx4/cmd.h |1 ofed_kernel-fixes/include/linux/mlx4/d
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
Jeff Squyres wrote: FWIW: the dealbreaker for me is that we're already at 1.5rc2. By OFED's own rules, new features are not to be allowed. Or you can reset the release clock and target Jan/Feb. Mellanox already has their own OFED distribution -- since there appears to be strong desire to get this stuff released ASAP, is there an issue with releasing it through Mellanox OFED. Then later release it through community OFED in the next go-round? We will discuss this in the meeting today Tziporet ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED with MPE
On Mon, Nov 23, 2009 at 12:27:23PM +0100, Stefan Kuhne wrote: > Jonathan Perkins schrieb: > > > Sorry for the late reply. You should be able to use mpirun_rsh in order > > to avoid setting up mpd. > > It works fine. > But i have to give -np. > > Back to Topic: > When i try to compile an MPE Code i get: > > u...@head:~$ mpicc mandelbrot_mpe.c -o Apfel > /tmp/ccwpw6so.o: In function `main': > mandelbrot_mpe.c:72: undefined reference to `MPE_Open_graphics' > mandelbrot_mpe.c:74: undefined reference to `MPE_Make_color_array' > ... > collect2: ld gab 1 als Ende-Status zurück > u...@head:~$ It looks like the X11 development headers weren't installed at the time the mvapich2 rpm was built. In this case certain parts of MPE may silently fail to build. Can you try building the mvapich2 tarball (http://mvapich.cse.ohio-state.edu/download/mvapich2/mvapich2-1.4.tgz) and see if you can get any more diagnostics during the build process? -- Jonathan Perkins http://www.cse.ohio-state.edu/~perkinjo pgps0GlWQDlEq.pgp Description: PGP signature ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
FWIW: the dealbreaker for me is that we're already at 1.5rc2. By OFED's own rules, new features are not to be allowed. Or you can reset the release clock and target Jan/Feb. Mellanox already has their own OFED distribution -- since there appears to be strong desire to get this stuff released ASAP, is there an issue with releasing it through Mellanox OFED. Then later release it through community OFED in the next go-round? On Nov 23, 2009, at 4:18 AM, Liran Liss wrote: In the past few months of review, the responsibility for rdmaoe addressing was moved to the rdmacm. So, any future addressing enhancements can be confined to the rdmacm module without breaking existing APIs. RFC 3041 deals with static global IP addresses on the Internet, especially for portable devices. rmdaoe allows using link-local GIDs for applications residing on the same subnet, so I don't see the relevance. Note that for rdmacm apps, the intention is to map the IP addresses that were assigned to the host's interfaces. Please see http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf. Regarding multicast, current switches will flood the traffic just as any other non-IP multicast traffic (e.g., fcoe). Using switches that support multicast pruning for additional ethertypes, you can optimize the traffic and achieve the same link utilization as normal IP multicast. In any case, this is not a correctness issue that prohibits experimentation with rdmaoe multicast on any network today. --Liran -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier Sent: Thursday, November 19, 2009 9:35 PM To: Richard Frank Cc: o...@lists.openfabrics.org; OpenFabrics EWG Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > Having lots of testing exposure can help in validating that all the > edge cases are handled.. To some extent -- but there also needs to be some thinking involved to make sure that the interface can actually handle future cases. > Are there a set of cases that you have in mind ? For example -- how is multicast going to interact with IGMP on ethernet switches? How is address resolution going to be done (current patches seem to assume that stateless IPv6 link-local addresses contain the ethernet address, which is not valid if RFC 3041 is used)? etc - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg -- Jeff Squyres jsquy...@cisco.com ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] OFED with MPE
Jonathan Perkins schrieb: > Sorry for the late reply. You should be able to use mpirun_rsh in order > to avoid setting up mpd. It works fine. But i have to give -np. Back to Topic: When i try to compile an MPE Code i get: u...@head:~$ mpicc mandelbrot_mpe.c -o Apfel /tmp/ccwpw6so.o: In function `main': mandelbrot_mpe.c:72: undefined reference to `MPE_Open_graphics' mandelbrot_mpe.c:74: undefined reference to `MPE_Make_color_array' ... collect2: ld gab 1 als Ende-Status zurück u...@head:~$ Regards, Stefan Kuhne <> signature.asc Description: OpenPGP digital signature ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] ofa_1_5_kernel 20091123-0200 daily build status
This email was generated automatically, please do not reply git_url: git://git.openfabrics.org/ofed_1_5/linux-2.6.git git_branch: ofed_kernel_1_5 Common build parameters: Passed: Passed on i686 with linux-2.6.18 Passed on i686 with linux-2.6.21.1 Passed on i686 with linux-2.6.19 Passed on i686 with linux-2.6.26 Passed on i686 with linux-2.6.24 Passed on i686 with linux-2.6.22 Passed on i686 with linux-2.6.27 Passed on x86_64 with linux-2.6.16.60-0.54.5-smp Passed on x86_64 with linux-2.6.16.60-0.21-smp Passed on x86_64 with linux-2.6.18 Passed on x86_64 with linux-2.6.18-128.el5 Passed on x86_64 with linux-2.6.18-164.el5 Passed on x86_64 with linux-2.6.19 Passed on x86_64 with linux-2.6.20 Passed on x86_64 with linux-2.6.18-93.el5 Passed on x86_64 with linux-2.6.21.1 Passed on x86_64 with linux-2.6.24 Passed on x86_64 with linux-2.6.22 Passed on x86_64 with linux-2.6.26 Passed on x86_64 with linux-2.6.27 Passed on x86_64 with linux-2.6.25 Passed on x86_64 with linux-2.6.27.19-5-smp Passed on x86_64 with linux-2.6.9-89.ELsmp Passed on x86_64 with linux-2.6.9-67.ELsmp Passed on x86_64 with linux-2.6.9-78.ELsmp Passed on ia64 with linux-2.6.18 Passed on ia64 with linux-2.6.21.1 Passed on ia64 with linux-2.6.19 Passed on ia64 with linux-2.6.23 Passed on ia64 with linux-2.6.24 Passed on ia64 with linux-2.6.22 Passed on ia64 with linux-2.6.26 Passed on ia64 with linux-2.6.25 Passed on ppc64 with linux-2.6.18 Passed on ppc64 with linux-2.6.19 Failed: ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
In the past few months of review, the responsibility for rdmaoe addressing was moved to the rdmacm. So, any future addressing enhancements can be confined to the rdmacm module without breaking existing APIs. RFC 3041 deals with static global IP addresses on the Internet, especially for portable devices. rmdaoe allows using link-local GIDs for applications residing on the same subnet, so I don't see the relevance. Note that for rdmacm apps, the intention is to map the IP addresses that were assigned to the host's interfaces. Please see http://www.t11.org/ftp/t11/pub/fc/study/09-543v0.pdf. Regarding multicast, current switches will flood the traffic just as any other non-IP multicast traffic (e.g., fcoe). Using switches that support multicast pruning for additional ethertypes, you can optimize the traffic and achieve the same link utilization as normal IP multicast. In any case, this is not a correctness issue that prohibits experimentation with rdmaoe multicast on any network today. --Liran -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier Sent: Thursday, November 19, 2009 9:35 PM To: Richard Frank Cc: o...@lists.openfabrics.org; OpenFabrics EWG Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > Having lots of testing exposure can help in validating that all the > edge cases are handled.. To some extent -- but there also needs to be some thinking involved to make sure that the interface can actually handle future cases. > Are there a set of cases that you have in mind ? For example -- how is multicast going to interact with IGMP on ethernet switches? How is address resolution going to be done (current patches seem to assume that stateless IPv6 link-local addresses contain the ethernet address, which is not valid if RFC 3041 is used)? etc - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] RE: [ofw] SC'09 BOF - Meeting notes
facts... the patch set sent from downtown Yoqne'am isn't an addition of feature turns out that some folks from the Mellanox R&D group found this sentence insulting, and I am apologizing for that. Mentioning the geographic location of the developers didn't come to serve why I find the patch set this or that, but rather send the author of the email I was responding on to go and do homework in his own company office. Or. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
90% of the changes are either in the mlx4 driver, or self-contained in the rdmaoe flow of the cma, which handles rdmaoe addressing and connection setup. The rest of the changes indeed touch various locations of the stack, but they are either definitions or follow the same logic: if (rdma_is_trasnport(ib_device, RDMA_TRANSPORT_RDMAOE)) do_something_rdmaoe_specific(); The patches don't change the logic of existing flows at all, so we are not risking *anything* in terms of the stability of the current stack. As for vlan id and priorities - we are fully aware to the importance of exposing vlan ids and priorities to the user, but thanks for pointing this out. There are deployments today that work fine with the current patches; but in any case, we are planning to send a follow-up patch set that adds vlan+priority support in the near future. --Liran -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Or Gerlitz Sent: Friday, November 20, 2009 1:39 AM To: Richard Frank Cc: Sean Hefty; Roland Dreier; OpenFabrics EWG Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes Richard Frank wrote: > How can 1500 lines out of 240k lines be a big change.. do I have these > numbers right > - is the big change you are referring too? Rick, the change set is way not self contained but rather touches various parts of the core IB stack (rdma-cm module, ib address resolution module, ib uverbs module and even the mad module) and ofcourse some of the kernel and user space IB hw specific libraries. > What is the risk area that you are worried about .. do you think it > will break current transports or existing ULPs? yes, this would be simply not supportable, think about that, you want to hand your customers with a code which didn't pass review nor acceptance by the Linux IB stack maintainers (Roland and Sean), say, next a crash happens at this or that module / line, next, what you except the maintainers to do? > If it's just about how the implementation is done.. can this be > resolved concurrently with getting the bits available for evaluation now.. an rdmaoe branch at the git tree was set and an releases are maintained, its all what you need for evaluation, five lines later you're talking on deployments... > As RoCEE is totally transparent to existing ULPs.. any potential > changes would not be visible.. and therefore not an issue for ULP / clients going forward.. right? this is how you see things, since the IBTA IBXoE annex isn't released, you just don't know what would be the bottom line. > Oracle would like to see RoCEE get into 1.5 you guys have set a note to the rds developer community that that Oracle recently moved from 1.3.x to 1.4.y, no special work is expected on 1.5.z and that you have lots of plans for 1.6.w ... what's the urgency to get these bits into 1.5? > We are testing with RoCEE now and plan to deploy it fairly soon.. in > very large configuratio the proposed patch set doesn't let you use non zero VLAN, aren't you expecting Ethernet customers to trivially require that? also you can't use non zero traffic class (priority bits), where all the IBXoE materials are talking about how much working on a lossless traffic class is a must... if indeed this is the case, the patch set is useless without the ability to specify a traffic class, as CEE switches would typically (always?) set only some of the traffic classes to be lossless (e.g the ones used for FCoE, IBXoE) and the rest to be lossy Or ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
On Mon, Nov 23, 2009 at 10:11:21AM +0200, Eli Cohen wrote: Would like to fix a typo: I meant bellow: Saying that the patch set did not go through a review process would **be** inaccurate. > On Fri, Nov 20, 2009 at 01:38:59AM +0200, Or Gerlitz wrote: > > > > yes, this would be simply not supportable, think about that, you want > > to hand your customers with a code which didn't pass review nor > > acceptance by the Linux IB stack maintainers (Roland and Sean), say, > > next a crash happens at this or that module / line, next, what you > > except the maintainers to do? > > > Saying that the patch set did not go through a review process would > not inaccurate. > Bellow is a brief log of major changes done on the RDMAoE patches for > your reference. A detailed correspondence can be found at the > openfabrics general list. > > Rev1 - June 15 2009, first patch set sent for review > Rev2 - June 25 2009, Sean - move path resolution to a new module > (rdmaoe_sa) > Rev3 - July 13 2009, Sean, Roland, share data structs between > multicast.c and rdmaoe_sa.c, distinguish between rdmaoe and ib calls > at the cma, increment ABI version > Rev4 - Aug 5 2009, Woody Sean Or, ports are differentiated by port > protocol rather than port type, move rdmaoe sa functionality to cma > Rev5 - Aug 19 2009, Roland, Sean, don't use broadcast MACs to map > multicast GIDs, MAD service disabled for userspace, add > rdma_is_transport_supported() > Annonuce - Sep 17 2009, OFED-RDMAoE branch announce, daily builds > available > Rev6 - Nov 16 2009, NIC programming moved from CMA to hw driver so > verbs consumer can utilize it. > > > ___ > ewg mailing list > ewg@lists.openfabrics.org > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
As far as core APIs go, the patch set introduces 2 basic additions rather than changes: - A new ABI function to resolve gids to macs - ib_get_mac() - A new kernel ib_device function to get the port transport - ib_get_port_transport(). There are no changes to the Verbs API. All the address resolution stuff is contained in the cma code, so I think we code extend its logic in the future without breaking things at the interface level. Do you have anything specific in mind? --Liran -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Roland Dreier Sent: Thursday, November 19, 2009 9:17 PM To: Richard Frank Cc: o...@lists.openfabrics.org; OpenFabrics EWG Subject: Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes > How can 1500 lines out of 240k lines be a big change.. do I have these > numbers right - is the > big change you are referring too? If there are significant changes to the core APIs -- and IBoE has exactly this impact -- then yes it can be a big change even if the line count is small. > What is the risk area that you are worried about .. do you think it > will break current > transports or existing ULPs ? I am worried that no one has thought through all the issues and corner cases around address resolution, multicast, etc, and that when we do get a standardized version of IBoE, we'll have to break core APIs yet again. - R. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Re: [ofw] SC'09 BOF - Meeting notes
On Fri, Nov 20, 2009 at 01:38:59AM +0200, Or Gerlitz wrote: > > yes, this would be simply not supportable, think about that, you want > to hand your customers with a code which didn't pass review nor > acceptance by the Linux IB stack maintainers (Roland and Sean), say, > next a crash happens at this or that module / line, next, what you > except the maintainers to do? > Saying that the patch set did not go through a review process would not inaccurate. Bellow is a brief log of major changes done on the RDMAoE patches for your reference. A detailed correspondence can be found at the openfabrics general list. Rev1 - June 15 2009, first patch set sent for review Rev2 - June 25 2009, Sean - move path resolution to a new module (rdmaoe_sa) Rev3 - July 13 2009, Sean, Roland, share data structs between multicast.c and rdmaoe_sa.c, distinguish between rdmaoe and ib calls at the cma, increment ABI version Rev4 - Aug 5 2009, Woody Sean Or, ports are differentiated by port protocol rather than port type, move rdmaoe sa functionality to cma Rev5 - Aug 19 2009, Roland, Sean, don't use broadcast MACs to map multicast GIDs, MAD service disabled for userspace, add rdma_is_transport_supported() Annonuce - Sep 17 2009, OFED-RDMAoE branch announce, daily builds available Rev6 - Nov 16 2009, NIC programming moved from CMA to hw driver so verbs consumer can utilize it. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
