[ewg] OpenFabrics Verbs working group
Hi, Following the creation of the OpenFabrics Verbs working group, the mailing list of the OFV WG was created and is now ready. In order to send and receive emails please subscribe to the mailing list at http://lists.openfabrics.org/mailman/listinfo/ofvwg Regards, Gilad Shainer Vice President, Marketing Mellanox Technologies 350 Oakmead Parkway, Suite 100, Sunnyvale CA, 94085 Office: 408-916-0048, Mobile: 408-421-0048, Fax: 408-585-0348 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/mailman/listinfo/ewg
Re: [ewg] To OFED or not to OFED
Jim, I believe that we should also include Tziporet and maybe others, therefore Friday will not work for folks from Israel. Gilad -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Ryan, Jim Sent: Wednesday, April 18, 2012 8:28 AM To: Tziporet Koren; ewg@lists.openfabrics.org Subject: Re: [ewg] To OFED or not to OFED Tziporet, I think this is a very important decision. I'd like to make it an agenda item for the Board meeting on Friday. Woody typically participates -- would this be ok? Thanks, Jim -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Tziporet Koren Sent: Wednesday, April 18, 2012 8:04 AM To: ewg@lists.openfabrics.org Subject: [ewg] To OFED or not to OFED Hi All, We started to work on OFED 3.2 and for now we already have the base code working with the new compat framework: - IB core (all modules) - IPoIB - mlx4, mthca - Mellanox drivers However for the last few months there was no progress on any other low level driver: - qib - Intel - nec - Intel - cxgb3/4 - Chelsio Since there is no reason to have OFED release for Mellanox only we need to understand the other companies position and commitment for OFED. We have two options: - Drop OFED releases all together. Focus our EWG efforts in pushing APIs, testing of the Linux inbox SW stack and driver enhancements for the RDMA technology - Get real commitment from the vendors on dates to provide the backport support (we can move to kernel 3.3 if we decide this) Please do your internal consideration and come back with your decision. The final decision will be taken in next EWG meeting on May 1. Thanks Tziporet Woody ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] dissappointing IPoIB performance
Richard, Critical missing is the setup information. What is the server, CPU etc. Can you please provide? Gilad From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of richard Croucher Sent: Monday, February 20, 2012 8:50 AM To: ewg@lists.openfabrics.org Subject: [ewg] dissappointing IPoIB performance I've been undertaking some internal QA testing of the Mellanox CX3's. An observation that I've seen for some time and is most likely to do with the IPoIB implementation rather than the HCA is that the latency of IPoIB is getting increasingly poor in comparison with the standard kernel TCP/IP stack over 10g Ethernet. If we look at the results for the CX3's running both 10G Ethernet and 40G InfiniBand, on same serve hardware, I get the following median latency with my test setup. Results are with my own test program so are only meaningful as a comparison with the other configurations running the same test. Running OFED 1.5.3 and RH 6.0 IPoIB (connected) TCP 33.67 uS(switchless) IPoIB (datagram)TCP 31.63 uS(switchless) IPoIB (connected) UDP 24.78 uS(switchless) IPoIB (datagram) UDP 24.28 uS (switchless) IPoIB (connected) UDP 25.37 uS(1 hop) between ports on same switch IPoIB (connected) TCP 34.48 uS (1 hop) 10G Ethernet UDP 24.04uS (2 hops) across a LAG connected pair of Ethernet switches 10G Ethernet TCP 34.59 uS(2 hops) The Mellanox Ethernet drivers are tuned for low latency rather than throughput, but I would have hoped that given the 4x extra bandwidth available it would have helped the InfiniBand drivers outperform. I've seen similar results for CX2 .10G ethernet is increasingly looking like the better option for low latency, particularly with the current generation of low latency Ethernet switches. Switchless Ethernet has been better for some time than switchless InfiniBand, but it now looks to be the case in switched environments as well. I think this reflects that there has been a lot of effort tweaking and tuning TCP/IP over Ethernet and its low level drivers, with very little activity on the IPoIB front. Unless we see improvements here it will get increasingly difficult to justify InfiniBand deployments. -- Richard Croucher www.informatix-sol.comhttp://www.informatix-sol.com +44-7802-213901 ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] Mellanox IB/10Gige Card
You may want to check with supp...@mellanox.com. Gilad From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Jagga Soorma Sent: Tuesday, February 15, 2011 8:07 PM To: ewg@lists.openfabrics.org Subject: [ewg] Mellanox IB/10Gige Card Hi Guys, I have a server that is supposed to have a QDR port and a port that is supposed to be a 10Gbe port. When I run ibstatus, the 10Gbe port seems to be disabled. Is there a way to enable this port? -- [root@node01 ~]# ibstatus Infiniband device 'mlx4_0' port 1 status: default gid: fe80::::78e7:d103:0021:887d base lid:0x12 sm lid: 0x1 state: 4: ACTIVE phys state: 5: LinkUp rate:40 Gb/sec (4X QDR) link_layer: IB Infiniband device 'mlx4_0' port 2 status: default gid: fe80::::7ae7:d1ff:fe21:887d base lid:0x0 sm lid: 0x0 state: 1: DOWN phys state: 3: Disabled rate:40 Gb/sec (4X QDR) link_layer: Ethernet -- Should I be using ibportstate to enable this port 2? Any help would be greatly appreciated. I have the mellanox ofed installed on this server. Thanks, -J ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
Re: [ewg] GPUDirect Support
It is not yet in the distro. If you need the code, please send me an email. Gilad From: ewg-boun...@lists.openfabrics.org To: ewg@lists.openfabrics.org Sent: Wed Feb 02 13:03:24 2011 Subject: [ewg] GPUDirect Support Hey Guys, I am trying to enable GPUDirect support in our environment and I have built OFED release 1.5.2. I was expecting the following directory to be created but don't see this directory and the files within that would help me enable gpudirect: /sys/module/ib_core/parameters The parameter file I am looking for is /sys/module/ib_core/parameters/gpu_direct_enable. Any ideas as to why the parameters directory is missing altogether? Anyone on this list have any experience with this. Here is some information about my environment: OS: Red Hat Enterprise Linux Server release 5.4 (Tikanga) kernel: 2.6.18-164.el5 hca info: -- Image type: ConnectX FW Version: 2.7.9100 Rom Info:type=PXE version=3.2.0 devid=26438 proto=VPI Device ID: 26438 Chip Revision: B0 Description: Node Port1Port2Sys image GUIDs: 78e7d10300218884 78e7d10300218885 78e7d10300218886 78e7d10300218887 MACs: 78e7d1218884 78e7d1218885 Board ID: (HP_020003) VSD: PSID:HP_020003 -- Any help would be appreciated. Thanks, -J ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] WinOF applications
And there are commercial MPIs that uses WinOF -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of John Russo Sent: Monday, June 01, 2009 9:28 AM To: Jeff Squyres; Steve Wise Cc: OpenFabrics EWG Subject: RE: [ewg] WinOF applications WinOF actually supports Microsoft MPI over Network Direct on Windows 2008 HPC -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Jeff Squyres Sent: Monday, June 01, 2009 12:28 PM To: Steve Wise Cc: OpenFabrics EWG Subject: Re: [ewg] WinOF applications Correct -- Open MPI does not use WinOF. On Jun 1, 2009, at 12:22 PM, Steve Wise wrote: Can someone point me to documentation or email threads that define what apps actually run over WinOF on Windows Server 2008? For instance, are there any MPIs that run on this? I believe Open MPI does not. -- Jeff Squyres Cisco Systems ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] WinOF applications
Intel, HP and Platform are better to answer this question. AFAIK, Intel and HP have solutions. -Original Message- From: Steve Wise [mailto:sw...@opengridcomputing.com] Sent: Monday, June 01, 2009 9:46 AM To: Gilad Shainer Cc: John Russo; Jeff Squyres; OpenFabrics EWG Subject: Re: [ewg] WinOF applications Gilad Shainer wrote: And there are commercial MPIs that uses WinOF Which ones? Scali? Intel? HP? Thanks, Steve. ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
RE: [ewg] RE: OFED Jan 5, 2009 meeting minutes on OFED plans
We need to look on this from the right angel. This is not a feature but rather a core component that adds support for a new adapter/NIC. This is the same as the core drivers for the other adapters that are supported already. In general we need to look not only on spec related features, but also to cover features that can benefit OFED and WinOF users (such as IPoIB connected mode or WinVerbs). Gilad. -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Ryan, Jim Sent: Tuesday, January 06, 2009 2:01 PM To: Hefty, Sean; Tziporet Koren; ewg@lists.openfabrics.org Cc: gene...@lists.openfabrics.org Subject: RE: [ewg] RE: OFED Jan 5, 2009 meeting minutes on OFED plans Sean, I think that's a good point. What it suggests to me is asking when someone proposes a non-standard feature, what process, procedures, documentation, support, etc. if any, should be made available by the entity making the proposal? It seems to me asking the same questions of all proposed features is fair and reasonable, and shouldn't represent an unreasonable barrier to progress. Thoughts? If this already exists, it's my ignorance and I will apologize in advance Thanks again, Jim -Original Message- From: ewg-boun...@lists.openfabrics.org [mailto:ewg-boun...@lists.openfabrics.org] On Behalf Of Sean Hefty Sent: Tuesday, January 06, 2009 1:54 PM To: 'Tziporet Koren'; ewg@lists.openfabrics.org Cc: gene...@lists.openfabrics.org Subject: [ewg] RE: OFED Jan 5, 2009 meeting minutes on OFED plans * Mellanox suggested to add IB over Eth - this is similar to iWARP but more like IB (e.g. including UD), and can work over ConnectX. A concern was raised by Intel (Dave Sommers) since it is not a standard transport. Decision: This request will be raised in the MWG, and they should decide if OFA can support it. Just is just my opinion, but in the past, OFED has included non-standard features, like extended connected mode, that are still not part of the IBTA spec. Do we know if such a feature would be accepted into the Linux kernel? I think OFED should base their decision more on the answer to that question than IBTA approval. - Sean ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
[ewg] RE: [ofw] RE: OFA BOF slides for SC'08 (Revision 0.8)
Please send the update Woody, else you had it already in the deck you send before -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Woodruff, Robert J Sent: Monday, November 03, 2008 3:50 PM To: Hefty, Sean; ewg@lists.openfabrics.org; [EMAIL PROTECTED] Cc: Wayne Augsburger Subject: [ofw] RE: OFA BOF slides for SC'08 (Revision 0.8) Thanks, applied in latest version. woody -Original Message- From: Hefty, Sean Sent: Monday, November 03, 2008 2:58 PM To: Woodruff, Robert J; Woodruff, Robert J; ewg@lists.openfabrics.org; [EMAIL PROTECTED] Cc: Wayne Augsburger Subject: RE: OFA BOF slides for SC'08 (Revision 0.8) Slide 18 Add WinVerbs, WinMad, and 'OFED libraries on Windows' with myself as maintainer. A maintainer may be needed for the OFED diagnostics tools if we cannot agree to share a common code-base. Slide 20 Under WinVerbs, add: Supports OFED libibverbs Slide 21 Under WinVerbs, add: Support OFED librdmacm For discussion points: Slide 7 As we provide the OFED diagnostics on Windows, we need to rethink having libraries that can 'change without notice'. - Sean ___ ofw mailing list [EMAIL PROTECTED] http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ofw ___ ewg mailing list ewg@lists.openfabrics.org http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
