Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Fri, Oct 14, 2011 at 2:22 PM, Doug Ledford dledf...@redhat.com wrote: - Original Message - On Wed, Oct 12, 2011 at 9:32 AM, Wendy Cheng s.wendy.ch...@gmail.com wrote: The OFED package itself does include XRC support. The issue here (my guess) is that its build script needs to understand the running system's kernel version to decide what should be pulled (from the source). Linux 3.1 could be too new for OFED build script to make a correct decision. Nevertheless, mix-matching OFED modules/libraries is a *bad* idea. No. The same userspace build should work with all kernel versions. Wendy's referring to something other than what you are thinking. The same libibverbs user space build should work on all kernels going back a long way, except when you are talking about OFED their libibverbs is hard coded to assume XRC support and fail if it isn't present, so an OFED libibverbs won't really work without also the OFED kernel module stack. The script Wendy referred to is the script that checks the running kernel's version in order to determine which backport patches need applied to the ofa_kernel source tree in order to build the OFED kernel modules for your running kernel. Without that ofa_kernel build, the OFED libibverbs will indeed fail to run on the running kernel. And that script hasn't been updated to support version 3.x kernels last I checked, so she's right, the script itself doesn't recognize the running kernel version, so ofa_kernel modules don't get built, so OFED libibverbs won't work anyway. So, she's absolutely right, unless you want to start ripping hard coded assumptions about the existence of XRC support out of things like OFED's libibverbs, then out of qperf and a number of their other various packages, then you have to pair the OFED kernel modules and user space packages, they can not be separated. Yes, that (above) is exactly what I was referring to . The conversations in this thread remind me of the tire-swing cartoon that has been passing around for years: http://bibiananunes.com/user-requirements-the-tire-swing-cartoon ) -- Wendy -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
- Original Message - On Thu, Oct 13, 2011 at 10:24 AM, Christoph Lameter c...@gentwo.org wrote: Clean reviewed patches for this are where? They are in OFED-1.5.3.1 so they were already released. OFED is neither clean nor reviewed. Really. The stuff in OFED always needs a bunch of review before it is suitable for upstream. Ignoring ABI stability is just one problem that OFED code typically has. Indeed. Because it is in OFED means it is in testing, nothing more. A better acronym for OFED is Open Fabrics Experimental Distribution. These issues are the primary reason we don't follow OFED in RHEL6. -- Doug Ledford dledf...@redhat.com GPG KeyID: 0E572FDD http://people.redhat.com/dledford -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
My latest patches are at: git://git.openfabrics.org/~shefty/rdma-dev.git xrc So I went through this and merged it to my tree (pretty much only conflicts from 3.0-3.1-rc fixed, commit log changes and other minor cleanups). The result is pushed out to my github for-next branch, with the expectation that I'll ask Linus to pull for 3.2. However I do have one question: the last patch (RDMA/uverbs: Export ib_open_qp() capability to user space in my tree) adds IB_USER_VERBS_CMD_OPEN_QP but I don't see any driver add that to its uverbs_cmd_mask... does this work? Thanks, Roland -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, 13 Oct 2011, Roland Dreier wrote: My latest patches are at: git://git.openfabrics.org/~shefty/rdma-dev.git xrc So I went through this and merged it to my tree (pretty much only conflicts from 3.0-3.1-rc fixed, commit log changes and other minor cleanups). We got the XRC support via the for-next branch on github. Is that current? The result is pushed out to my github for-next branch, with the expectation that I'll ask Linus to pull for 3.2. However I do have one question: the last patch (RDMA/uverbs: Export ib_open_qp() capability to user space in my tree) adds IB_USER_VERBS_CMD_OPEN_QP but I don't see any driver add that to its uverbs_cmd_mask... does this work? Thanks, Roland There seems to be some stuff missing in the upstream code compared to the OFED releases: 1. Raw ethernet support (IB_QPT_RAW_ETH) is missing. 2. MLX4_IB_QP_BLOCK_LOOPBACK is broken it seems? All packets are fed back via loopback?
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
Oh yeah and can we can FDR support in for-next as well? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, Oct 13, 2011 at 10:17 AM, Christoph Lameter c...@gentwo.org wrote: There seems to be some stuff missing in the upstream code compared to the OFED releases: 1. Raw ethernet support (IB_QPT_RAW_ETH) is missing. 2. MLX4_IB_QP_BLOCK_LOOPBACK is broken it seems? All packets are fed back via loopback? Clean reviewed patches for this are where? - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, 13 Oct 2011, Roland Dreier wrote: On Thu, Oct 13, 2011 at 10:17 AM, Christoph Lameter c...@gentwo.org wrote: There seems to be some stuff missing in the upstream code compared to the OFED releases: 1. Raw ethernet support (IB_QPT_RAW_ETH) is missing. 2. MLX4_IB_QP_BLOCK_LOOPBACK is broken it seems? All packets are fed back via loopback? Clean reviewed patches for this are where? They are in OFED-1.5.3.1 so they were already released. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, Oct 13, 2011 at 10:24 AM, Christoph Lameter c...@gentwo.org wrote: Clean reviewed patches for this are where? They are in OFED-1.5.3.1 so they were already released. OFED is neither clean nor reviewed. Really. The stuff in OFED always needs a bunch of review before it is suitable for upstream. Ignoring ABI stability is just one problem that OFED code typically has. - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries
The result is pushed out to my github for-next branch, with the expectation that I'll ask Linus to pull for 3.2. Thanks - I'll take a look and test again. However I do have one question: the last patch (RDMA/uverbs: Export ib_open_qp() capability to user space in my tree) adds IB_USER_VERBS_CMD_OPEN_QP but I don't see any driver add that to its uverbs_cmd_mask... does this work? ib_open_qp() is implemented entirely in the verbs layer (verbs.c). The OFED API compatibility support that I added to libibverbs makes use of this call, which I tested by running OSU's mvapich2. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, 13 Oct 2011, Roland Dreier wrote: On Thu, Oct 13, 2011 at 10:24 AM, Christoph Lameter c...@gentwo.org wrote: Clean reviewed patches for this are where? They are in OFED-1.5.3.1 so they were already released. OFED is neither clean nor reviewed. Really. The stuff in OFED always needs a bunch of review before it is suitable for upstream. Ignoring ABI stability is just one problem that OFED code typically has. Yeah. ABI stability etc is really what we want. Thats why I would like to see it upstream rather than continue to work with this OFED tarball ofa_kernel mess. So the patches need to be resubmitted for upstream inclusion from Mellanox to you? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, Oct 13, 2011 at 10:31 AM, Hefty, Sean sean.he...@intel.com wrote: ib_open_qp() is implemented entirely in the verbs layer (verbs.c). The OFED API compatibility support that I added to libibverbs makes use of this call, which I tested by running OSU's mvapich2. Wait, now I'm baffled by the patch (ie http://git.openfabrics.org/git?p=~shefty/rdma-dev.git;a=commitdiff;h=1ec4e62a6e967ddc258e7c4e674168debb727d39) I don't see anything that calls ib_uverbs_open_qp(). Am I missing something?? Does the OFED API compatibility actually call this function and care if it fails? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries
Wait, now I'm baffled by the patch (ie http://git.openfabrics.org/git?p=~shefty/rdma- dev.git;a=commitdiff;h=1ec4e62a6e967ddc258e7c4e674168debb727d39) I don't see anything that calls ib_uverbs_open_qp(). Am I missing something?? Does the OFED API compatibility actually call this function and care if it fails? I'm confused now. The patch definitely looks like it's missing a change to uverbs_main.c to setup the command table. (I thought you were referring to needing a change to struct ib_device before.) I'm looking at the code on my test system and trying to understand what exactly is happening. The OFED compatibility should call this routine and care if it fails, and it looks like mpi cares too, but I don't see how it can work... Give me some time to look into this. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Thu, Oct 13, 2011 at 11:32 AM, Hefty, Sean sean.he...@intel.com wrote: I'm confused now. The patch definitely looks like it's missing a change to uverbs_main.c to setup the command table. (I thought you were referring to needing a change to struct ib_device before.) Right, we need to add it to the command table, and also the hardware driver needs to opt into the function in its uverbs cmd mask. (Even if the call never hits the hardware driver ... we could clean that up maybe, but I'm just talking about how things work at the moment) I'm looking at the code on my test system and trying to understand what exactly is happening. The OFED compatibility should call this routine and care if it fails, and it looks like mpi cares too, but I don't see how it can work... Give me some time to look into this No problem. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Tue, Oct 11, 2011 at 7:39 PM, Jason Gunthorpe jguntho...@obsidianresearch.com wrote: On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote: Has XRC support not been merged? How can I build the OFED libraries against Linux 3.1? I'd really like to get rid of the OFED kernel tree nightmare. You have to use upstream libraries with upstream kernels. Be warned that the OFED libraries of the same SONAME are not ABI compatible with upstream libraries. Why is the OFED libibverbs library binary incompatible with the non-OFED libibverbs library ? Why hasn't XRC support been implemented in the OFED libibverbs library such that applications built against the upstream libibverbs headers also work with the latest OFED version of that library ? Bart. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Wed, Oct 12, 2011 at 3:41 AM, Bart Van Assche bvanass...@acm.org wrote: On Tue, Oct 11, 2011 at 7:39 PM, Jason Gunthorpe jguntho...@obsidianresearch.com wrote: On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote: Has XRC support not been merged? How can I build the OFED libraries against Linux 3.1? I'd really like to get rid of the OFED kernel tree nightmare. You have to use upstream libraries with upstream kernels. Be warned that the OFED libraries of the same SONAME are not ABI compatible with upstream libraries. Why is the OFED libibverbs library binary incompatible with the non-OFED libibverbs library ? Why hasn't XRC support been implemented in the OFED libibverbs library such that applications built against the upstream libibverbs headers also work with the latest OFED version of that library ? I'm relatively new to OFED but happened to bump into a similar build issue two weeks ago. The OFED package itself does include XRC support. The issue here (my guess) is that its build script needs to understand the running system's kernel version to decide what should be pulled (from the source). Linux 3.1 could be too new for OFED build script to make a correct decision. Nevertheless, mix-matching OFED modules/libraries is a *bad* idea. It is difficult to love OFED build :) but it seems to work ok (so far for me). Plus, I don't have a better proposal myself anyway . -- Wendy -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries
Why is the OFED libibverbs library binary incompatible with the non-OFED libibverbs library ? Why hasn't XRC support been implemented in the OFED libibverbs library such that applications built against the upstream libibverbs headers also work with the latest OFED version of that library ? The original XRC patches were submitted upstream, but were never accepted. Along with those patches were modifications to ibverbs. Those changes broke existing applications and were also never accepted upstream. However, OFED still provided a release based on that code. This is only a guess, but the push to release may have been a time to market decision. (I'm just trying to relay history here.) I have since re-worked both the kernel and user space XRC patches and submitted them for inclusion upstream. Those patches do maintain backwards compatibility. I believe that OFA is waiting for those patches (or some deviation) to be accepted upstream. OFED will then be rebased on those patches, released as 2.0, with a stronger attempt made to keep everything in sync. Hopefully XRC can make it into the 3.2 kernel. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Wed, Oct 12, 2011 at 9:32 AM, Wendy Cheng s.wendy.ch...@gmail.com wrote: The OFED package itself does include XRC support. The issue here (my guess) is that its build script needs to understand the running system's kernel version to decide what should be pulled (from the source). Linux 3.1 could be too new for OFED build script to make a correct decision. Nevertheless, mix-matching OFED modules/libraries is a *bad* idea. No. The same userspace build should work with all kernel versions. We did have some ABI churn early in the history of the RDMA stack -- the last time we changed IB_USER_VERBS_ABI_VERSION was early in 2006. But we really need to add XRC and other support in such a way that such ABI bumps are not necessary. - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: Building 3.1-rc9 in kernel infiniband support with OFED libraries
Did we every resolve the controversy about the rcv QPs with MPI users? The design seemed sane to me, but Yes - I believe so. Also (I'm sure you already posted this once, but...) Sean, do you have a git tree with all the kernel patches included? My latest patches are at: git://git.openfabrics.org/~shefty/rdma-dev.git xrc though this is based on 3.0. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Wed, Oct 12, 2011 at 9:46 AM, Hefty, Sean sean.he...@intel.com wrote: though this is based on 3.0. Thanks, that's fine. I don't think I merged many changes for 3.1 ;) - R. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Building 3.1-rc9 in kernel infiniband support with OFED libraries
Seems to work mostly but some userspace libraries (mlx4) complain about kernel version mismatch and missing XRC support. Has XRC support not been merged? How can I build the OFED libraries against Linux 3.1? I'd really like to get rid of the OFED kernel tree nightmare. -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote: Has XRC support not been merged? How can I build the OFED libraries against Linux 3.1? I'd really like to get rid of the OFED kernel tree nightmare. You have to use upstream libraries with upstream kernels. Be warned that the OFED libraries of the same SONAME are not ABI compatible with upstream libraries. Jason -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Building 3.1-rc9 in kernel infiniband support with OFED libraries
On Tue, 11 Oct 2011, Jason Gunthorpe wrote: On Tue, Oct 11, 2011 at 09:02:41AM -0500, Christoph Lameter wrote: Has XRC support not been merged? How can I build the OFED libraries against Linux 3.1? I'd really like to get rid of the OFED kernel tree nightmare. You have to use upstream libraries with upstream kernels. Be warned that the OFED libraries of the same SONAME are not ABI compatible with upstream libraries. Thats a pretty bad situation. Could we not at least get the kernel API standardized? Publish git trees for ofed that are based on supported upstream and distro versions? -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
