RE: [RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
I was able to run mvapich2 osu benchmarks with xrc support using the following options: configure --with-rdma=gen2 --enable-xrc=yes CFALGS='-DOFED_VERBS -D_ENABLE_XRC' export MV2_USE_SHARED_MEM=0 export MV2_USE_XRC=1 Testing revealed a couple of issues in the libraries and kernel patches. All changes have been pushed to git.openfabrics.org into my various git trees: rdma-dev.git, libibverbs.git, libmlx4.git, and librdmacm.git -- xrc branches. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
Hi Sean, I am pleased that you are putting in the effort to enable the existing OFED user base to continue using its code without changes to the XRC calls. Regarding XRC and MPI, see below. On Wednesday 13 July 2011 20:21, Hefty, Sean wrote: I was able to build and run mvapich2 successfully against libibverbs with this patch applied on top of the current XRC patches. (The XRC patches are still undergoing work.) I built mvapich2 using the following configure options: --with-rdma=gen2 CFLAGS=-DOFED_VERBS and --with-rdma=gen2 CFLAGS='-DOFED_VERBS -D_ENABLE_XRC_ It didn't appear that mvapichs ever used XRC You are correct, mvapich does not use XRC. openMPI uses XRC, so hopefully you can use openMPI to test out your XRC stuff. You can contact Jeff Squyres for details/help. In the meantime, I include the following from the ewg list: = On 11/08/2010 08:06 PM, Jeff Squyres wrote: Steve pinged me on IM this morning and told me that you want OMPI v1.4.3 for the next OFED release. I just logged into www.openfabrics.org and apparently the server has changed -- my entire $HOME is empty. Where do you want me to put the new OMPI SRPM? Alternatively, anyone can grab the SRPM from the URL below -- there's nothing special about the SRPM for OpenFabrics that's not already in our community SRPM: http://www.open-mpi.org/software/ompi/v1.4/ Hi Jeff, The place for the Open MPI on the new server is under: /var/www/openfabrics.org/downloads/openmpi/ (http://www.openfabrics.org/downloads/openmpi/) I updated Open MPI version there to v1.4.3. Regards, Vladimir == -Jack -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
Sean pinged me last night about XRC in Open MPI last night (note that I am no longer on the linux-rdma list). Open MPI uses XRC, but in a non-default manner -- the user has to specifically ask for it at run time. On Jul 14, 2011, at 9:13 AM, Jack Morgenstein wrote: Hi Sean, I am pleased that you are putting in the effort to enable the existing OFED user base to continue using its code without changes to the XRC calls. Regarding XRC and MPI, see below. On Wednesday 13 July 2011 20:21, Hefty, Sean wrote: I was able to build and run mvapich2 successfully against libibverbs with this patch applied on top of the current XRC patches. (The XRC patches are still undergoing work.) I built mvapich2 using the following configure options: --with-rdma=gen2 CFLAGS=-DOFED_VERBS and --with-rdma=gen2 CFLAGS='-DOFED_VERBS -D_ENABLE_XRC_ It didn't appear that mvapichs ever used XRC You are correct, mvapich does not use XRC. openMPI uses XRC, so hopefully you can use openMPI to test out your XRC stuff. You can contact Jeff Squyres for details/help. In the meantime, I include the following from the ewg list: = On 11/08/2010 08:06 PM, Jeff Squyres wrote: Steve pinged me on IM this morning and told me that you want OMPI v1.4.3 for the next OFED release. I just logged into www.openfabrics.org and apparently the server has changed -- my entire $HOME is empty. Where do you want me to put the new OMPI SRPM? Alternatively, anyone can grab the SRPM from the URL below -- there's nothing special about the SRPM for OpenFabrics that's not already in our community SRPM: http://www.open-mpi.org/software/ompi/v1.4/ Hi Jeff, The place for the Open MPI on the new server is under: /var/www/openfabrics.org/downloads/openmpi/ (http://www.openfabrics.org/downloads/openmpi/) I updated Open MPI version there to v1.4.3. Regards, Vladimir == -Jack -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
On Thursday 14 July 2011 16:39, Dhabaleswar Panda wrote: Both MVAPICH1 and MVAPICH2 have XRC support for many years and are being used by many large-scale production clusters. Sorry about that! My bad. -Jack -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
Sean, Both MVAPICH1 and MVAPICH2 have XRC support for many years and are being used by many large-scale production clusters. For MVAPICH1, this support is available in both the standard Gen2 interface and the Hybrid (UD-RC/XRC) interface. From MVAPICH1 user guide (sec 4.4.1), you can find the following: eXtended Reliable Connnection Support (XRC): By default this support is compiled in to allow the usage of the new scalable XRC transport of InfiniBand. OFED 1.3 or later is required. If using an older version, then remove the -DXRC from the CFLAGS variable. The runtime flag is VIADEV_USE_XRC Class: Run Time Applicable device(s): Gen2 When MVAPICH is compiled with the XRC CFLAGS, this parameter enables use of the XRC transport of InfiniBand available on certain adapters. Enabling XRC automatically enables Shared Receive Queue and on-demand connection management. The hybrid (UD-RC/XRC) interface automatically selects the appropriate interface at run time. Not sure which interface (gen2 or hybrid you are using). If you are using the gen2 interface, looks like you are not using the correct runtime flag to enable XRC with MVAPICH1. Thanks, DK On Thu, 14 Jul 2011, Jeff Squyres wrote: Sean pinged me last night about XRC in Open MPI last night (note that I am no longer on the linux-rdma list). Open MPI uses XRC, but in a non-default manner -- the user has to specifically ask for it at run time. On Jul 14, 2011, at 9:13 AM, Jack Morgenstein wrote: Hi Sean, I am pleased that you are putting in the effort to enable the existing OFED user base to continue using its code without changes to the XRC calls. Regarding XRC and MPI, see below. On Wednesday 13 July 2011 20:21, Hefty, Sean wrote: I was able to build and run mvapich2 successfully against libibverbs with this patch applied on top of the current XRC patches. (The XRC patches are still undergoing work.) I built mvapich2 using the following configure options: --with-rdma=gen2 CFLAGS=-DOFED_VERBS and --with-rdma=gen2 CFLAGS='-DOFED_VERBS -D_ENABLE_XRC_ It didn't appear that mvapichs ever used XRC You are correct, mvapich does not use XRC. openMPI uses XRC, so hopefully you can use openMPI to test out your XRC stuff. You can contact Jeff Squyres for details/help. In the meantime, I include the following from the ewg list: = On 11/08/2010 08:06 PM, Jeff Squyres wrote: Steve pinged me on IM this morning and told me that you want OMPI v1.4.3 for the next OFED release. I just logged into www.openfabrics.org and apparently the server has changed -- my entire $HOME is empty. Where do you want me to put the new OMPI SRPM? Alternatively, anyone can grab the SRPM from the URL below -- there's nothing special about the SRPM for OpenFabrics that's not already in our community SRPM: http://www.open-mpi.org/software/ompi/v1.4/ Hi Jeff, The place for the Open MPI on the new server is under: /var/www/openfabrics.org/downloads/openmpi/ (http://www.openfabrics.org/downloads/openmpi/) I updated Open MPI version there to v1.4.3. Regards, Vladimir == -Jack -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/ -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
Both MVAPICH1 and MVAPICH2 have XRC support for many years and are being used by many large-scale production clusters. To clarify my comment: --with-rdma=gen2 CFLAGS='-DOFED_VERBS -D_ENABLE_XRC_ It didn't appear that mvapichs ever used XRC I only meant that mvapich2 didn't invoke any XRC calls in my testing. The portion of my patch which converted the ofed xrc calls to the new format weren't being tested, so this was a statement about my testing, not mvapich. The runtime flag is VIADEV_USE_XRC Class: Run Time Applicable device(s): Gen2 When MVAPICH is compiled with the XRC CFLAGS, this parameter enables use of the XRC transport of InfiniBand available on certain adapters. Enabling XRC automatically enables Shared Receive Queue and on-demand connection management. The hybrid (UD-RC/XRC) interface automatically selects the appropriate interface at run time. Not sure which interface (gen2 or hybrid you are using). If you are using the gen2 interface, looks like you are not using the correct runtime flag to enable XRC with MVAPICH1. FYI - I was running in loopback on the same system. I'll try using the run time flag mentioned above if xrc can be used in that configuration (with shared memory disabled). Thanks. - Sean -- To unsubscribe from this list: send the line unsubscribe linux-rdma in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[RFC v2] [OFED] libibverbs: Support both OFED verbs and ibverbs
This patch allows libibverbs to support both libibverbs API that shipped with OFED 1.5 and the upstream libibverbs API. This supports existing apps that are compiled against the upstream libibverbs (ibverbs). And in ideal cases, an application coded to the OFED version of libibverbs (ofverbs) would only need to be recompiled with 'CFLAGS=-DOFED_VERBS' given as a configuration option. Support for OFED verbs is done using macros that convert the OFED APIs to ibverbs APIs. In most cases, simple data casts are all that are necessary, with XRC support being the primary exception. Signed-off-by: Sean Hefty sean.he...@intel.com --- I was able to build and run mvapich2 successfully against libibverbs with this patch applied on top of the current XRC patches. (The XRC patches are still undergoing work.) I built mvapich2 using the following configure options: --with-rdma=gen2 CFLAGS=-DOFED_VERBS and --with-rdma=gen2 CFLAGS='-DOFED_VERBS -D_ENABLE_XRC_ It didn't appear that mvapichs ever used XRC libibverbs: Expose OFED 1.5 XRC APIs Makefile.am |5 - include/infiniband/ofverbs-defs.h | 54 ++ include/infiniband/ofverbs.h | 358 + include/infiniband/verbs.h|8 + src/libibverbs.map|7 + src/ofverbs.c | 173 ++ 6 files changed, 603 insertions(+), 2 deletions(-) create mode 100644 include/infiniband/ofverbs-defs.h create mode 100644 include/infiniband/ofverbs.h create mode 100644 src/ofverbs.c diff --git a/Makefile.am b/Makefile.am index 4702a2b..5148415 100644 --- a/Makefile.am +++ b/Makefile.am @@ -10,7 +10,7 @@ libibverbs_version_script = @LIBIBVERBS_VERSION_SCRIPT@ src_libibverbs_la_SOURCES = src/cmd.c src/compat-1_0.c src/device.c src/init.c \ src/marshall.c src/memory.c src/sysfs.c src/verbs.c \ - src/enum_strs.c + src/enum_strs.c src/ofverbs.c src_libibverbs_la_LDFLAGS = -version-info 1 -export-dynamic \ $(libibverbs_version_script) src_libibverbs_la_DEPENDENCIES = $(srcdir)/src/libibverbs.map @@ -37,7 +37,8 @@ libibverbsincludedir = $(includedir)/infiniband libibverbsinclude_HEADERS = include/infiniband/arch.h include/infiniband/driver.h \ include/infiniband/kern-abi.h include/infiniband/opcode.h include/infiniband/verbs.h \ -include/infiniband/sa-kern-abi.h include/infiniband/sa.h include/infiniband/marshall.h +include/infiniband/sa-kern-abi.h include/infiniband/sa.h include/infiniband/marshall.h \ +include/infiniband/ofverbs.h include/infiniband/ofverbs-defs.h man_MANS = man/ibv_asyncwatch.1 man/ibv_devices.1 man/ibv_devinfo.1\ man/ibv_rc_pingpong.1 man/ibv_uc_pingpong.1 man/ibv_ud_pingpong.1 \ diff --git a/include/infiniband/ofverbs-defs.h b/include/infiniband/ofverbs-defs.h new file mode 100644 index 000..c260006 --- /dev/null +++ b/include/infiniband/ofverbs-defs.h @@ -0,0 +1,54 @@ +/* + * Copyright (c) 2004, 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2004, 2011 Intel Corporation. All rights reserved. + * Copyright (c) 2005, 2006, 2007 Cisco Systems, Inc. All rights reserved. + * Copyright (c) 2005 PathScale, Inc. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + *copyright notice, this list of conditions and the following + *disclaimer. + * + * - Redistributions in binary form must reproduce the above + *copyright notice, this list of conditions and the following + *disclaimer in the documentation and/or other materials + *provided with the distribution. + * + * THE SOFTWARE IS PROVIDED AS IS, WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + */ + +#ifndef OFVERBS_DEFS_H +#define OFVERBS_DEFS_H + +/* + * Include this file before verbs.h or at the top of verbs.h. This will + * map the names of the ibv_* structures to _real_ibv_*. The + * ofv_* structure names are then mapped to ibv_*. + */ +#ifdef OFED_VERBS + +#define ibv_srq
