Does XRC work with the UDCM CPC?
On Nov 14, 2013, at 9:35 AM, Ralph Castain <r...@open-mpi.org> wrote: > I think the problems in udcm were fixed by Nathan quite some time ago, but > never moved to 1.7 as everyone was told that the connect code in openib was > already deprecated pending merge with the new ofacm common code. Looking over > at that area, I see only oob and xoob - so if the users of the common ofacm > code are finding that it works, the simple answer may just be to finally > complete the switchover. > > Meantime, perhaps someone can CMR and review a copying of the udcm cpc to the > 1.7 branch? > > > On Nov 14, 2013, at 5:14 AM, Joshua Ladd <josh...@mellanox.com> wrote: > >> Um, no. It's supposed to work with UDCM which doesn't appear to be enabled >> in 1.7. >> >> Per Ralph's comment to me last night: >> >> "... you cannot use the oob connection manager. It doesn't work and was >> deprecated. You must use udcm, which is why things are supposed to be set to >> do so by default. Please check the openib connect priorities and correct >> them if necessary." >> >> However, it's never been enabled in 1.7 - don't know what "borked" means, >> and from what Devendar tells me, several UDCM commits that are in the trunk >> have not been pushed over to 1.7: >> >> So, as of this moment, OpenIB BTL is essentially dead-in-the-water in 1.7. >> >> >> >> [enable_connectx_xrc="$enableval"], >> [enable_connectx_xrc="yes"]) >> # >> # Unconnect Datagram (UD) based connection manager >> # >> # AC_ARG_ENABLE([openib-udcm], >> # [AC_HELP_STRING([--enable-openib-udcm], >> # [Enable datagram connection support in openib BTL >> (default: enabled)])], >> # [enable_openib_udcm="$enableval"], >> [enable_openib_udcm="yes"]) >> # Per discussion with Ralph and Nathan, disable UDCM for now. >> # It's borked and needs some surgery to get back on its feet. >> enable_openib_udcm=no >> >> >> Josh >> >> >> -----Original Message----- >> From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Jeff Squyres >> (jsquyres) >> Sent: Thursday, November 14, 2013 6:44 AM >> To: <de...@open-mpi.org> >> Subject: Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r29703 - in trunk: >> contrib/platform/iu/odin ompi/mca/btl/openib ompi/mca/btl/openib/connect >> >> Does the openib *only* work with RDMACM now? >> >> That's surprising (and bad!). >> >> Did someone ask Mellanox about fixing the OOB and XOOB CPCs? >> >> >> On Nov 13, 2013, at 11:16 PM, svn-commit-mai...@open-mpi.org wrote: >> >>> Author: rhc (Ralph Castain) >>> Date: 2013-11-13 23:16:53 EST (Wed, 13 Nov 2013) >>> New Revision: 29703 >>> URL: https://svn.open-mpi.org/trac/ompi/changeset/29703 >>> >>> Log: >>> Given that the oob and xoob cpc's are no longer operable and haven't been >>> since the OOB update, remove them to avoid confusion >>> >>> cmr:v1.7.4:reviewer=hjelmn:subject=Remove stale cpcs from openib >>> >>> Deleted: >>> trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c >>> trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.h >>> trunk/ompi/mca/btl/openib/connect/btl_openib_connect_xoob.c >>> trunk/ompi/mca/btl/openib/connect/btl_openib_connect_xoob.h >>> Text files modified: >>> trunk/contrib/platform/iu/odin/optimized.conf | 1 >>> >>> trunk/contrib/platform/iu/odin/static.conf | 1 >>> >>> trunk/ompi/mca/btl/openib/Makefile.am | 10 >>> >>> trunk/ompi/mca/btl/openib/connect/btl_openib_connect_base.c | 14 >>> >>> /dev/null | 975 >>> --------------------------------- >>> /dev/null | 18 >>> >>> /dev/null | 1150 >>> ---------------------------------------- >>> /dev/null | 19 >>> >>> 8 files changed, 5 insertions(+), 2183 deletions(-) >>> >>> Modified: trunk/contrib/platform/iu/odin/optimized.conf >>> ============================================================================== >>> --- trunk/contrib/platform/iu/odin/optimized.conf Wed Nov 13 19:34:15 >>> 2013 (r29702) >>> +++ trunk/contrib/platform/iu/odin/optimized.conf 2013-11-13 23:16:53 EST >>> (Wed, 13 Nov 2013) (r29703) >>> @@ -80,7 +80,6 @@ >>> >>> ## Setup OpenIB >>> btl_openib_want_fork_support = 0 >>> -btl_openib_cpc_include = oob >>> #btl_openib_receive_queues = >>> P,128,256,64,32,32:S,2048,1024,128,32:S,12288,1024,128,32:S,65536,1024,128,32 >>> >>> >>> ## Setup TCP >>> >>> Modified: trunk/contrib/platform/iu/odin/static.conf >>> ============================================================================== >>> --- trunk/contrib/platform/iu/odin/static.conf Wed Nov 13 19:34:15 >>> 2013 (r29702) >>> +++ trunk/contrib/platform/iu/odin/static.conf 2013-11-13 23:16:53 EST >>> (Wed, 13 Nov 2013) (r29703) >>> @@ -80,7 +80,6 @@ >>> >>> ## Setup OpenIB >>> btl_openib_want_fork_support = 0 >>> -btl_openib_cpc_include = oob >>> #btl_openib_receive_queues = >>> P,128,256,64,32,32:S,2048,1024,128,32:S,12288,1024,128,32:S,65536,1024,128,32 >>> >>> >>> ## Setup TCP >>> >>> Modified: trunk/ompi/mca/btl/openib/Makefile.am >>> ============================================================================== >>> --- trunk/ompi/mca/btl/openib/Makefile.am Wed Nov 13 19:34:15 2013 >>> (r29702) >>> +++ trunk/ompi/mca/btl/openib/Makefile.am 2013-11-13 23:16:53 EST (Wed, >>> 13 Nov 2013) (r29703) >>> @@ -14,6 +14,7 @@ >>> # Copyright (c) 2011 NVIDIA Corporation. All rights reserved. >>> # Copyright (c) 2011 Mellanox Technologies. All rights reserved. >>> # Copyright (c) 2012 Oak Ridge National Laboratory. All rights >>> reserved >>> +# Copyright (c) 2013 Intel, Inc. All rights reserved. >>> # $COPYRIGHT$ >>> # >>> # Additional copyrights may follow >>> @@ -60,8 +61,6 @@ >>> btl_openib_ip.c \ >>> connect/base.h \ >>> connect/btl_openib_connect_base.c \ >>> - connect/btl_openib_connect_oob.c \ >>> - connect/btl_openib_connect_oob.h \ >>> connect/btl_openib_connect_empty.c \ >>> connect/btl_openib_connect_empty.h \ >>> connect/connect.h >>> @@ -73,13 +72,6 @@ >>> btl_openib_failover.h >>> endif >>> >>> -# If we have XRC support, build that CPC >>> -if MCA_btl_openib_have_xrc >>> -sources += \ >>> - connect/btl_openib_connect_xoob.c \ >>> - connect/btl_openib_connect_xoob.h >>> -endif >>> - >>> # If we have rdmacm support, build that CPC >>> if MCA_btl_openib_have_rdmacm >>> sources += \ >>> >>> Modified: trunk/ompi/mca/btl/openib/connect/btl_openib_connect_base.c >>> ============================================================================== >>> --- trunk/ompi/mca/btl/openib/connect/btl_openib_connect_base.c Wed Nov >>> 13 19:34:15 2013 (r29702) >>> +++ trunk/ompi/mca/btl/openib/connect/btl_openib_connect_base.c >>> 2013-11-13 23:16:53 EST (Wed, 13 Nov 2013) (r29703) >>> @@ -17,11 +17,7 @@ >>> #include "btl_openib.h" >>> #include "btl_openib_proc.h" >>> #include "connect/base.h" >>> -#include "connect/btl_openib_connect_oob.h" >>> #include "connect/btl_openib_connect_empty.h" >>> -#if HAVE_XRC >>> -#include "connect/btl_openib_connect_xoob.h" >>> -#endif >>> #if OMPI_HAVE_RDMACM && OPAL_HAVE_THREADS >>> #include "connect/btl_openib_connect_rdmacm.h" >>> #endif >>> @@ -37,15 +33,13 @@ >>> * Array of all possible connection functions >>> */ >>> static ompi_btl_openib_connect_base_component_t *all[] = { >>> - &ompi_btl_openib_connect_oob, >>> + /* Always have an entry here so that the CP indexes will always be >>> + the same: OOB has been removed, so use the "empty" CPC */ >>> + &ompi_btl_openib_connect_empty, >>> >>> /* Always have an entry here so that the CP indexes will always be >>> - the same: if XRC is not available, use the "empty" CPC */ >>> -#if HAVE_XRC >>> - &ompi_btl_openib_connect_xoob, >>> -#else >>> + the same: XOOB has been removed, so use the "empty" CPC */ >>> &ompi_btl_openib_connect_empty, >>> -#endif >>> >>> /* Always have an entry here so that the CP indexes will always be >>> the same: if RDMA CM is not available, use the "empty" CPC */ >>> >>> Deleted: trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c >>> ============================================================================== >>> --- trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c >>> 2013-11-13 23:16:53 EST (Wed, 13 Nov 2013) (r29702) >>> +++ /dev/null 00:00:00 1970 (deleted) >>> @@ -1,975 +0,0 @@ >>> -/* >>> - * Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana >>> - * University Research and Technology >>> - * Corporation. All rights reserved. >>> - * Copyright (c) 2004-2011 The University of Tennessee and The University >>> - * of Tennessee Research Foundation. All rights >>> - * reserved. >>> - * Copyright (c) 2004-2005 High Performance Computing Center Stuttgart, >>> - * University of Stuttgart. All rights reserved. >>> - * Copyright (c) 2004-2005 The Regents of the University of California. >>> - * All rights reserved. >>> - * Copyright (c) 2006-2013 Cisco Systems, Inc. All rights reserved. >>> - * Copyright (c) 2006-2012 Los Alamos National Security, LLC. All rights >>> - * reserved. >>> - * Copyright (c) 2008-2013 Mellanox Technologies. All rights reserved. >>> - * Copyright (c) 2009-2011 IBM Corporation. All rights reserved. >>> - * Copyright (c) 2010 Oracle and/or its affiliates. All rights >>> reserved >>> - * >>> - * $COPYRIGHT$ >>> - * >>> - * Additional copyrights may follow >>> - * >>> - * $HEADER$ >>> - */ >>> - >>> -#include "ompi_config.h" >>> - >>> -#include "opal/dss/dss.h" >>> -#include "opal_stdint.h" >>> -#include "opal/util/error.h" >>> -#include "opal/util/output.h" >>> -#include "opal/util/show_help.h" >>> - >>> -#include "ompi/mca/rte/rte.h" >>> -#include "btl_openib.h" >>> -#include "btl_openib_endpoint.h" >>> -#include "btl_openib_proc.h" >>> -#include "connect/connect.h" >>> - >>> -#if (ENABLE_DYNAMIC_SL) >>> -#include "connect/btl_openib_connect_sl.h" >>> -#endif >>> - >>> -#ifdef HAVE_UNISTD_H >>> -#include <unistd.h> >>> -#endif >>> - >>> -typedef enum { >>> - ENDPOINT_CONNECT_REQUEST, >>> - ENDPOINT_CONNECT_RESPONSE, >>> - ENDPOINT_CONNECT_ACK >>> -} connect_message_type_t; >>> - >>> -static int oob_priority = 0; >>> -static bool rml_recv_posted = false; >>> - >>> -static void oob_component_register(void); >>> -static int oob_component_query(mca_btl_openib_module_t *openib_btl, >>> - ompi_btl_openib_connect_base_module_t >>> **cpc); >>> -static int oob_component_finalize(void); >>> - >>> -static int oob_module_start_connect(ompi_btl_openib_connect_base_module_t >>> *cpc, >>> - mca_btl_base_endpoint_t *endpoint); >>> -static int reply_start_connect(mca_btl_openib_endpoint_t *endpoint, >>> - mca_btl_openib_rem_info_t *rem_info); >>> -static int set_remote_info(mca_btl_base_endpoint_t* endpoint, >>> - mca_btl_openib_rem_info_t* rem_info); >>> -static int qp_connect_all(mca_btl_base_endpoint_t* endpoint); >>> -static int qp_create_all(mca_btl_base_endpoint_t* endpoint); >>> -static int qp_create_one(mca_btl_base_endpoint_t* endpoint, int qp, >>> - struct ibv_srq *srq, uint32_t max_recv_wr, uint32_t max_send_wr); >>> -static int send_connect_data(mca_btl_base_endpoint_t* endpoint, >>> - uint8_t message_type); >>> - >>> -static void rml_send_cb(int status, ompi_process_name_t* endpoint, >>> - opal_buffer_t* buffer, ompi_rml_tag_t tag, >>> - void* cbdata); >>> -static void rml_recv_cb(int status, ompi_process_name_t* process_name, >>> - opal_buffer_t* buffer, ompi_rml_tag_t tag, >>> - void* cbdata); >>> - >>> -/* >>> - * The "component" struct -- the top-level function pointers for the >>> - * oob connection scheme. >>> - */ >>> -ompi_btl_openib_connect_base_component_t ompi_btl_openib_connect_oob = { >>> - "oob", >>> - /* Register */ >>> - oob_component_register, >>> - /* Init */ >>> - NULL, >>> - /* Query */ >>> - oob_component_query, >>> - /* Finalize */ >>> - oob_component_finalize, >>> -}; >>> - >>> -/* Open - this functions sets up any oob specific commandline params */ >>> -static void oob_component_register(void) >>> -{ >>> - /* the priority is initialized in the declaration above */ >>> - (void) >>> mca_base_component_var_register(&mca_btl_openib_component.super.btl_version, >>> - "connect_oob_priority", >>> - "The selection method priority >>> for oob", >>> - MCA_BASE_VAR_TYPE_INT, NULL, 0, >>> 0, >>> - OPAL_INFO_LVL_9, >>> - MCA_BASE_VAR_SCOPE_READONLY, >>> - &oob_priority); >>> - >>> - if (oob_priority > 100) { >>> - oob_priority = 100; >>> - } else if (oob_priority < -1) { >>> - oob_priority = -1; >>> - } >>> -} >>> - >>> -/* >>> - * Init function. Post non-blocking RML receive to accept incoming >>> - * connection requests. >>> - */ >>> -static int oob_component_query(mca_btl_openib_module_t *btl, >>> - ompi_btl_openib_connect_base_module_t **cpc) >>> -{ >>> - /* If we have the transport_type member, check to ensure we're on >>> - IB (this CPC will not work with iWarp). If we do not have the >>> - transport_type member, then we must be < OFED v1.2, and >>> - therefore we must be IB. */ >>> -#if defined(HAVE_STRUCT_IBV_DEVICE_TRANSPORT_TYPE) && >>> HAVE_DECL_IBV_LINK_LAYER_ETHERNET >>> - if (BTL_OPENIB_CONNECT_BASE_CHECK_IF_NOT_IB(btl)) { >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: oob CPC only supported on >>> InfiniBand; skipped on %s:%d", >>> - ibv_get_device_name(btl->device->ib_dev), >>> - btl->port_num); >>> - return OMPI_ERR_NOT_SUPPORTED; >>> - } >>> -#endif >>> - >>> - if (mca_btl_openib_component.num_xrc_qps > 0) { >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: oob CPC not supported with XRC >>> receive queues, please try xoob CPC; skipped on %s:%d", >>> - ibv_get_device_name(btl->device->ib_dev), >>> - btl->port_num); >>> - return OMPI_ERR_NOT_SUPPORTED; >>> - } >>> - /* If this btl supports OOB, then post the RML message. But >>> - ensure to only post it *once*, because another btl may have >>> - come in before this and already posted it. */ >>> - if (!rml_recv_posted) { >>> - ompi_rte_recv_buffer_nb(OMPI_NAME_WILDCARD, >>> - OMPI_RML_TAG_OPENIB, >>> - OMPI_RML_PERSISTENT, >>> - rml_recv_cb, >>> - NULL); >>> - rml_recv_posted = true; >>> - } >>> - >>> - *cpc = (ompi_btl_openib_connect_base_module_t *) >>> malloc(sizeof(ompi_btl_openib_connect_base_module_t)); >>> - if (NULL == *cpc) { >>> - ompi_rte_recv_cancel(OMPI_NAME_WILDCARD, OMPI_RML_TAG_OPENIB); >>> - rml_recv_posted = false; >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: oob CPC system error (malloc >>> failed)"); >>> - return OMPI_ERR_OUT_OF_RESOURCE; >>> - } >>> - >>> - if (oob_priority > 100) { >>> - oob_priority = 100; >>> - } else if (oob_priority < -1) { >>> - oob_priority = -1; >>> - } >>> - >>> - (*cpc)->data.cbm_component = &ompi_btl_openib_connect_oob; >>> - (*cpc)->data.cbm_priority = oob_priority; >>> - (*cpc)->data.cbm_modex_message = NULL; >>> - (*cpc)->data.cbm_modex_message_len = 0; >>> - >>> - (*cpc)->cbm_endpoint_init = NULL; >>> - (*cpc)->cbm_start_connect = oob_module_start_connect; >>> - (*cpc)->cbm_endpoint_finalize = NULL; >>> - (*cpc)->cbm_finalize = NULL; >>> - (*cpc)->cbm_uses_cts = false; >>> - >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: oob CPC available for use on %s:%d", >>> - ibv_get_device_name(btl->device->ib_dev), >>> - btl->port_num); >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* >>> - * Connect function. Start initiation of connections to a remote >>> - * peer. We send our Queue Pair information over the RML/OOB >>> - * communication mechanism. On completion of our send, a send >>> - * completion handler is called. >>> - */ >>> -static int oob_module_start_connect(ompi_btl_openib_connect_base_module_t >>> *cpc, >>> - mca_btl_base_endpoint_t *endpoint) >>> -{ >>> - int rc; >>> - >>> - if (OMPI_SUCCESS != (rc = qp_create_all(endpoint))) { >>> - return rc; >>> - } >>> - >>> - /* Send connection info over to remote endpoint */ >>> - endpoint->endpoint_state = MCA_BTL_IB_CONNECTING; >>> - if (OMPI_SUCCESS != >>> - (rc = send_connect_data(endpoint, ENDPOINT_CONNECT_REQUEST))) { >>> - BTL_ERROR(("error sending connect request, error code %d", rc)); >>> - return rc; >>> - } >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* >>> - * Component finalize function. Cleanup RML non-blocking receive. >>> - */ >>> -static int oob_component_finalize(void) >>> -{ >>> - if (rml_recv_posted) { >>> - ompi_rte_recv_cancel(OMPI_NAME_WILDCARD, OMPI_RML_TAG_OPENIB); >>> - rml_recv_posted = false; >>> - } >>> -#if (ENABLE_DYNAMIC_SL) >>> - btl_openib_connect_sl_finalize(); >>> -#endif >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/**************************************************************************/ >>> - >>> -/* >>> - * Reply to a `start - connect' message >>> - */ >>> -static int reply_start_connect(mca_btl_openib_endpoint_t *endpoint, >>> - mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - int rc; >>> - >>> - BTL_VERBOSE(("Initialized QPs, LID = %d", >>> - ((mca_btl_openib_module_t*)endpoint->endpoint_btl)->lid)); >>> - >>> - /* Create local QP's and post receive resources */ >>> - if (OMPI_SUCCESS != (rc = qp_create_all(endpoint))) { >>> - return rc; >>> - } >>> - >>> - /* Set the remote side info */ >>> - set_remote_info(endpoint, rem_info); >>> - >>> - /* Connect to remote endpoint qp's */ >>> - if (OMPI_SUCCESS != (rc = qp_connect_all(endpoint))) { >>> - return rc; >>> - } >>> - >>> - /* Send connection info over to remote endpoint */ >>> - endpoint->endpoint_state = MCA_BTL_IB_CONNECT_ACK; >>> - if (OMPI_SUCCESS != >>> - (rc = send_connect_data(endpoint, ENDPOINT_CONNECT_RESPONSE))) { >>> - BTL_ERROR(("error in endpoint send connect request error code is >>> %d", >>> - rc)); >>> - return rc; >>> - } >>> - return OMPI_SUCCESS; >>> -} >>> - >>> - >>> -static int set_remote_info(mca_btl_base_endpoint_t* endpoint, >>> - mca_btl_openib_rem_info_t* rem_info) >>> -{ >>> - /* Free up the memory pointed to by rem_qps before overwriting the >>> pointer >>> - in the following memcpy */ >>> - free(endpoint->rem_info.rem_qps); >>> - >>> - /* copy the rem_info stuff */ >>> - memcpy(&((mca_btl_openib_endpoint_t*) endpoint)->rem_info, >>> - rem_info, sizeof(mca_btl_openib_rem_info_t)); >>> - >>> - BTL_VERBOSE(("Setting QP info, LID = %d", >>> endpoint->rem_info.rem_lid)); >>> - return OMPI_SUCCESS; >>> -} >>> - >>> - >>> -/* >>> - * Connect the local ends of all qp's to the remote side >>> - */ >>> -static int qp_connect_all(mca_btl_openib_endpoint_t *endpoint) >>> -{ >>> - int i; >>> - mca_btl_openib_module_t* openib_btl = >>> - (mca_btl_openib_module_t*)endpoint->endpoint_btl; >>> - >>> - for (i = 0; i < mca_btl_openib_component.num_qps; i++) { >>> - struct ibv_qp_attr attr; >>> - struct ibv_qp* qp = endpoint->qps[i].qp->lcl_qp; >>> - enum ibv_mtu mtu = (enum ibv_mtu) ((openib_btl->device->mtu < >>> endpoint->rem_info.rem_mtu) ? >>> - openib_btl->device->mtu : endpoint->rem_info.rem_mtu) ; >>> - >>> - memset(&attr, 0, sizeof(attr)); >>> - attr.qp_state = IBV_QPS_RTR; >>> - attr.path_mtu = mtu; >>> - attr.dest_qp_num = endpoint->rem_info.rem_qps[i].rem_qp_num; >>> - attr.rq_psn = endpoint->rem_info.rem_qps[i].rem_psn; >>> - attr.max_dest_rd_atomic = >>> mca_btl_openib_component.ib_max_rdma_dst_ops; >>> - attr.min_rnr_timer = mca_btl_openib_component.ib_min_rnr_timer; >>> - attr.ah_attr.is_global = 0; >>> - attr.ah_attr.dlid = endpoint->rem_info.rem_lid; >>> - attr.ah_attr.src_path_bits = openib_btl->src_path_bits; >>> - attr.ah_attr.port_num = openib_btl->port_num; >>> - attr.ah_attr.sl = >>> mca_btl_openib_component.ib_service_level; >>> - >>> -#if (ENABLE_DYNAMIC_SL) >>> - /* if user enabled dynamic SL, get it from PathRecord */ >>> - if (0 != mca_btl_openib_component.ib_path_record_service_level) { >>> - int rc = btl_openib_connect_get_pathrecord_sl(qp->context, >>> - >>> attr.ah_attr.port_num, >>> - openib_btl->lid, >>> - >>> attr.ah_attr.dlid); >>> - if (OMPI_ERROR == rc) { >>> - return OMPI_ERROR; >>> - } >>> - attr.ah_attr.sl = rc; >>> - } >>> -#endif >>> - >>> - /* JMS to be filled in later dynamically */ >>> - attr.ah_attr.static_rate = 0; >>> - >>> - if (mca_btl_openib_component.verbose) { >>> - BTL_OUTPUT(("Set MTU to IBV value %d (%s bytes)", mtu, >>> - (mtu == IBV_MTU_256) ? "256" : >>> - (mtu == IBV_MTU_512) ? "512" : >>> - (mtu == IBV_MTU_1024) ? "1024" : >>> - (mtu == IBV_MTU_2048) ? "2048" : >>> - (mtu == IBV_MTU_4096) ? "4096" : >>> - "unknown (!)")); >>> - } >>> - >>> - if (ibv_modify_qp(qp, &attr, >>> - IBV_QP_STATE | >>> - IBV_QP_AV | >>> - IBV_QP_PATH_MTU | >>> - IBV_QP_DEST_QPN | >>> - IBV_QP_RQ_PSN | >>> - IBV_QP_MAX_DEST_RD_ATOMIC | >>> - IBV_QP_MIN_RNR_TIMER)) { >>> - BTL_ERROR(("error modifing QP to RTR errno says %s", >>> - strerror(errno))); >>> - return OMPI_ERROR; >>> - } >>> - attr.qp_state = IBV_QPS_RTS; >>> - attr.timeout = mca_btl_openib_component.ib_timeout; >>> - attr.retry_cnt = mca_btl_openib_component.ib_retry_count; >>> - /* On PP QPs we have SW flow control, no need for rnr retries. >>> Setting >>> - * it to zero helps to catch bugs */ >>> - attr.rnr_retry = BTL_OPENIB_QP_TYPE_PP(i) ? 0 : >>> - mca_btl_openib_component.ib_rnr_retry; >>> - attr.sq_psn = endpoint->qps[i].qp->lcl_psn; >>> - attr.max_rd_atomic = mca_btl_openib_component.ib_max_rdma_dst_ops; >>> - if (ibv_modify_qp(qp, &attr, >>> - IBV_QP_STATE | >>> - IBV_QP_TIMEOUT | >>> - IBV_QP_RETRY_CNT | >>> - IBV_QP_RNR_RETRY | >>> - IBV_QP_SQ_PSN | >>> - IBV_QP_MAX_QP_RD_ATOMIC)) { >>> - BTL_ERROR(("error modifying QP to RTS errno says %s", >>> - strerror(errno))); >>> - return OMPI_ERROR; >>> - } >>> - } >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> - >>> -static void permute_array(int *permuted_qps, int nqps) >>> -{ >>> - int i; >>> - int idx; >>> - int tmp; >>> - int control[nqps]; >>> - >>> - for (i = 0; i < nqps; i++) { >>> - permuted_qps[i] = i; >>> - control[i] = 0; >>> - } >>> - >>> - for (i = 0; i < nqps - 1; i++) { >>> - idx = i + random() % (nqps - i); >>> - tmp = permuted_qps[i]; >>> - permuted_qps[i] = permuted_qps[idx]; >>> - permuted_qps[idx] = tmp; >>> - } >>> - >>> - /* verify that permutation is ok: */ >>> - for (i = 0; i < nqps; i++) { >>> - control[permuted_qps[i]] ++; >>> - } >>> - for (i = 0; i < nqps; i++) { >>> - if (control[i] != 1) { >>> - BTL_VERBOSE(("bad permutation detected: ")); >>> - for (i = 0; i < nqps; i++) BTL_VERBOSE(("%d ", >>> permuted_qps[i])); >>> - BTL_VERBOSE(("\n")); >>> - abort(); >>> - } >>> - } >>> -} >>> - >>> - >>> -/* >>> - * Create the local side of all the qp's. The remote sides will be >>> - * connected later. >>> - */ >>> -static int qp_create_all(mca_btl_base_endpoint_t* endpoint) >>> -{ >>> - int qp, rc, pp_qp_num = 0; >>> - int32_t rd_rsv_total = 0; >>> - >>> - int rand_qpns[mca_btl_openib_component.num_qps]; >>> - int i; >>> - >>> - permute_array(rand_qpns, mca_btl_openib_component.num_qps); >>> - >>> - >>> - for (qp = 0; qp < mca_btl_openib_component.num_qps; ++qp) >>> - if(BTL_OPENIB_QP_TYPE_PP(qp)) { >>> - rd_rsv_total += >>> - mca_btl_openib_component.qp_infos[qp].u.pp_qp.rd_rsv; >>> - pp_qp_num++; >>> - } >>> - >>> - /* if there is no pp QPs we still need reserved WQE for eager rdma flow >>> - * control */ >>> - if(0 == pp_qp_num && true == endpoint->use_eager_rdma) >>> - pp_qp_num = 1; >>> - >>> - for (i = 0; i < mca_btl_openib_component.num_qps; ++i) { >>> - struct ibv_srq *srq = NULL; >>> - uint32_t max_recv_wr, max_send_wr; >>> - int32_t rd_rsv, rd_num_credits; >>> - >>> - qp = rand_qpns[i]; >>> - /* QP used for SW flow control need some additional recourses */ >>> - if(qp == mca_btl_openib_component.credits_qp) { >>> - rd_rsv = rd_rsv_total; >>> - rd_num_credits = pp_qp_num; >>> - } else { >>> - rd_rsv = rd_num_credits = 0; >>> - } >>> - >>> - if(BTL_OPENIB_QP_TYPE_PP(qp)) { >>> - max_recv_wr = mca_btl_openib_component.qp_infos[qp].rd_num + >>> rd_rsv; >>> - max_send_wr = mca_btl_openib_component.qp_infos[qp].rd_num + >>> - rd_num_credits; >>> - } else { >>> - srq = endpoint->endpoint_btl->qps[qp].u.srq_qp.srq; >>> - /* no receives are posted to SRQ qp */ >>> - max_recv_wr = 0; >>> - max_send_wr = >>> mca_btl_openib_component.qp_infos[qp].u.srq_qp.sd_max >>> - + rd_num_credits; >>> - } >>> - >>> - rc = qp_create_one(endpoint, qp, srq, max_recv_wr, max_send_wr); >>> - if (OMPI_SUCCESS != rc) { >>> - return rc; >>> - } >>> - } >>> - >>> - /* Now that all the qp's are created locally, post some receive >>> - buffers, setup credits, etc. */ >>> - return mca_btl_openib_endpoint_post_recvs(endpoint); >>> -} >>> - >>> - >>> -/* Returns max inlne size for qp #N */ >>> -static uint32_t max_inline_size(int qp, mca_btl_openib_device_t *device) >>> -{ >>> - if (mca_btl_openib_component.qp_infos[qp].size <= >>> device->max_inline_data) { >>> - /* If qp message size is smaller than max_inline_data, >>> - * we should enable inline messages */ >>> - return mca_btl_openib_component.qp_infos[qp].size; >>> - } else if (mca_btl_openib_component.rdma_qp == qp || 0 == qp) { >>> - /* If qp message size is bigger that max_inline_data, we >>> - * should enable inline messages only for RDMA QP (for PUT/GET >>> - * fin messages) and for the first qp */ >>> - return device->max_inline_data; >>> - } >>> - /* Otherway it is no reason for inline */ >>> - return 0; >>> -} >>> - >>> -/* >>> - * Create the local side of one qp. The remote side will be connected >>> - * later. >>> - */ >>> -static int qp_create_one(mca_btl_base_endpoint_t* endpoint, int qp, >>> - struct ibv_srq *srq, uint32_t max_recv_wr, uint32_t max_send_wr) >>> -{ >>> - mca_btl_openib_module_t *openib_btl = endpoint->endpoint_btl; >>> - struct ibv_qp *my_qp; >>> - struct ibv_qp_init_attr init_attr; >>> - struct ibv_qp_attr attr; >>> - size_t req_inline; >>> - >>> - memset(&init_attr, 0, sizeof(init_attr)); >>> - memset(&attr, 0, sizeof(attr)); >>> - >>> - init_attr.qp_type = IBV_QPT_RC; >>> - init_attr.send_cq = openib_btl->device->ib_cq[BTL_OPENIB_RDMA_QP(qp) ? >>> BTL_OPENIB_HP_CQ: BTL_OPENIB_LP_CQ]; >>> - init_attr.recv_cq = openib_btl->device->ib_cq[qp_cq_prio(qp)]; >>> - init_attr.srq = srq; >>> - init_attr.cap.max_inline_data = req_inline = >>> - max_inline_size(qp, openib_btl->device); >>> - init_attr.cap.max_send_sge = 1; >>> - init_attr.cap.max_recv_sge = 1; /* we do not use SG list */ >>> - if(BTL_OPENIB_QP_TYPE_PP(qp)) { >>> - init_attr.cap.max_recv_wr = max_recv_wr; >>> - } else { >>> - init_attr.cap.max_recv_wr = 0; >>> - } >>> - init_attr.cap.max_send_wr = max_send_wr; >>> - >>> - my_qp = ibv_create_qp(openib_btl->device->ib_pd, &init_attr); >>> - >>> - if (NULL == my_qp) { >>> - opal_show_help("help-mpi-btl-openib-cpc-base.txt", >>> - "ibv_create_qp failed", true, >>> - ompi_process_info.nodename, >>> - ibv_get_device_name(openib_btl->device->ib_dev), >>> - "Reliable connected (RC)"); >>> - return OMPI_ERROR; >>> - } >>> - endpoint->qps[qp].qp->lcl_qp = my_qp; >>> - >>> - if (init_attr.cap.max_inline_data < req_inline) { >>> - endpoint->qps[qp].ib_inline_max = init_attr.cap.max_inline_data; >>> - opal_show_help("help-mpi-btl-openib-cpc-base.txt", >>> - "inline truncated", true, >>> ompi_process_info.nodename, >>> - ibv_get_device_name(openib_btl->device->ib_dev), >>> - openib_btl->port_num, >>> - req_inline, init_attr.cap.max_inline_data); >>> - } else { >>> - endpoint->qps[qp].ib_inline_max = req_inline; >>> - } >>> - >>> - attr.qp_state = IBV_QPS_INIT; >>> - attr.pkey_index = openib_btl->pkey_index; >>> - attr.port_num = openib_btl->port_num; >>> - attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | >>> IBV_ACCESS_REMOTE_READ; >>> - >>> - if (ibv_modify_qp(endpoint->qps[qp].qp->lcl_qp, >>> - &attr, >>> - IBV_QP_STATE | >>> - IBV_QP_PKEY_INDEX | >>> - IBV_QP_PORT | >>> - IBV_QP_ACCESS_FLAGS )) { >>> - BTL_ERROR(("error modifying qp to INIT errno says %s", >>> strerror(errno))); >>> - return OMPI_ERROR; >>> - } >>> - >>> - /* Setup meta data on the endpoint */ >>> - endpoint->qps[qp].qp->lcl_psn = lrand48() & 0xffffff; >>> - endpoint->qps[qp].credit_frag = NULL; >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> - >>> -/* >>> - * RML send connect information to remote endpoint >>> - */ >>> -static int send_connect_data(mca_btl_base_endpoint_t* endpoint, >>> - uint8_t message_type) >>> -{ >>> - opal_buffer_t* buffer = OBJ_NEW(opal_buffer_t); >>> - int rc; >>> - >>> - if (NULL == buffer) { >>> - OMPI_ERROR_LOG(OMPI_ERR_OUT_OF_RESOURCE); >>> - return OMPI_ERR_OUT_OF_RESOURCE; >>> - } >>> - >>> - /* pack the info in the send buffer */ >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT8)); >>> - rc = opal_dss.pack(buffer, &message_type, 1, OPAL_UINT8); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT64)); >>> - rc = opal_dss.pack(buffer, &endpoint->subnet_id, 1, OPAL_UINT64); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - if (message_type != ENDPOINT_CONNECT_REQUEST) { >>> - /* send the QP connect request info we respond to */ >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, >>> - &endpoint->rem_info.rem_qps[0].rem_qp_num, 1, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT16)); >>> - rc = opal_dss.pack(buffer, &endpoint->rem_info.rem_lid, 1, >>> OPAL_UINT16); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - >>> - if (message_type != ENDPOINT_CONNECT_ACK) { >>> - int qp; >>> - /* stuff all the QP info into the buffer */ >>> - for (qp = 0; qp < mca_btl_openib_component.num_qps; qp++) { >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, >>> &endpoint->qps[qp].qp->lcl_qp->qp_num, >>> - 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &endpoint->qps[qp].qp->lcl_psn, 1, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT16)); >>> - rc = opal_dss.pack(buffer, &endpoint->endpoint_btl->lid, 1, >>> OPAL_UINT16); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &endpoint->endpoint_btl->device->mtu, 1, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &endpoint->index, 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - >>> - /* send to remote endpoint */ >>> - rc = >>> ompi_rte_send_buffer_nb(&endpoint->endpoint_proc->proc_ompi->proc_name, >>> - buffer, OMPI_RML_TAG_OPENIB, >>> - rml_send_cb, NULL); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("Sent QP Info, LID = %d, SUBNET = %" PRIx64 "\n", >>> - endpoint->endpoint_btl->lid, >>> - endpoint->subnet_id)); >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> - >>> -/* >>> - * Callback when we have finished RML sending the connect data to a >>> - * remote peer >>> - */ >>> -static void rml_send_cb(int status, ompi_process_name_t* endpoint, >>> - opal_buffer_t* buffer, ompi_rml_tag_t tag, >>> - void* cbdata) >>> -{ >>> - OBJ_RELEASE(buffer); >>> -} >>> - >>> - >>> -/* >>> - * Non blocking RML recv callback. Read incoming QP and other info, >>> - * and if this endpoint is trying to connect, reply with our QP info, >>> - * otherwise try to modify QP's and establish reliable connection >>> - */ >>> -static void rml_recv_cb(int status, ompi_process_name_t* process_name, >>> - opal_buffer_t* buffer, ompi_rml_tag_t tag, >>> - void* cbdata) >>> -{ >>> - mca_btl_openib_proc_t *ib_proc; >>> - mca_btl_openib_endpoint_t *ib_endpoint = NULL; >>> - int endpoint_state; >>> - int rc; >>> - uint32_t i, lcl_qp = 0; >>> - uint16_t lcl_lid = 0; >>> - int32_t cnt = 1; >>> - mca_btl_openib_rem_info_t rem_info; >>> - uint8_t message_type; >>> - bool master; >>> - >>> - /* We later memcpy this whole structure. Make sure >>> - that all the parameters are initialized, especially >>> - the pointers */ >>> - memset(&rem_info,0, sizeof(rem_info)); >>> - >>> - /* start by unpacking data first so we know who is knocking at >>> - our door */ >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT8)); >>> - rc = opal_dss.unpack(buffer, &message_type, &cnt, OPAL_UINT8); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT64)); >>> - rc = opal_dss.unpack(buffer, &rem_info.rem_subnet_id, &cnt, >>> OPAL_UINT64); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - >>> - if (ENDPOINT_CONNECT_REQUEST != message_type) { >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &lcl_qp, &cnt, OPAL_UINT32); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT16)); >>> - rc = opal_dss.unpack(buffer, &lcl_lid, &cnt, OPAL_UINT16); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - } >>> - if (ENDPOINT_CONNECT_ACK != message_type) { >>> - int qp; >>> - /* get ready for the data */ >>> - rem_info.rem_qps = >>> - (mca_btl_openib_rem_qp_info_t*) >>> malloc(sizeof(mca_btl_openib_rem_qp_info_t) * >>> - >>> mca_btl_openib_component.num_qps); >>> - >>> - /* unpack all the qp info */ >>> - for (qp = 0; qp < mca_btl_openib_component.num_qps; ++qp) { >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &rem_info.rem_qps[qp].rem_qp_num, >>> &cnt, >>> - OPAL_UINT32); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &rem_info.rem_qps[qp].rem_psn, >>> &cnt, >>> - OPAL_UINT32); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - } >>> - >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT16)); >>> - rc = opal_dss.unpack(buffer, &rem_info.rem_lid, &cnt, OPAL_UINT16); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &rem_info.rem_mtu, &cnt, OPAL_UINT32); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &rem_info.rem_index, &cnt, >>> OPAL_UINT32); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - } >>> - >>> - BTL_VERBOSE(("Received QP Info, LID = %d, SUBNET = %" PRIx64 "\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id)); >>> - >>> - master = ompi_rte_compare_name_fields(OMPI_RTE_CMP_ALL, >>> OMPI_PROC_MY_NAME, >>> - process_name) > 0 ? true : false; >>> - >>> - /* Need to protect the ib_procs list */ >>> - OPAL_THREAD_LOCK(&mca_btl_openib_component.ib_lock); >>> - >>> - for (ib_proc = (mca_btl_openib_proc_t*) >>> - opal_list_get_first(&mca_btl_openib_component.ib_procs); >>> - ib_proc != (mca_btl_openib_proc_t*) >>> - opal_list_get_end(&mca_btl_openib_component.ib_procs); >>> - ib_proc = (mca_btl_openib_proc_t*)opal_list_get_next(ib_proc)) { >>> - bool found = false; >>> - >>> - if (OPAL_EQUAL != ompi_rte_compare_name_fields(OMPI_RTE_CMP_ALL, >>> - &ib_proc->proc_ompi->proc_name, >>> process_name)) { >>> - continue; >>> - } >>> - >>> - if (ENDPOINT_CONNECT_REQUEST != message_type) { >>> - /* This is a reply message. Try to get the endpoint >>> - instance the reply belongs to */ >>> - for (i = 0; i < ib_proc->proc_endpoint_count; i++) { >>> - ib_endpoint = ib_proc->proc_endpoints[i]; >>> - if (ib_endpoint->qps[0].qp->lcl_qp != NULL && >>> - lcl_lid == ib_endpoint->endpoint_btl->lid && >>> - lcl_qp == ib_endpoint->qps[0].qp->lcl_qp->qp_num && >>> - rem_info.rem_subnet_id == ib_endpoint->subnet_id) { >>> - found = true; >>> - break; >>> - } >>> - } >>> - } else { >>> - /* This is new connection request. If this is master try >>> - to find endpoint in a connecting state. If this is >>> - slave try to find endpoint in closed state and >>> - initiate connection back */ >>> - mca_btl_openib_endpoint_t *ib_endpoint_found = NULL; >>> - int master_first_closed = -1; >>> - >>> - for (i = 0; i < ib_proc->proc_endpoint_count; i++) { >>> - ib_endpoint = ib_proc->proc_endpoints[i]; >>> - if (ib_endpoint->subnet_id != rem_info.rem_subnet_id || >>> - (ib_endpoint->endpoint_state != MCA_BTL_IB_CONNECTING >>> - && ib_endpoint->endpoint_state != MCA_BTL_IB_CLOSED)) >>> - continue; >>> - found = true; >>> - ib_endpoint_found = ib_endpoint; >>> - >>> - if (master && -1 == master_first_closed && >>> - MCA_BTL_IB_CLOSED == ib_endpoint->endpoint_state ) { >>> - /* capture in case no endpoint in connecting state */ >>> - master_first_closed = i; >>> - } >>> - >>> - if ((master && >>> - MCA_BTL_IB_CONNECTING == ib_endpoint->endpoint_state) >>> || >>> - (!master && >>> - MCA_BTL_IB_CLOSED == ib_endpoint->endpoint_state)) >>> - break; /* Found one. No point to continue */ >>> - } >>> - ib_endpoint = ib_endpoint_found; >>> - >>> - if (found && master && >>> - MCA_BTL_IB_CLOSED == ib_endpoint->endpoint_state ) { >>> - /* since this is master and no endpoints found in >>> - * connecting state use the first endpoint found >>> - * in closed state */ >>> - ib_endpoint = ib_proc->proc_endpoints[master_first_closed]; >>> - } >>> - >>> - /* if this is slave and there is no endpoints in closed >>> - state then all connection are already in progress so >>> - just ignore this connection request */ >>> - if (found && !master && >>> - MCA_BTL_IB_CLOSED != ib_endpoint->endpoint_state) { >>> - OPAL_THREAD_UNLOCK(&mca_btl_openib_component.ib_lock); >>> - return; >>> - } >>> - } >>> - >>> - if (!found) { >>> - BTL_ERROR(("can't find suitable endpoint for this peer\n")); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - OPAL_THREAD_UNLOCK(&mca_btl_openib_component.ib_lock); >>> - return; >>> - } >>> - >>> - OPAL_THREAD_LOCK(&ib_endpoint->endpoint_lock); >>> - endpoint_state = ib_endpoint->endpoint_state; >>> - >>> - /* Update status */ >>> - switch (endpoint_state) { >>> - case MCA_BTL_IB_CLOSED : >>> - /* We had this connection closed before. The endpoint is >>> - trying to connect. Move the status of this connection >>> - to CONNECTING, and then reply with our QP >>> - information */ >>> - if (master) { >>> - assert(rem_info.rem_qps != NULL); >>> - rc = reply_start_connect(ib_endpoint, &rem_info); >>> - } else { >>> - rc = >>> oob_module_start_connect(ib_endpoint->endpoint_local_cpc, >>> - ib_endpoint); >>> - } >>> - >>> - if (OMPI_SUCCESS != rc) { >>> - BTL_ERROR(("error in endpoint reply start connect")); >>> - mca_btl_openib_endpoint_invoke_error(ib_endpoint); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - break; >>> - } >>> - >>> - /* As long as we expect a message from the peer (in order >>> - to setup the connection) let the event engine pool the >>> - RML events. Note: we increment it once peer active >>> - connection. */ >>> - opal_progress_event_users_increment(); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - break; >>> - >>> - case MCA_BTL_IB_CONNECTING : >>> - assert(rem_info.rem_qps != NULL); >>> - set_remote_info(ib_endpoint, &rem_info); >>> - if (OMPI_SUCCESS != (rc = qp_connect_all(ib_endpoint))) { >>> - BTL_ERROR(("endpoint connect error: %d", rc)); >>> - mca_btl_openib_endpoint_invoke_error(ib_endpoint); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - break; >>> - } >>> - >>> - if (master) { >>> - ib_endpoint->endpoint_state = MCA_BTL_IB_WAITING_ACK; >>> - >>> - /* Send him an ACK */ >>> - send_connect_data(ib_endpoint, ENDPOINT_CONNECT_RESPONSE); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - } else { >>> - send_connect_data(ib_endpoint, ENDPOINT_CONNECT_ACK); >>> - /* Tell main BTL that we're done */ >>> - mca_btl_openib_endpoint_cpc_complete(ib_endpoint); >>> - /* cpc complete unlock the endpoint */ >>> - } >>> - break; >>> - >>> - case MCA_BTL_IB_WAITING_ACK: >>> - /* Tell main BTL that we're done */ >>> - mca_btl_openib_endpoint_cpc_complete(ib_endpoint); >>> - /* cpc complete unlock the endpoint */ >>> - break; >>> - >>> - case MCA_BTL_IB_CONNECT_ACK: >>> - send_connect_data(ib_endpoint, ENDPOINT_CONNECT_ACK); >>> - /* Tell main BTL that we're done */ >>> - mca_btl_openib_endpoint_cpc_complete(ib_endpoint); >>> - /* cpc complete unlock the endpoint */ >>> - break; >>> - >>> - case MCA_BTL_IB_CONNECTED: >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - break; >>> - >>> - case MCA_BTL_IB_FAILED: >>> - /* This connection has been put in the failed state >>> - * so just ignore the connection message. */ >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - break; >>> - >>> - default : >>> - BTL_ERROR(("Invalid endpoint state %d", endpoint_state)); >>> - mca_btl_openib_endpoint_invoke_error(ib_endpoint); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - } >>> - break; >>> - } >>> - OPAL_THREAD_UNLOCK(&mca_btl_openib_component.ib_lock); >>> -} >>> >>> Deleted: trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.h >>> ============================================================================== >>> --- trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.h >>> 2013-11-13 23:16:53 EST (Wed, 13 Nov 2013) (r29702) >>> +++ /dev/null 00:00:00 1970 (deleted) >>> @@ -1,18 +0,0 @@ >>> -/* >>> - * Copyright (c) 2007-2008 Cisco Systems, Inc. All rights reserved. >>> - * >>> - * $COPYRIGHT$ >>> - * >>> - * Additional copyrights may follow >>> - * >>> - * $HEADER$ >>> - */ >>> - >>> -#ifndef BTL_OPENIB_CONNECT_OOB_H >>> -#define BTL_OPENIB_CONNECT_OOB_H >>> - >>> -#include "connect/connect.h" >>> - >>> -extern ompi_btl_openib_connect_base_component_t >>> ompi_btl_openib_connect_oob; >>> - >>> -#endif >>> >>> Deleted: trunk/ompi/mca/btl/openib/connect/btl_openib_connect_xoob.c >>> ============================================================================== >>> --- trunk/ompi/mca/btl/openib/connect/btl_openib_connect_xoob.c >>> 2013-11-13 23:16:53 EST (Wed, 13 Nov 2013) (r29702) >>> +++ /dev/null 00:00:00 1970 (deleted) >>> @@ -1,1150 +0,0 @@ >>> -/* >>> - * Copyright (c) 2007-2011 Mellanox Technologies. All rights reserved. >>> - * Copyright (c) 2009-2013 Cisco Systems, Inc. All rights reserved. >>> - * Copyright (c) 2009 IBM Corporation. All rights reserved. >>> - * Copyright (c) 2010-2011 The University of Tennessee and The University >>> - * of Tennessee Research Foundation. All rights >>> - * reserved. >>> - * Copyright (c) 2012 Los Alamos National Security, LLC. All rights >>> - * reserved. >>> - * Copyright (c) 2013 NVIDIA Corporation. All rights reserved. >>> - * >>> - * $COPYRIGHT$ >>> - * >>> - * Additional copyrights may follow >>> - * >>> - * $HEADER$ >>> - */ >>> - >>> -#include "ompi_config.h" >>> - >>> -#include "opal_stdint.h" >>> -#include "opal/dss/dss.h" >>> -#include "opal/util/error.h" >>> -#include "opal/util/output.h" >>> -#include "opal/util/show_help.h" >>> - >>> -#include "ompi/mca/rte/rte.h" >>> - >>> -#include "btl_openib.h" >>> -#include "btl_openib_endpoint.h" >>> -#include "btl_openib_proc.h" >>> -#include "btl_openib_xrc.h" >>> -#include "btl_openib_async.h" >>> -#include "connect/connect.h" >>> -#if (ENABLE_DYNAMIC_SL) >>> -#include "connect/btl_openib_connect_sl.h" >>> -#endif >>> - >>> -static void xoob_component_register(void); >>> -static int xoob_component_query(mca_btl_openib_module_t *openib_btl, >>> - ompi_btl_openib_connect_base_module_t >>> **cpc); >>> -static int xoob_component_finalize(void); >>> - >>> -static int xoob_module_start_connect(ompi_btl_openib_connect_base_module_t >>> *cpc, >>> - mca_btl_base_endpoint_t *endpoint); >>> - >>> -/* >>> - * The "component" struct -- the top-level function pointers for the >>> - * xoob connection scheme. >>> - */ >>> -ompi_btl_openib_connect_base_component_t ompi_btl_openib_connect_xoob = { >>> - "xoob", >>> - /* Register */ >>> - xoob_component_register, >>> - /* Init */ >>> - NULL, >>> - /* Query */ >>> - xoob_component_query, >>> - /* Finalize */ >>> - xoob_component_finalize, >>> -}; >>> - >>> -typedef enum { >>> - ENDPOINT_XOOB_CONNECT_REQUEST, >>> - ENDPOINT_XOOB_CONNECT_RESPONSE, >>> - ENDPOINT_XOOB_CONNECT_XRC_REQUEST, >>> - ENDPOINT_XOOB_CONNECT_XRC_RESPONSE, >>> - ENDPOINT_XOOB_CONNECT_XRC_NR_RESPONSE /* The xrc recv qp already was >>> destroyed */ >>> -} connect_message_type_t; >>> - >>> -static bool rml_recv_posted = false; >>> - >>> -#define XOOB_SET_REMOTE_INFO(EP, INFO) \ >>> -do { \ >>> - /* copy the rem_info stuff */ \ >>> - EP.rem_lid = INFO.rem_lid; \ >>> - EP.rem_subnet_id = INFO.rem_subnet_id; \ >>> - EP.rem_mtu = INFO.rem_mtu; \ >>> - EP.rem_index = INFO.rem_index; \ >>> - memcpy((void*)EP.rem_qps, (void*)INFO.rem_qps, \ >>> - sizeof(mca_btl_openib_rem_qp_info_t)); \ >>> - /* copy the rem_info stuff */ \ >>> - memcpy((void*)EP.rem_srqs, (void*)INFO.rem_srqs, \ >>> - sizeof(mca_btl_openib_rem_srq_info_t) * \ >>> - mca_btl_openib_component.num_xrc_qps); \ >>> -} while (0) >>> - >>> -static int xoob_priority = 60; >>> - >>> -/* >>> - * Callback when we have finished RML sending the connect data to a >>> - * remote peer >>> - */ >>> -static void xoob_rml_send_cb(int status, ompi_process_name_t* endpoint, >>> - opal_buffer_t* buffer, ompi_rml_tag_t tag, >>> - void* cbdata) >>> -{ >>> - OBJ_RELEASE(buffer); >>> -} >>> - >>> -/* Receive connect information to remote endpoint */ >>> -static int xoob_receive_connect_data(mca_btl_openib_rem_info_t *info, >>> uint16_t *lid, >>> - uint8_t *message_type, opal_buffer_t* buffer) >>> -{ >>> - int cnt = 1, rc, srq; >>> - >>> - /* Recv standart header */ >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT8)); >>> - rc = opal_dss.unpack(buffer, message_type, &cnt, OPAL_UINT8); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack Message type = %d\n", *message_type)); >>> - >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT64)); >>> - rc = opal_dss.unpack(buffer, &info->rem_subnet_id, &cnt, OPAL_UINT64); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack sid = %" PRIx64 "\n", info->rem_subnet_id)); >>> - >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT16)); >>> - rc = opal_dss.unpack(buffer, &info->rem_lid, &cnt, OPAL_UINT16); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack lid = %d", info->rem_lid)); >>> - >>> - /* Till now we got the standart header, now we continue to recieve >>> data for >>> - * different packet types >>> - */ >>> - if (ENDPOINT_XOOB_CONNECT_REQUEST == *message_type || >>> - ENDPOINT_XOOB_CONNECT_RESPONSE == *message_type) { >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &info->rem_qps->rem_qp_num, &cnt, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack remote qp = %x", >>> info->rem_qps->rem_qp_num)); >>> - >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &info->rem_qps->rem_psn, &cnt, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack remote psn = %d", >>> info->rem_qps->rem_psn)); >>> - >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &info->rem_mtu, &cnt, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack remote mtu = %d", info->rem_mtu)); >>> - } >>> - >>> - if (ENDPOINT_XOOB_CONNECT_REQUEST == *message_type || >>> - ENDPOINT_XOOB_CONNECT_XRC_REQUEST == *message_type) { >>> - /* unpack requested lid info */ >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT16)); >>> - rc = opal_dss.unpack(buffer, lid, &cnt, OPAL_UINT16); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack requested lid = %d", *lid)); >>> - } >>> - >>> - /* Unpack requested recv qp number */ >>> - if (ENDPOINT_XOOB_CONNECT_XRC_REQUEST == *message_type) { >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - /* In XRC request case we will use rem_qp_num as container for >>> requested qp number */ >>> - rc = opal_dss.unpack(buffer, &info->rem_qps->rem_qp_num, &cnt, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("Recv unpack requested qp = %x", >>> info->rem_qps->rem_qp_num)); >>> - } >>> - >>> - if (ENDPOINT_XOOB_CONNECT_RESPONSE == *message_type || >>> - ENDPOINT_XOOB_CONNECT_XRC_RESPONSE == *message_type) { >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &info->rem_index, &cnt, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack remote index = %d", info->rem_index)); >>> - >>> - for (srq = 0; srq < mca_btl_openib_component.num_xrc_qps; srq++) { >>> - BTL_VERBOSE(("unpacking %d of %d\n", cnt, OPAL_UINT32)); >>> - rc = opal_dss.unpack(buffer, &info->rem_srqs[srq].rem_srq_num, >>> &cnt, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return OMPI_ERROR; >>> - } >>> - BTL_VERBOSE(("Recv unpack remote index srq num[%d]= %d", srq, >>> info->rem_srqs[srq].rem_srq_num)); >>> - } >>> - } >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* >>> - * send connect information to remote endpoint >>> - */ >>> -static int xoob_send_connect_data(mca_btl_base_endpoint_t* endpoint, >>> - uint8_t message_type) >>> -{ >>> - opal_buffer_t* buffer = OBJ_NEW(opal_buffer_t); >>> - int rc, srq; >>> - >>> - if (NULL == buffer) { >>> - OMPI_ERROR_LOG(OMPI_ERR_OUT_OF_RESOURCE); >>> - return OMPI_ERR_OUT_OF_RESOURCE; >>> - } >>> - >>> - /* Bulding standart header that we use in all messages: >>> - * - Message type, >>> - * - Our subnet id >>> - * - Our LID >>> - */ >>> - /* pack the info in the send buffer */ >>> - BTL_VERBOSE(("Send pack Message type = %d", message_type)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT8)); >>> - rc = opal_dss.pack(buffer, &message_type, 1, OPAL_UINT8); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - BTL_VERBOSE(("Send pack sid = %" PRIx64 "\n", endpoint->subnet_id)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT64)); >>> - rc = opal_dss.pack(buffer, &endpoint->subnet_id, 1, OPAL_UINT64); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - BTL_VERBOSE(("Send pack lid = %d", endpoint->endpoint_btl->lid)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT16)); >>> - rc = opal_dss.pack(buffer, &endpoint->endpoint_btl->lid, 1, >>> OPAL_UINT16); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - /* Now we append to standart header additional information >>> - * that is required for full (open qp,etc..) connect request and >>> response: >>> - * - qp_num of first qp >>> - * - psn of first qp >>> - * - MTU >>> - */ >>> - if (ENDPOINT_XOOB_CONNECT_REQUEST == message_type || >>> - ENDPOINT_XOOB_CONNECT_RESPONSE == message_type) { >>> - uint32_t psn, qp_num; >>> - >>> - if (ENDPOINT_XOOB_CONNECT_REQUEST == message_type) { >>> - qp_num = endpoint->qps[0].qp->lcl_qp->qp_num; >>> - psn = endpoint->qps[0].qp->lcl_psn; >>> - } else { >>> - qp_num = endpoint->xrc_recv_qp_num; >>> - psn = endpoint->xrc_recv_psn; >>> - } >>> - /* stuff all the QP info into the buffer */ >>> - /* we need to send only one QP */ >>> - BTL_VERBOSE(("Send pack qp num = %x", qp_num)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &qp_num, 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - BTL_VERBOSE(("Send pack lpsn = %d", psn)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &psn, 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - BTL_VERBOSE(("Send pack mtu = %d", >>> endpoint->endpoint_btl->device->mtu)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &endpoint->endpoint_btl->device->mtu, 1, >>> - OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - >>> - /* We append to header above additional information >>> - * that is required for full & XRC connect request: >>> - * - The lid ob btl on remote site that we want to connect >>> - */ >>> - if (ENDPOINT_XOOB_CONNECT_REQUEST == message_type || >>> - ENDPOINT_XOOB_CONNECT_XRC_REQUEST == message_type) { >>> - /* when we are sending request we add remote lid that we want to >>> connect */ >>> - >>> - BTL_VERBOSE(("Send pack remote lid = %d", endpoint->ib_addr->lid)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT16)); >>> - rc = opal_dss.pack(buffer, &endpoint->ib_addr->lid, 1, >>> OPAL_UINT16); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - >>> - /* when we are sending xrc request we add remote >>> - * recv qp number that we want to connect. */ >>> - if (ENDPOINT_XOOB_CONNECT_XRC_REQUEST == message_type) { >>> - BTL_VERBOSE(("Send pack remote qp = %x", >>> endpoint->ib_addr->remote_xrc_rcv_qp_num)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, >>> &endpoint->ib_addr->remote_xrc_rcv_qp_num, >>> - 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - /* We append to header above additional information >>> - * that is required for full & XRC connect response: >>> - * - index of our endpoint >>> - * - array of xrc-srq numbers >>> - */ >>> - if (ENDPOINT_XOOB_CONNECT_RESPONSE == message_type || >>> - ENDPOINT_XOOB_CONNECT_XRC_RESPONSE == message_type) { >>> - /* we need to send the endpoint index for immidate send */ >>> - BTL_VERBOSE(("Send pack index = %d", endpoint->index)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, &endpoint->index, 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - /* on response we add all SRQ numbers */ >>> - for (srq = 0; srq < mca_btl_openib_component.num_xrc_qps; srq++) { >>> - BTL_VERBOSE(("Send pack srq[%d] num = %d", srq, >>> endpoint->endpoint_btl->qps[srq].u.srq_qp.srq->xrc_srq_num)); >>> - BTL_VERBOSE(("packing %d of %d\n", 1, OPAL_UINT32)); >>> - rc = opal_dss.pack(buffer, >>> &endpoint->endpoint_btl->qps[srq].u.srq_qp.srq->xrc_srq_num, >>> - 1, OPAL_UINT32); >>> - if (OPAL_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - } >>> - } >>> - >>> - /* send to remote endpoint */ >>> - rc = >>> ompi_rte_send_buffer_nb(&endpoint->endpoint_proc->proc_ompi->proc_name, >>> - buffer, OMPI_RML_TAG_XOPENIB, >>> - xoob_rml_send_cb, NULL); >>> - if (OMPI_SUCCESS != rc) { >>> - OMPI_ERROR_LOG(rc); >>> - return rc; >>> - } >>> - >>> - BTL_VERBOSE(("Send QP Info, LID = %d, SUBNET = %" PRIx64 ", Message >>> type = %d", >>> - endpoint->endpoint_btl->lid, >>> - endpoint->subnet_id, >>> - message_type)); >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* Create XRC send qp */ >>> -static int xoob_send_qp_create (mca_btl_base_endpoint_t* endpoint) >>> -{ >>> - int prio = BTL_OPENIB_LP_CQ; /* all send completions go to low prio CQ >>> */ >>> - uint32_t send_wr; >>> - struct ibv_qp **qp; >>> - uint32_t *psn; >>> - struct ibv_qp_init_attr qp_init_attr; >>> - struct ibv_qp_attr attr; >>> - int ret; >>> - size_t req_inline; >>> - >>> - mca_btl_openib_module_t *openib_btl = >>> - (mca_btl_openib_module_t*)endpoint->endpoint_btl; >>> - >>> - /* Prepare QP structs */ >>> - BTL_VERBOSE(("Creating Send QP\n")); >>> - qp = &endpoint->qps[0].qp->lcl_qp; >>> - psn = &endpoint->qps[0].qp->lcl_psn; >>> - /* reserve additional wr for eager rdma credit management */ >>> - send_wr = endpoint->ib_addr->qp->sd_wqe + >>> - (mca_btl_openib_component.use_eager_rdma ? >>> - mca_btl_openib_component.max_eager_rdma : 0); >>> - memset(&qp_init_attr, 0, sizeof(struct ibv_qp_init_attr)); >>> - memset(&attr, 0, sizeof(struct ibv_qp_attr)); >>> - >>> - qp_init_attr.send_cq = qp_init_attr.recv_cq = >>> openib_btl->device->ib_cq[prio]; >>> - >>> - /* no need recv queue; receives are posted to srq */ >>> - qp_init_attr.cap.max_recv_wr = 0; >>> - qp_init_attr.cap.max_send_wr = send_wr; >>> - qp_init_attr.cap.max_inline_data = req_inline = >>> - openib_btl->device->max_inline_data; >>> - qp_init_attr.cap.max_send_sge = 1; >>> - /* this one is ignored by driver */ >>> - qp_init_attr.cap.max_recv_sge = 1; /* we do not use SG list */ >>> - qp_init_attr.qp_type = IBV_QPT_XRC; >>> - qp_init_attr.xrc_domain = openib_btl->device->xrc_domain; >>> - *qp = ibv_create_qp(openib_btl->device->ib_pd, &qp_init_attr); >>> - if (NULL == *qp) { >>> - opal_show_help("help-mpi-btl-openib-cpc-base.txt", >>> - "ibv_create_qp failed", true, >>> - ompi_process_info.nodename, >>> - ibv_get_device_name(openib_btl->device->ib_dev), >>> - "Reliable connected (XRC)"); >>> - return OMPI_ERROR; >>> - } >>> - >>> - if (qp_init_attr.cap.max_inline_data < req_inline) { >>> - endpoint->qps[0].ib_inline_max = qp_init_attr.cap.max_inline_data; >>> - opal_show_help("help-mpi-btl-openib-cpc-base.txt", >>> - "inline truncated", ompi_process_info.nodename, >>> - ibv_get_device_name(openib_btl->device->ib_dev), >>> - openib_btl->port_num, >>> - req_inline, qp_init_attr.cap.max_inline_data); >>> - } else { >>> - endpoint->qps[0].ib_inline_max = req_inline; >>> - } >>> - >>> - attr.qp_state = IBV_QPS_INIT; >>> - attr.pkey_index = openib_btl->pkey_index; >>> - attr.port_num = openib_btl->port_num; >>> - attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | >>> IBV_ACCESS_REMOTE_READ; >>> - ret = ibv_modify_qp(*qp, &attr, >>> - IBV_QP_STATE | >>> - IBV_QP_PKEY_INDEX | >>> - IBV_QP_PORT | >>> - IBV_QP_ACCESS_FLAGS ); >>> - if (ret) { >>> - BTL_ERROR(("Error modifying QP[%x] to IBV_QPS_INIT errno says: %s >>> [%d]", >>> - (*qp)->qp_num, strerror(ret), ret)); >>> - return OMPI_ERROR; >>> - } >>> - >>> - /* Setup meta data on the endpoint */ >>> - *psn = lrand48() & 0xffffff; >>> - >>> - /* Now that all the qp's are created locally, post some receive >>> - buffers, setup credits, etc. */ >>> - return mca_btl_openib_endpoint_post_recvs(endpoint); >>> -} >>> - >>> -/* Send qp connect */ >>> -static int xoob_send_qp_connect(mca_btl_openib_endpoint_t *endpoint, >>> mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - struct ibv_qp* qp; >>> - struct ibv_qp_attr attr; >>> - uint32_t psn; >>> - int ret; >>> - >>> - mca_btl_openib_module_t* openib_btl = >>> - (mca_btl_openib_module_t*)endpoint->endpoint_btl; >>> - >>> - BTL_VERBOSE(("Connecting Send QP\n")); >>> - assert(NULL != endpoint->qps); >>> - qp = endpoint->qps[0].qp->lcl_qp; >>> - psn = endpoint->qps[0].qp->lcl_psn; >>> - >>> - memset(&attr, 0, sizeof(attr)); >>> - attr.qp_state = IBV_QPS_RTR; >>> - attr.path_mtu = (openib_btl->device->mtu < endpoint->rem_info.rem_mtu) >>> ? >>> - openib_btl->device->mtu : rem_info->rem_mtu; >>> - attr.dest_qp_num = rem_info->rem_qps->rem_qp_num; >>> - attr.rq_psn = rem_info->rem_qps->rem_psn; >>> - attr.max_dest_rd_atomic = mca_btl_openib_component.ib_max_rdma_dst_ops; >>> - attr.min_rnr_timer = mca_btl_openib_component.ib_min_rnr_timer; >>> - attr.ah_attr.is_global = 0; >>> - attr.ah_attr.dlid = rem_info->rem_lid; >>> - attr.ah_attr.src_path_bits = openib_btl->src_path_bits; >>> - attr.ah_attr.port_num = openib_btl->port_num; >>> - attr.ah_attr.static_rate = 0; >>> - attr.ah_attr.sl = mca_btl_openib_component.ib_service_level; >>> - >>> -#if (ENABLE_DYNAMIC_SL) >>> - /* if user enabled dynamic SL, get it from PathRecord */ >>> - if (0 != mca_btl_openib_component.ib_path_record_service_level) { >>> - int rc = btl_openib_connect_get_pathrecord_sl(qp->context, >>> - >>> attr.ah_attr.port_num, >>> - openib_btl->lid, >>> - attr.ah_attr.dlid); >>> - if (OMPI_ERROR == rc) { >>> - return OMPI_ERROR; >>> - } >>> - attr.ah_attr.sl = rc; >>> - } >>> -#endif >>> - >>> - if (mca_btl_openib_component.verbose) { >>> - BTL_VERBOSE(("Set MTU to IBV value %d (%s bytes)", attr.path_mtu, >>> - (attr.path_mtu == IBV_MTU_256) ? "256" : >>> - (attr.path_mtu == IBV_MTU_512) ? "512" : >>> - (attr.path_mtu == IBV_MTU_1024) ? "1024" : >>> - (attr.path_mtu == IBV_MTU_2048) ? "2048" : >>> - (attr.path_mtu == IBV_MTU_4096) ? "4096" : >>> - "unknown (!)")); >>> - } >>> - ret = ibv_modify_qp(qp, &attr, >>> - IBV_QP_STATE | >>> - IBV_QP_AV | >>> - IBV_QP_PATH_MTU | >>> - IBV_QP_DEST_QPN | >>> - IBV_QP_RQ_PSN | >>> - IBV_QP_MAX_DEST_RD_ATOMIC | >>> - IBV_QP_MIN_RNR_TIMER); >>> - if (ret) { >>> - BTL_ERROR(("Error modifying QP[%x] to IBV_QPS_RTR errno says: %s >>> [%d]", >>> - qp->qp_num, strerror(ret), ret)); >>> - return OMPI_ERROR; >>> - } >>> - >>> - attr.qp_state = IBV_QPS_RTS; >>> - attr.timeout = mca_btl_openib_component.ib_timeout; >>> - attr.retry_cnt = mca_btl_openib_component.ib_retry_count; >>> - attr.rnr_retry = mca_btl_openib_component.ib_rnr_retry; >>> - attr.sq_psn = psn; >>> - attr.max_rd_atomic = mca_btl_openib_component.ib_max_rdma_dst_ops; >>> - ret = ibv_modify_qp(qp, &attr, >>> - IBV_QP_STATE | >>> - IBV_QP_TIMEOUT | >>> - IBV_QP_RETRY_CNT | >>> - IBV_QP_RNR_RETRY | >>> - IBV_QP_SQ_PSN | >>> - IBV_QP_MAX_QP_RD_ATOMIC); >>> - if (ret) { >>> - BTL_ERROR(("Error modifying QP[%x] to IBV_QPS_RTS errno says: %s >>> [%d]", >>> - qp->qp_num, strerror(ret), ret)); >>> - return OMPI_ERROR; >>> - } >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* Recv qp create */ >>> -static int xoob_recv_qp_create(mca_btl_openib_endpoint_t *endpoint, >>> mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - struct ibv_qp_init_attr qp_init_attr; >>> - struct ibv_qp_attr attr; >>> - int ret; >>> - >>> - mca_btl_openib_module_t* openib_btl = >>> - (mca_btl_openib_module_t*)endpoint->endpoint_btl; >>> - >>> - BTL_VERBOSE(("Connecting Recv QP\n")); >>> - >>> - memset(&qp_init_attr, 0, sizeof(struct ibv_qp_init_attr)); >>> - /* Only xrc_domain is required, all other are ignored */ >>> - qp_init_attr.xrc_domain = openib_btl->device->xrc_domain; >>> - ret = ibv_create_xrc_rcv_qp(&qp_init_attr, &endpoint->xrc_recv_qp_num); >>> - if (ret) { >>> - BTL_ERROR(("Error creating XRC recv QP[%x], errno says: %s [%d]", >>> - endpoint->xrc_recv_qp_num, strerror(ret), ret)); >>> - return OMPI_ERROR; >>> - } >>> - >>> - memset(&attr, 0, sizeof(struct ibv_qp_attr)); >>> - attr.qp_state = IBV_QPS_INIT; >>> - attr.pkey_index = openib_btl->pkey_index; >>> - attr.port_num = openib_btl->port_num; >>> - attr.qp_access_flags = IBV_ACCESS_REMOTE_WRITE | >>> IBV_ACCESS_REMOTE_READ; >>> - ret = ibv_modify_xrc_rcv_qp(openib_btl->device->xrc_domain, >>> - endpoint->xrc_recv_qp_num, >>> - &attr, >>> - IBV_QP_STATE| >>> - IBV_QP_PKEY_INDEX| >>> - IBV_QP_PORT| >>> - IBV_QP_ACCESS_FLAGS); >>> - if (ret) { >>> - BTL_ERROR(("Error modifying XRC recv QP[%x] to IBV_QPS_INIT, errno >>> says: %s [%d]", >>> - endpoint->xrc_recv_qp_num, strerror(ret), ret)); >>> - while(1); >>> - return OMPI_ERROR; >>> - } >>> - >>> - memset(&attr, 0, sizeof(struct ibv_qp_attr)); >>> - attr.qp_state = IBV_QPS_RTR; >>> - attr.path_mtu = (openib_btl->device->mtu < endpoint->rem_info.rem_mtu) >>> ? >>> - openib_btl->device->mtu : rem_info->rem_mtu; >>> - attr.dest_qp_num = rem_info->rem_qps->rem_qp_num; >>> - attr.rq_psn = rem_info->rem_qps->rem_psn; >>> - attr.max_dest_rd_atomic = mca_btl_openib_component.ib_max_rdma_dst_ops; >>> - attr.min_rnr_timer = mca_btl_openib_component.ib_min_rnr_timer; >>> - attr.ah_attr.is_global = 0; >>> - attr.ah_attr.dlid = rem_info->rem_lid; >>> - attr.ah_attr.src_path_bits = openib_btl->src_path_bits; >>> - attr.ah_attr.port_num = openib_btl->port_num; >>> - attr.ah_attr.static_rate = 0; >>> - attr.ah_attr.sl = mca_btl_openib_component.ib_service_level; >>> - >>> -#if (ENABLE_DYNAMIC_SL) >>> - /* if user enabled dynamic SL, get it from PathRecord */ >>> - if (0 != mca_btl_openib_component.ib_path_record_service_level) { >>> - int rc = btl_openib_connect_get_pathrecord_sl( >>> - openib_btl->device->xrc_domain->context, >>> - attr.ah_attr.port_num, >>> - openib_btl->lid, >>> - attr.ah_attr.dlid); >>> - if (OMPI_ERROR == rc) { >>> - return OMPI_ERROR; >>> - } >>> - attr.ah_attr.sl = rc; >>> - } >>> -#endif >>> - >>> - ret = ibv_modify_xrc_rcv_qp(openib_btl->device->xrc_domain, >>> - endpoint->xrc_recv_qp_num, >>> - &attr, >>> - IBV_QP_STATE| >>> - IBV_QP_AV| >>> - IBV_QP_PATH_MTU| >>> - IBV_QP_DEST_QPN| >>> - IBV_QP_RQ_PSN| >>> - IBV_QP_MAX_DEST_RD_ATOMIC| >>> - IBV_QP_MIN_RNR_TIMER); >>> - if (ret) { >>> - BTL_ERROR(("Error modifying XRC recv QP[%x] to IBV_QPS_RTR, errno >>> says: %s [%d]", >>> - endpoint->xrc_recv_qp_num, strerror(ret), ret)); >>> - return OMPI_ERROR; >>> - } >>> -#if OPAL_HAVE_THREADS >>> - if (APM_ENABLED) { >>> - mca_btl_openib_load_apm_xrc_rcv(endpoint->xrc_recv_qp_num, >>> endpoint); >>> - } >>> -#endif >>> - >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* Recv qp connect */ >>> -static int xoob_recv_qp_connect(mca_btl_openib_endpoint_t *endpoint, >>> mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - int ret; >>> - >>> - mca_btl_openib_module_t* openib_btl = >>> - (mca_btl_openib_module_t*)endpoint->endpoint_btl; >>> - >>> - BTL_VERBOSE(("Connecting Recv QP\n")); >>> - ret = ibv_reg_xrc_rcv_qp(openib_btl->device->xrc_domain, >>> rem_info->rem_qps->rem_qp_num); >>> - if (ret) { /* failed to regester the qp, so it is already die and we >>> should create new one */ >>> - /* Return NOT READY !!!*/ >>> - BTL_ERROR(("Failed to register qp_num: %d , get error: %s (%d)\n. >>> Replying with RNR", >>> - rem_info->rem_qps->rem_qp_num, strerror(ret), ret)); >>> - return OMPI_ERROR; >>> - } else { >>> - /* save the qp number for unregister */ >>> - endpoint->xrc_recv_qp_num = rem_info->rem_qps->rem_qp_num; >>> - return OMPI_SUCCESS; >>> - } >>> -} >>> - >>> -/* >>> - * Reply to a `start - connect' message >>> - */ >>> -static int xoob_reply_first_connect(mca_btl_openib_endpoint_t *endpoint, >>> - mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - int rc; >>> - >>> - BTL_VERBOSE(("Initialized QPs, LID = %d", >>> - ((mca_btl_openib_module_t*)endpoint->endpoint_btl)->lid)); >>> - >>> - /* Create local QP's and post receive resources */ >>> - if (OMPI_SUCCESS != (rc = xoob_recv_qp_create(endpoint, rem_info))) { >>> - return rc; >>> - } >>> - >>> - if (OMPI_SUCCESS != >>> - (rc = xoob_send_connect_data(endpoint, >>> ENDPOINT_XOOB_CONNECT_RESPONSE))) { >>> - BTL_ERROR(("error in endpoint send connect request error code is >>> %d", >>> - rc)); >>> - return rc; >>> - } >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* Find endpoint for specific subnet/lid/message */ >>> -static mca_btl_openib_endpoint_t* xoob_find_endpoint(ompi_process_name_t* >>> process_name, >>> - uint64_t subnet_id, uint16_t lid, uint8_t message_type) >>> -{ >>> - size_t i; >>> - mca_btl_openib_proc_t *ib_proc; >>> - mca_btl_openib_endpoint_t *ib_endpoint = NULL; >>> - bool found = false; >>> - >>> - BTL_VERBOSE(("Searching for ep and proc with follow parameters:" >>> - "jobid %d, vpid %d, " >>> - "sid %" PRIx64 ", lid %d", >>> - process_name->jobid, process_name->vpid, >>> - subnet_id, lid)); >>> - >>> - >>> - /* find ibproc */ >>> - OPAL_THREAD_LOCK(&mca_btl_openib_component.ib_lock); >>> - for (ib_proc = (mca_btl_openib_proc_t*) >>> - opal_list_get_first(&mca_btl_openib_component.ib_procs); >>> - ib_proc != (mca_btl_openib_proc_t*) >>> - opal_list_get_end(&mca_btl_openib_component.ib_procs); >>> - ib_proc = >>> (mca_btl_openib_proc_t*)opal_list_get_next(ib_proc)) { >>> - if (OPAL_EQUAL == ompi_rte_compare_name_fields(OMPI_RTE_CMP_ALL, >>> - &ib_proc->proc_ompi->proc_name, process_name)) { >>> - found = true; >>> - break; >>> - } >>> - } >>> - /* we found our ib_proc, lets find endpoint now */ >>> - if (found) { >>> - for (i = 0; i < ib_proc->proc_endpoint_count; i++) { >>> - ib_endpoint = ib_proc->proc_endpoints[i]; >>> - /* we need to check different >>> - * lid for different message type */ >>> - if (ENDPOINT_XOOB_CONNECT_RESPONSE == message_type || >>> - ENDPOINT_XOOB_CONNECT_XRC_RESPONSE == message_type) { >>> - /* response message */ >>> - if (ib_endpoint->subnet_id == subnet_id && >>> - ib_endpoint->ib_addr->lid == lid) { >>> - break; /* Found one */ >>> - } >>> - } else { >>> - /* request message */ >>> - if (ib_endpoint->subnet_id == subnet_id && >>> - ib_endpoint->endpoint_btl->lid == lid) { >>> - break; /* Found one */ >>> - } >>> - } >>> - } >>> - if (NULL == ib_endpoint) { >>> - BTL_ERROR(("can't find suitable endpoint for this >>> peer\n")); >>> - } >>> - } else { >>> - BTL_ERROR(("can't find suitable endpoint for this peer\n")); >>> - } >>> - OPAL_THREAD_UNLOCK(&mca_btl_openib_component.ib_lock); >>> - return ib_endpoint; >>> -} >>> - >>> -/* In case if XRC recv qp was closed and sender still don't know about it >>> - * we need close the qp, reset the ib_adrr status to CLOSED and start >>> everything >>> - * from scratch. >>> - */ >>> -static void xoob_restart_connect(mca_btl_base_endpoint_t *endpoint) >>> -{ >>> - BTL_VERBOSE(("Restarting the connection for the endpoint")); >>> - OPAL_THREAD_LOCK(&endpoint->ib_addr->addr_lock); >>> - switch (endpoint->ib_addr->status) { >>> - case MCA_BTL_IB_ADDR_CONNECTED: >>> - /* so we have the send qp, we just need the recive site. >>> - * Send request for SRQ numbers */ >>> - BTL_VERBOSE(("Restart The IB addr: sid %" PRIx64 " lid %d" >>> - "in MCA_BTL_IB_ADDR_CONNECTED status," >>> - " Changing to MCA_BTL_IB_ADDR_CLOSED and starting >>> from scratch\n", >>> - >>> endpoint->ib_addr->subnet_id,endpoint->ib_addr->lid)); >>> - /* Switching back to closed and starting from scratch */ >>> - endpoint->ib_addr->status = MCA_BTL_IB_ADDR_CLOSED; >>> - /* destroy the qp */ >>> - /* the reciver site was alredy closed so all pending list must >>> be clean ! */ >>> - assert >>> (opal_list_is_empty(&endpoint->qps->no_wqe_pending_frags[0])); >>> - assert >>> (opal_list_is_empty(&endpoint->qps->no_wqe_pending_frags[1])); >>> - if(ibv_destroy_qp(endpoint->qps[0].qp->lcl_qp)) >>> - BTL_ERROR(("Failed to destroy QP")); >>> - case MCA_BTL_IB_ADDR_CLOSED: >>> - case MCA_BTL_IB_ADDR_CONNECTING: >>> - BTL_VERBOSE(("Restart The IB addr: sid %" PRIx64 " lid %d" >>> - "in MCA_BTL_IB_ADDR_CONNECTING or >>> MCA_BTL_IB_ADDR_CLOSED status," >>> - " starting from scratch\n", >>> - >>> endpoint->ib_addr->subnet_id,endpoint->ib_addr->lid)); >>> - OPAL_THREAD_UNLOCK(&endpoint->ib_addr->addr_lock); >>> - /* xoob_module_start_connect() should automaticly handle all >>> other cases */ >>> - if (OMPI_SUCCESS != xoob_module_start_connect(NULL, endpoint)) >>> - BTL_ERROR(("Failed to restart connection from >>> MCA_BTL_IB_ADDR_CONNECTING/CLOSED")); >>> - break; >>> - default : >>> - BTL_ERROR(("Invalid endpoint status %d", >>> endpoint->ib_addr->status)); >>> - OPAL_THREAD_UNLOCK(&endpoint->ib_addr->addr_lock); >>> - } >>> -} >>> - >>> -/* Init remote information structs */ >>> -static int init_rem_info(mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - rem_info->rem_qps = >>> (mca_btl_openib_rem_qp_info_t*)malloc(sizeof(mca_btl_openib_rem_qp_info_t)); >>> - if (NULL == rem_info->rem_qps) { >>> - BTL_ERROR(("Failed to allocate memory for remote QP data\n")); >>> - return OMPI_ERROR; >>> - } >>> - rem_info->rem_srqs = >>> (mca_btl_openib_rem_srq_info_t*)malloc(sizeof(mca_btl_openib_rem_srq_info_t) >>> * >>> - mca_btl_openib_component.num_xrc_qps); >>> - if (NULL == rem_info->rem_srqs) { >>> - BTL_ERROR(("Failed to allocate memory for remote SRQ data\n")); >>> - return OMPI_ERROR; >>> - } >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* Free remote information structs */ >>> -static void free_rem_info(mca_btl_openib_rem_info_t *rem_info) >>> -{ >>> - if (NULL != rem_info->rem_qps) { >>> - free(rem_info->rem_qps); >>> - } >>> - if (NULL != rem_info->rem_srqs) { >>> - free(rem_info->rem_srqs); >>> - } >>> -} >>> - >>> -/* >>> - * Non blocking RML recv callback. Read incoming QP and other info, >>> - * and if this endpoint is trying to connect, reply with our QP info, >>> - * otherwise try to modify QP's and establish reliable connection >>> - */ >>> -static void xoob_rml_recv_cb(int status, ompi_process_name_t* process_name, >>> - opal_buffer_t* buffer, ompi_rml_tag_t tag, >>> - void* cbdata) >>> -{ >>> - int rc; >>> - uint8_t message_type; >>> - uint16_t requested_lid = 0; >>> - mca_btl_openib_rem_info_t rem_info; >>> - mca_btl_openib_endpoint_t *ib_endpoint = NULL; >>> - >>> - if ( OMPI_SUCCESS != init_rem_info(&rem_info)) { >>> - return; >>> - } >>> - >>> - /* Get data. */ >>> - if ( OMPI_SUCCESS != xoob_receive_connect_data(&rem_info, >>> &requested_lid, &message_type, buffer)) { >>> - BTL_ERROR(("Failed to read data\n")); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - >>> - /* Processing message */ >>> - switch (message_type) { >>> - case ENDPOINT_XOOB_CONNECT_REQUEST: >>> - BTL_VERBOSE(("Received ENDPOINT_XOOB_CONNECT_REQUEST: lid %d, >>> sid %" PRIx64 ", rlid %d\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id, >>> - requested_lid)); >>> - ib_endpoint = >>> xoob_find_endpoint(process_name,rem_info.rem_subnet_id, >>> - requested_lid, message_type); >>> - if ( NULL == ib_endpoint) { >>> - BTL_ERROR(("Got ENDPOINT_XOOB_CONNECT_REQUEST." >>> - " Failed to find endpoint with subnet %" PRIx64 >>> - " and LID %d", >>> - rem_info.rem_subnet_id,requested_lid)); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - OPAL_THREAD_LOCK(&ib_endpoint->endpoint_lock); >>> - /* prepost data on receiver site */ >>> - if (OMPI_SUCCESS != >>> mca_btl_openib_endpoint_post_recvs(ib_endpoint)) { >>> - BTL_ERROR(("Failed to post on XRC SRQs")); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - return; >>> - } >>> - /* we should create qp and send the info + srq to requestor */ >>> - rc = xoob_reply_first_connect(ib_endpoint, &rem_info); >>> - if (OMPI_SUCCESS != rc) { >>> - BTL_ERROR(("error in endpoint reply start connect")); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - return; >>> - } >>> - /* enable pooling for this btl */ >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - break; >>> - case ENDPOINT_XOOB_CONNECT_XRC_REQUEST: >>> - /* pasha we don't need the remote lid here ??*/ >>> - BTL_VERBOSE(("Received ENDPOINT_XOOB_CONNECT_XRC_REQUEST: lid >>> %d, sid %" PRIx64 "\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id)); >>> - ib_endpoint = >>> xoob_find_endpoint(process_name,rem_info.rem_subnet_id, >>> - requested_lid, message_type); >>> - if ( NULL == ib_endpoint) { >>> - BTL_ERROR(("Got ENDPOINT_XOOB_CONNECT_XRC_REQUEST." >>> - " Failed to find endpoint with subnet %" >>> PRIx64 " and LID %d", >>> - rem_info.rem_subnet_id,requested_lid)); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - if (OMPI_SUCCESS == xoob_recv_qp_connect(ib_endpoint, >>> &rem_info)) { >>> - if (OMPI_SUCCESS != >>> mca_btl_openib_endpoint_post_recvs(ib_endpoint)) { >>> - BTL_ERROR(("Failed to post on XRC SRQs")); >>> - mca_btl_openib_endpoint_invoke_error(ib_endpoint); >>> - return; >>> - } >>> - OPAL_THREAD_LOCK(&ib_endpoint->endpoint_lock); >>> - rc = xoob_send_connect_data(ib_endpoint, >>> ENDPOINT_XOOB_CONNECT_XRC_RESPONSE); >>> - if (OMPI_SUCCESS != rc) { >>> - BTL_ERROR(("error in endpoint reply start connect")); >>> - mca_btl_openib_endpoint_invoke_error(ib_endpoint); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - return; >>> - } >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - } else { >>> - /* The XRC recv qp was destroyed */ >>> - OPAL_THREAD_LOCK(&ib_endpoint->endpoint_lock); >>> - rc = xoob_send_connect_data(ib_endpoint, >>> ENDPOINT_XOOB_CONNECT_XRC_NR_RESPONSE); >>> - if (OMPI_SUCCESS != rc) { >>> - BTL_ERROR(("error in endpoint reply start connect")); >>> - mca_btl_openib_endpoint_invoke_error(ib_endpoint); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - return; >>> - } >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - } >>> - /* enable pooling for this btl */ >>> - break; >>> - case ENDPOINT_XOOB_CONNECT_RESPONSE: >>> - BTL_VERBOSE(("Received ENDPOINT_XOOB_CONNECT_RESPONSE: lid %d, >>> sid %" PRIx64 "\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id)); >>> - ib_endpoint = xoob_find_endpoint(process_name, >>> rem_info.rem_subnet_id, >>> - rem_info.rem_lid, message_type); >>> - if ( NULL == ib_endpoint) { >>> - BTL_ERROR(("Got ENDPOINT_XOOB_CONNECT_RESPONSE." >>> - " Failed to find endpoint with subnet %" >>> PRIx64 " and LID %d", >>> - rem_info.rem_subnet_id,rem_info.rem_lid)); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - OPAL_THREAD_LOCK(&ib_endpoint->endpoint_lock); >>> - /* we got all the data srq. switch the endpoint to connect >>> mode */ >>> - XOOB_SET_REMOTE_INFO(ib_endpoint->rem_info, rem_info); >>> - /* update ib_addr with remote qp number */ >>> - ib_endpoint->ib_addr->remote_xrc_rcv_qp_num = >>> - ib_endpoint->rem_info.rem_qps->rem_qp_num; >>> - BTL_VERBOSE(("rem_info: lid %d, sid %" PRIx64 >>> - " ep %d %" PRIx64 "\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id, >>> - ib_endpoint->rem_info.rem_lid, >>> - ib_endpoint->rem_info.rem_subnet_id)); >>> - if (OMPI_SUCCESS != xoob_send_qp_connect(ib_endpoint, >>> &rem_info)) { >>> - BTL_ERROR(("Failed to connect endpoint\n")); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - OPAL_THREAD_UNLOCK(&ib_endpoint->endpoint_lock); >>> - return; >>> - } >>> - mca_btl_openib_endpoint_cpc_complete(ib_endpoint); >>> - /* cpc complete unlock the endpoint */ >>> - break; >>> - case ENDPOINT_XOOB_CONNECT_XRC_RESPONSE: >>> - BTL_VERBOSE(("Received ENDPOINT_XOOB_CONNECT_XRC_RESPONSE: lid >>> %d, sid %" PRIx64 "\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id)); >>> - ib_endpoint = xoob_find_endpoint(process_name, >>> rem_info.rem_subnet_id, >>> - rem_info.rem_lid, message_type); >>> - if ( NULL == ib_endpoint) { >>> - BTL_ERROR(("Got ENDPOINT_XOOB_CONNECT_XRC_RESPONSE." >>> - " Failed to find endpoint with subnet %" >>> PRIx64 " and LID %d", >>> - rem_info.rem_subnet_id,rem_info.rem_lid)); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - OPAL_THREAD_LOCK(&ib_endpoint->endpoint_lock); >>> - /* we got srq numbers on our request */ >>> - XOOB_SET_REMOTE_INFO(ib_endpoint->rem_info, rem_info); >>> - mca_btl_openib_endpoint_cpc_complete(ib_endpoint); >>> - /* cpc complete unlock the endpoint */ >>> - break; >>> - case ENDPOINT_XOOB_CONNECT_XRC_NR_RESPONSE: >>> - /* The XRC recv site already was destroyed so we need >>> - * start to bringup the connection from scratch */ >>> - BTL_VERBOSE(("Received ENDPOINT_XOOB_CONNECT_XRC_NR_RESPONSE: >>> lid %d, sid %" PRIx64 "\n", >>> - rem_info.rem_lid, >>> - rem_info.rem_subnet_id)); >>> - ib_endpoint = xoob_find_endpoint(process_name, >>> rem_info.rem_subnet_id, >>> - rem_info.rem_lid, message_type); >>> - if ( NULL == ib_endpoint) { >>> - BTL_ERROR(("Got ENDPOINT_XOOB_CONNECT_XRC_NR_RESPONSE." >>> - " Failed to find endpoint with subnet %" >>> PRIx64 " and LID %d", >>> - rem_info.rem_subnet_id,rem_info.rem_lid)); >>> - mca_btl_openib_endpoint_invoke_error(NULL); >>> - return; >>> - } >>> - xoob_restart_connect(ib_endpoint); >>> - break; >>> - default : >>> - BTL_ERROR(("Invalid message type %d", message_type)); >>> - } >>> - >>> - free_rem_info(&rem_info); >>> -} >>> - >>> -/* >>> - * XOOB interface functions >>> - */ >>> - >>> -/* Quere for the XOOB priority - will be highest in XRC case */ >>> -static int xoob_component_query(mca_btl_openib_module_t *openib_btl, >>> - ompi_btl_openib_connect_base_module_t **cpc) >>> -{ >>> - if (mca_btl_openib_component.num_xrc_qps <= 0) { >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: xoob CPC only supported with XRC >>> receive queues; skipped on %s:%d", >>> - >>> ibv_get_device_name(openib_btl->device->ib_dev), >>> - openib_btl->port_num); >>> - return OMPI_ERR_NOT_SUPPORTED; >>> - } >>> - >>> - *cpc = malloc(sizeof(ompi_btl_openib_connect_base_module_t)); >>> - if (NULL == *cpc) { >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: xoob CPC system error (malloc >>> failed)"); >>> - return OMPI_ERR_OUT_OF_RESOURCE; >>> - } >>> - >>> - /* If this btl supports XOOB, then post the RML message. But >>> - ensure to only post it *once*, because another btl may have >>> - come in before this and already posted it. */ >>> - if (!rml_recv_posted) { >>> - ompi_rte_recv_buffer_nb(OMPI_NAME_WILDCARD, >>> - OMPI_RML_TAG_XOPENIB, >>> - OMPI_RML_PERSISTENT, >>> - xoob_rml_recv_cb, >>> - NULL); >>> - rml_recv_posted = true; >>> - } >>> - >>> - if (xoob_priority > 100) { >>> - xoob_priority = 100; >>> - } else if (xoob_priority < -1) { >>> - xoob_priority = -1; >>> - } >>> - >>> - (*cpc)->data.cbm_component = &ompi_btl_openib_connect_xoob; >>> - (*cpc)->data.cbm_priority = xoob_priority; >>> - (*cpc)->data.cbm_modex_message = NULL; >>> - (*cpc)->data.cbm_modex_message_len = 0; >>> - >>> - (*cpc)->cbm_endpoint_init = NULL; >>> - (*cpc)->cbm_start_connect = xoob_module_start_connect; >>> - (*cpc)->cbm_endpoint_finalize = NULL; >>> - (*cpc)->cbm_finalize = NULL; >>> - (*cpc)->cbm_uses_cts = false; >>> - >>> - opal_output_verbose(5, ompi_btl_base_framework.framework_output, >>> - "openib BTL: xoob CPC available for use on %s:%d", >>> - ibv_get_device_name(openib_btl->device->ib_dev), >>> - openib_btl->port_num); >>> - return OMPI_SUCCESS; >>> -} >>> - >>> -/* Open - this functions sets up any xoob specific commandline params */ >>> -static void xoob_component_register(void) >>> -{ >>> - /* the priority is initialized in the declaration above */ >>> - (void) >>> mca_base_component_var_register(&mca_btl_openib_component.super.btl_version, >>> - "connect_xoob_priority", >>> - "The selection method priority >>> for xoob", >>> - MCA_BASE_VAR_TYPE_INT, NULL, 0, >>> 0, >>> - OPAL_INFO_LVL_9, >>> - MCA_BASE_VAR_SCOPE_READONLY, >>> - &xoob_priority); >>> - >>> - if (xoob_priority > 100) { >>> - xoob_priority = 100; >>> - } else if (xoob_priority < -1) { >>> - xoob_priority = -1; >>> - } >>> -} >>> - >>> -/* >>> - * Connect function. Start initiation of connections to a remote >>> - * peer. We send our Queue Pair information over the RML/OOB >>> - * communication mechanism. On completion of our send, a send >>> - * completion handler is called. >>> - */ >>> -static int xoob_module_start_connect(ompi_btl_openib_connect_base_module_t >>> *cpc, >>> - mca_btl_base_endpoint_t *endpoint) >>> -{ >>> - int rc = OMPI_SUCCESS; >>> - >>> - OPAL_THREAD_LOCK(&endpoint->ib_addr->addr_lock); >>> - switch (endpoint->ib_addr->status) { >>> - case MCA_BTL_IB_ADDR_CLOSED: >>> - BTL_VERBOSE(("The IB addr: sid %" PRIx64 " lid %d" >>> - "in MCA_BTL_IB_ADDR_CLOSED status," >>> - " sending ENDPOINT_XOOB_CONNECT_REQUEST\n", >>> - >>> endpoint->ib_addr->subnet_id,endpoint->ib_addr->lid)); >>> - if (OMPI_SUCCESS != (rc = xoob_send_qp_create(endpoint))) { >>> - break; >>> - } >>> - >>> - /* Send connection info over to remote endpoint */ >>> - endpoint->endpoint_state = MCA_BTL_IB_CONNECTING; >>> - endpoint->ib_addr->status = MCA_BTL_IB_ADDR_CONNECTING; >>> - if (OMPI_SUCCESS != >>> - (rc = xoob_send_connect_data(endpoint, >>> ENDPOINT_XOOB_CONNECT_REQUEST))) { >>> - BTL_ERROR(("Error sending connect request, error code %d", >>> rc)); >>> - } >>> - break; >>> - case MCA_BTL_IB_ADDR_CONNECTING: >>> - BTL_VERBOSE(("The IB addr: sid %" PRIx64 " lid %d" >>> - "in MCA_BTL_IB_ADDR_CONNECTING status," >>> - " Subscribing to this address\n", >>> - >>> endpoint->ib_addr->subnet_id,endpoint->ib_addr->lid)); >>> - /* some body already connectng to this machine, lets wait */ >>> - opal_list_append(&endpoint->ib_addr->pending_ep, >>> &(endpoint->super)); >>> - endpoint->endpoint_state = MCA_BTL_IB_CONNECTING; >>> - break; >>> - case MCA_BTL_IB_ADDR_CONNECTED: >>> - /* so we have the send qp, we just need the recive site. >>> - * Send request for SRQ numbers */ >>> - BTL_VERBOSE(("The IB addr: sid %" PRIx64 " lid %d" >>> - "in MCA_BTL_IB_ADDR_CONNECTED status," >>> - " sending ENDPOINT_XOOB_CONNECT_XRC_REQUEST\n", >>> - >>> endpoint->ib_addr->subnet_id,endpoint->ib_addr->lid)); >>> - endpoint->endpoint_state = MCA_BTL_IB_CONNECTING; >>> - if (OMPI_SUCCESS != >>> - (rc = xoob_send_connect_data(endpoint, >>> ENDPOINT_XOOB_CONNECT_XRC_REQUEST))) { >>> - BTL_ERROR(("error sending xrc connect request, error code >>> %d", rc)); >>> - } >>> - break; >>> - default : >>> - BTL_ERROR(("Invalid endpoint status %d", >>> endpoint->ib_addr->status)); >>> - } >>> - OPAL_THREAD_UNLOCK(&endpoint->ib_addr->addr_lock); >>> - return rc; >>> -} >>> - >>> - >>> -/* >>> - * Finalize function. Cleanup RML non-blocking receive. >>> - */ >>> -static int xoob_component_finalize(void) >>> -{ >>> - if (rml_recv_posted) { >>> - ompi_rte_recv_cancel(OMPI_NAME_WILDCARD, OMPI_RML_TAG_XOPENIB); >>> - rml_recv_posted = false; >>> - } >>> -#if (ENABLE_DYNAMIC_SL) >>> - btl_openib_connect_sl_finalize(); >>> -#endif >>> - return OMPI_SUCCESS; >>> -} >>> >>> Deleted: trunk/ompi/mca/btl/openib/connect/btl_openib_connect_xoob.h >>> ============================================================================== >>> --- trunk/ompi/mca/btl/openib/connect/btl_openib_connect_xoob.h >>> 2013-11-13 23:16:53 EST (Wed, 13 Nov 2013) (r29702) >>> +++ /dev/null 00:00:00 1970 (deleted) >>> @@ -1,19 +0,0 @@ >>> -/* >>> - * Copyright (c) 2007 Mellanox Technologies. All rights reserved. >>> - * Copyright (c) 2008 Cisco Systems, Inc. All rights reserved. >>> - * >>> - * $COPYRIGHT$ >>> - * >>> - * Additional copyrights may follow >>> - * >>> - * $HEADER$ >>> - */ >>> - >>> -#ifndef BTL_OPENIB_CONNECT_XOOB_H >>> -#define BTL_OPENIB_CONNECT_XOOB_H >>> - >>> -#include "connect/connect.h" >>> - >>> -extern ompi_btl_openib_connect_base_component_t >>> ompi_btl_openib_connect_xoob; >>> - >>> -#endif >>> _______________________________________________ >>> svn-full mailing list >>> svn-f...@open-mpi.org >>> http://www.open-mpi.org/mailman/listinfo.cgi/svn-full >> >> >> -- >> Jeff Squyres >> jsquy...@cisco.com >> For corporate legal information go to: >> http://www.cisco.com/web/about/doing_business/legal/cri/ >> >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel >> _______________________________________________ >> devel mailing list >> de...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/devel > > _______________________________________________ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
