Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24836
Hi Jeff, Thanks, fixed - see r24852. Basically, I did what you wrote here, with some small modifications. Hope that this is OK this time. -- YK On 29-Jun-11 5:59 PM, Jeff Squyres wrote: > That's still not quite right, per OMPI conventions. > > If you don't find it, you shouldn't warn, you should have some other > AC_MSG_CHECKING and then indicate that that particular check failed. > > E.g. > > AC_MSG_CHECKING([if can use dynamic SL support]) > AS_IF([test "$1_have_dynamic_sl" = "1"], >[AC_MSG_RESULT([yes])], >[AC_MSG_RESULT([no]) > AS_IF([test "$enable_openib_dynamic_sl" = "yes"], > [AC_MSG_WARN([--enable-openib-dynamic-sl was specified but the]) >AC_MSG_WARN([appropriate header files could not be found.]) >AC_MSG_ERROR([Cannot continue])]) >]) > > (I just typed that in my mail client; I don't know if all the [] and () match > properly) > > > > On Jun 29, 2011, at 10:52 AM, klit...@osl.iu.edu wrote: > >> Author: kliteyn >> Date: 2011-06-29 10:52:11 EDT (Wed, 29 Jun 2011) >> New Revision: 24836 >> URL: https://svn.open-mpi.org/trac/ompi/changeset/24836 >> >> Log: >> Changed default behavior when opensm-devel pachege not found - warn, not exit >> >> >> Text files modified: >>trunk/ompi/config/ompi_check_openib.m4 | 2 +- >>1 files changed, 1 insertions(+), 1 deletions(-) >> >> Modified: trunk/ompi/config/ompi_check_openib.m4 >> == >> --- trunk/ompi/config/ompi_check_openib.m4 (original) >> +++ trunk/ompi/config/ompi_check_openib.m4 2011-06-29 10:52:11 EDT (Wed, >> 29 Jun 2011) >> @@ -195,7 +195,7 @@ >> # ib_types.h, but it doesn't include any other IB-related >> files. >> AC_CHECK_HEADER([infiniband/complib/cl_types_osd.h], >> [$1_have_dynamic_sl=1], >> - [AC_MSG_ERROR([opensm-devel package not >> found - please install it or disable dynamic SL support with >> \"--disable-openib-dynamic-sl\"])], >> + [AC_MSG_WARN([opensm-devel package not found >> - please install it or disable dynamic SL support with >> \"--disable-openib-dynamic-sl\"])], >> []) >> fi >> >> ___ >> svn-full mailing list >> svn-f...@open-mpi.org >> http://www.open-mpi.org/mailman/listinfo.cgi/svn-full > >
Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24830
Hi Jeff, On 02-Jul-11 11:52 PM, Jeff Squyres (jsquyres) wrote: > Were all the issueswith this code fixed? There were m4 issues and solaris > issues, IIRC. I took all the fixes I could find based on the trac: "Be sure also to look at r24196; Josh committed a bunch of warning fixes for you after r24915" I also removed all the libibmad dependencies and unneded macros, so I hope that this is OK. However, I don't have any Solaris machine to try this to make sure that there are no issues. The only complaint w.r.t. Solaris that I could find was Terry's mail from last week, but it turned out to be a different problem. Are there any other problems that I'm not aware of? -- YK > Sent from my phone. No type good. > > On Jun 28, 2011, at 9:28 AM, "klit...@osl.iu.edu" wrote: > >> Author: kliteyn >> Date: 2011-06-28 10:28:29 EDT (Tue, 28 Jun 2011) >> New Revision: 24830 >> URL: https://svn.open-mpi.org/trac/ompi/changeset/24830 >> >> Log: >> Supporting dynamic SL (#2674) >> >> - Added enable/disable configuration parameter for dynamic SL >> - All the dynamic SL code is conditionalized >> - Removed libibmad dependency >> - Using only one include - ib_types.h (part of opensm-devel package) >> - Removed all the macro and data types definitions, using the >>existing definitions from ib_types.h instead >> - general cleaning here and there >> >> The async mode is not implemented yet - stay tuned... >> >> >> Text files modified: >>trunk/ompi/config/ompi_check_openib.m4 |38 >>trunk/ompi/mca/btl/openib/btl_openib.h | 5 >>trunk/ompi/mca/btl/openib/btl_openib_mca.c |10 >>trunk/ompi/mca/btl/openib/connect/btl_openib_connect_oob.c | 309 >> +-- >>4 files changed, 182 insertions(+), 180 deletions(-) >> >> Modified: trunk/ompi/config/ompi_check_openib.m4 >> == >> --- trunk/ompi/config/ompi_check_openib.m4(original) >> +++ trunk/ompi/config/ompi_check_openib.m42011-06-28 10:28:29 EDT (Tue, >> 28 Jun 2011) >> @@ -155,11 +155,21 @@ >> [$ompi_cv_func_ibv_create_cq_args], >> [Number of arguments to >> ibv_create_cq])])]) >> >> +# >> +# OpenIB dynamic SL >> +# >> +AC_ARG_ENABLE([openib-dynamic-sl], >> +[AC_HELP_STRING([--enable-openib-dynamic-sl], >> +[Enable openib BTL to query Subnet Manager for IB >> SL (default: enabled)])], >> +[enable_openib_dynamic_sl="$enableval"], >> +[enable_openib_dynamic_sl="yes"]) >> + >> # Set these up so that we can do an AC_DEFINE below >> # (unconditionally) >> $1_have_xrc=0 >> $1_have_rdmacm=0 >> $1_have_ibcm=0 >> +$1_have_dynamic_sl=0 >> >> # If we have the openib stuff available, find out what we've got >> AS_IF([test "$ompi_check_openib_happy" = "yes"], >> @@ -176,6 +186,19 @@ >> AC_CHECK_FUNCS([ibv_create_xrc_rcv_qp], [$1_have_xrc=1]) >> fi >> >> + if test "$enable_openib_dynamic_sl" = "yes"; then >> + # We need ib_types.h file, which is installed with >> opensm-devel >> + # package. However, ib_types.h has a bad include directive, >> + # which will cause AC_CHECK_HEADER to fail. >> + # So instead, we will look for another file that is also >> + # installed as part of opensm-devel package and included in >> + # ib_types.h, but it doesn't include any other IB-related >> files. >> + AC_CHECK_HEADER([infiniband/complib/cl_types_osd.h], >> + [$1_have_dynamic_sl=1], >> + [AC_MSG_ERROR([opensm-devel package not >> found - please install it or disable dynamic SL support with >> \"--disable-openib-dynamic-sl\"])], >> + []) >> + fi >> + >> # Do we have a recent enough RDMA CM? Need to have the >> # rdma_get_peer_addr (inline) function (originally appeared >> # in OFED v1.3). >> @@ -244,6 +267,15 @@ >> else >> AC_MSG_RESULT([no]) >> fi >> + >> +AC_MSG_CHECKING([if dynamic SL is enabled]) >> +AC_DEFINE_UNQUOTED([OMPI_ENABLE_DYNAMIC_SL], [$$1_have_dynamic_sl], >> +[Enable features required for dynamic SL support]) >> +if test "1" = "$$1_have_dynamic_sl"; then >> +AC_MSG_RESULT([yes]) >> +else >> +AC_MSG_RESULT([no]) >> +fi >> >> AC_MSG_CHECKING([if OpenFabrics RDMACM support is enabled]) >> AC_DEFINE_UNQUOTED([OMPI_HAVE_RDMACM], [$$1_have_rdmacm], >> @@ -267,7 +299,11 @@ >> AC_MSG_RESULT([no]) >> fi >> >> -CPPFLAGS="$ompi_check_openib_$1_save_CPPFLAGS" >> +AS_IF([test -z "$ompi_check_openib_dir"], >> + [openib_include_dir="/usr/inc
Re: [OMPI devel] "Open MPI"-based MPI library used by K computer
Hi Jeff,
> Does your llp sed path order MPI matching ordering? Eg if some prior isend
> is already queued, could the llp send overtake it?
Yes, LLP send may overtake queued isend.
But we use correct PML send_sequence. So the LLP message is queued as
unexpected message on receiver side, and I think it's no problem.
> >rc = MCA_LLP_CALL(send(buf, size, OMPI_PML_OB1_MATCH_HDR_LEN,
> > (bool)OMPI_ENABLE_OB1_PAD_MATCH_HDR,
> > ompi_comm_peer_lookup(comm, dst),
> > MCA_PML_OB1_HDR_TYPE_MATCH));
> >
> >if (rc == OMPI_SUCCESS) {
> >/* NOTE this is not thread safe */
> >OPAL_THREAD_ADD32(&proc->send_sequence, 1);
> >}
Takahiro Kawashima,
MPI development team,
Fujitsu
> Does your llp sed path order MPI matching ordering? Eg if some prior isend
> is already queued, could the llp send overtake it?
>
> Sent from my phone. No type good.
>
> On Jun 29, 2011, at 8:27 AM, "Kawashima" wrote:
>
> > Hi Jeff,
> >
> >>> First, we created a new BTL component, 'tofu BTL'. It's not so special
> >>> one but dedicated to our Tofu interconnect. But its latency was not
> >>> enough for us.
> >>>
> >>> So we created a new framework, 'LLP', and its component, 'tofu LLP'.
> >>> It bypasses request object creation in PML and BML/BTL, and sends
> >>> a message immediately if possible.
> >>
> >> Gotcha. Was the sendi pml call not sufficient? (sendi = "send
> >> immediate") This call was designed to be part of a latency reduction
> >> mechanism. I forget offhand what we don't do before calling sendi, but
> >> the rationale was that if the message was small enough, we could skip some
> >> steps in the sending process and "just send it."
> >
> > I know sendi, but its latency was not sufficient for us.
> > To come at sendi call, we must do:
> > - allocate send request (MCA_PML_OB1_SEND_REQUEST_ALLOC)
> > - initialize send request (MCA_PML_OB1_SEND_REQUEST_INIT)
> > - select BTL module (mca_pml_ob1_send_request_start)
> > - select protocol (mca_pml_ob1_send_request_start_btl)
> > We want to eliminate these overheads. We want to send more immediately.
> >
> > Here is a code snippet:
> >
> >
> >
> > #if OMPI_ENABLE_LLP
> > static inline int mca_pml_ob1_call_llp_send(void *buf,
> >size_t size,
> >int dst,
> >int tag,
> >ompi_communicator_t *comm)
> > {
> >int rc;
> >mca_pml_ob1_comm_proc_t *proc = &comm->c_pml_comm->procs[dst];
> >mca_pml_ob1_match_hdr_t *match = mca_pml_ob1.llp_send_buf;
> >
> >match->hdr_common.hdr_type = MCA_PML_OB1_HDR_TYPE_MATCH;
> >match->hdr_common.hdr_flags = 0;
> >match->hdr_ctx = comm->c_contextid;
> >match->hdr_src = comm->c_my_rank;
> >match->hdr_tag = tag;
> >match->hdr_seq = proc->send_sequence + 1;
> >
> >rc = MCA_LLP_CALL(send(buf, size, OMPI_PML_OB1_MATCH_HDR_LEN,
> > (bool)OMPI_ENABLE_OB1_PAD_MATCH_HDR,
> > ompi_comm_peer_lookup(comm, dst),
> > MCA_PML_OB1_HDR_TYPE_MATCH));
> >
> >if (rc == OMPI_SUCCESS) {
> >/* NOTE this is not thread safe */
> >OPAL_THREAD_ADD32(&proc->send_sequence, 1);
> >}
> >
> >return rc;
> > }
> > #endif
> >
> > int mca_pml_ob1_send(void *buf,
> > size_t count,
> > ompi_datatype_t * datatype,
> > int dst,
> > int tag,
> > mca_pml_base_send_mode_t sendmode,
> > ompi_communicator_t * comm)
> > {
> >int rc;
> >mca_pml_ob1_send_request_t *sendreq;
> >
> > #if OMPI_ENABLE_LLP
> >/* try to send message via LLP if
> > * - one of LLP modules is available, and
> > * - datatype is basic, and
> > * - data is small, and
> > * - communication mode is standard, buffered, or ready, and
> > * - destination is not myself
> > */
> >if (((datatype->flags & DT_FLAG_BASIC) == DT_FLAG_BASIC) &&
> >(datatype->size * count < mca_pml_ob1.llp_max_payload_size) &&
> >(sendmode == MCA_PML_BASE_SEND_STANDARD ||
> > sendmode == MCA_PML_BASE_SEND_BUFFERED ||
> > sendmode == MCA_PML_BASE_SEND_READY) &&
> >(dst != comm->c_my_rank)) {
> >rc = mca_pml_ob1_call_llp_send(buf, datatype->size * count, dst,
> > tag, comm);
> >if (rc != OMPI_ERR_NOT_AVAILABLE) {
> >/* successfully sent out via LLP or unrecoverable error occurred
> > */
> >return rc;
> >}
> >}
> > #endif
> >
> >MCA_PML_OB1_SEND_REQUEST_ALLOC(comm, dst, sendreq, rc);
> >if (rc != OMPI_SUCCESS)
> >return rc;
> >
> >MCA_PML_OB1_SEND_REQUEST_INIT(sendreq,
> >
