Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
Oracle --
Is this really only specific to Solaris? More comments below about
configure.m4.
On Feb 16, 2011, at 12:37 AM, dk...@osl.iu.edu wrote:
> Author: dkerr
> Date: 2011-02-16 00:37:22 EST (Wed, 16 Feb 2011)
> New Revision: 24395
> URL: https://svn.open-mpi.org/trac/ompi/changeset/24395
>
> Log:
> on Solaris, when IBV_ACCESS_SO is available, use strong ordered memory region
> for eager rdma connection
> Text files modified:
> trunk/ompi/mca/btl/openib/btl_openib_component.c |13 ++---
>
> trunk/ompi/mca/btl/openib/btl_openib_endpoint.c |19
> +--
> trunk/ompi/mca/btl/openib/configure.m4 |16 +++-
>
> 3 files changed, 42 insertions(+), 6 deletions(-)
>
> Modified: trunk/ompi/mca/btl/openib/btl_openib_component.c
> ==
> --- trunk/ompi/mca/btl/openib/btl_openib_component.c (original)
> +++ trunk/ompi/mca/btl/openib/btl_openib_component.c 2011-02-16 00:37:22 EST
> (Wed, 16 Feb 2011)
> @@ -15,7 +15,7 @@
> * Copyright (c) 2006-2007 Los Alamos National Security, LLC. All rights
> * reserved.
> * Copyright (c) 2006-2007 Voltaire All rights reserved.
> - * Copyright (c) 2009-2010 Oracle and/or its affiliates. All rights
> reserved.
> + * Copyright (c) 2009-2011 Oracle and/or its affiliates. All rights
> reserved.
> * $COPYRIGHT$
> *
> * Additional copyrights may follow
> @@ -527,9 +527,16 @@
> {
> mca_btl_openib_device_t *device = (mca_btl_openib_device_t*)reg_data;
> mca_btl_openib_reg_t *openib_reg = (mca_btl_openib_reg_t*)reg;
> +enum ibv_access_flags access_flag = IBV_ACCESS_LOCAL_WRITE |
> +IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ;
>
> -openib_reg->mr = ibv_reg_mr(device->ib_pd, base, size,
> IBV_ACCESS_LOCAL_WRITE |
> -IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);
> +#if defined(HAVE_IBV_ACCESS_SO)
> +if (reg->flags & MCA_MPOOL_FLAGS_SO_MEM) {
> +access_flag |= IBV_ACCESS_SO;
> +}
> +#endif
> +
> +openib_reg->mr = ibv_reg_mr(device->ib_pd, base, size, access_flag);
>
> if (NULL == openib_reg->mr) {
> return OMPI_ERR_OUT_OF_RESOURCE;
>
> Modified: trunk/ompi/mca/btl/openib/btl_openib_endpoint.c
> ==
> --- trunk/ompi/mca/btl/openib/btl_openib_endpoint.c (original)
> +++ trunk/ompi/mca/btl/openib/btl_openib_endpoint.c 2011-02-16 00:37:22 EST
> (Wed, 16 Feb 2011)
> @@ -16,7 +16,7 @@
> * Copyright (c) 2006-2007 Voltaire All rights reserved.
> * Copyright (c) 2006-2009 Mellanox Technologies, Inc. All rights reserved.
> * Copyright (c) 2010 IBM Corporation. All rights reserved.
> - * Copyright (c) 2010 Oracle and/or its affiliates. All rights reserved
> + * Copyright (c) 2010-2011 Oracle and/or its affiliates. All rights reserved
> *
> * $COPYRIGHT$
> *
> @@ -911,6 +911,7 @@
> char *buf;
> mca_btl_openib_recv_frag_t *headers_buf;
> int i;
> +uint32_t flag = MCA_MPOOL_FLAGS_CACHE_BYPASS;
>
> /* Set local rdma pointer to 1 temporarily so other threads will not try
> * to enter the function */
> @@ -925,11 +926,25 @@
> if(NULL == headers_buf)
>goto unlock_rdma_local;
>
> +#if defined(HAVE_IBV_ACCESS_SO)
> +/* Solaris implements the Relaxed Ordering feature defined in the
> + PCI Specification. With this in mind any memory region which
> + relies on a buffer being written in a specific order, for
> + example the eager rdma connections created in this routinue,
> + must set a strong order flag when registering the memory for
> + rdma operations.
> +
> + The following flag will be interpreted and the appropriate
> + steps will be taken when the memory is registered in
> + openib_reg_mr(). */
> +flag |= MCA_MPOOL_FLAGS_SO_MEM;
> +#endif
> +
> buf = (char *)
> openib_btl->super.btl_mpool->mpool_alloc(openib_btl->super.btl_mpool,
> openib_btl->eager_rdma_frag_size *
> mca_btl_openib_component.eager_rdma_num,
> mca_btl_openib_component.buffer_alignment,
> -MCA_MPOOL_FLAGS_CACHE_BYPASS,
> +flag,
> (mca_mpool_base_registration_t**)&endpoint->eager_rdma_local.reg);
>
> if(!buf)
>
> Modified: trunk/ompi/mca/btl/openib/configure.m4
> ==
> --- trunk/ompi/mca/btl/openib/configure.m4(original)
> +++ trunk/ompi/mca/btl/openib/configure.m42011-02-16 00:37:22 EST (Wed,
> 16 Feb 2011)
> @@ -12,6 +12,7 @@
> # All rights reserved.
> # Copyright (c) 2007-2010 Cisco Systems, Inc. All rights reserved.
> # Copyright (c) 2008 Mellanox Technologies. All rights reserved.
> +# Copyright (c) 2011 Oracle and/or
Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
Yes this is Solaris only. OFED has not bought back the IBV_ACCESS_SO
flag. Not sure they ever will.
On 02/16/11 08:15, Jeff Squyres wrote:
Oracle --
Is this really only specific to Solaris? More comments below about
configure.m4.
On Feb 16, 2011, at 12:37 AM, dk...@osl.iu.edu wrote:
Author: dkerr
Date: 2011-02-16 00:37:22 EST (Wed, 16 Feb 2011)
New Revision: 24395
URL: https://svn.open-mpi.org/trac/ompi/changeset/24395
Log:
on Solaris, when IBV_ACCESS_SO is available, use strong ordered memory region
for eager rdma connection
Text files modified:
trunk/ompi/mca/btl/openib/btl_openib_component.c |13 ++---
trunk/ompi/mca/btl/openib/btl_openib_endpoint.c |19 +--
trunk/ompi/mca/btl/openib/configure.m4 |16 +++-
3 files changed, 42 insertions(+), 6 deletions(-)
Modified: trunk/ompi/mca/btl/openib/btl_openib_component.c
==
--- trunk/ompi/mca/btl/openib/btl_openib_component.c(original)
+++ trunk/ompi/mca/btl/openib/btl_openib_component.c2011-02-16 00:37:22 EST
(Wed, 16 Feb 2011)
@@ -15,7 +15,7 @@
* Copyright (c) 2006-2007 Los Alamos National Security, LLC. All rights
* reserved.
* Copyright (c) 2006-2007 Voltaire All rights reserved.
- * Copyright (c) 2009-2010 Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2009-2011 Oracle and/or its affiliates. All rights reserved.
* $COPYRIGHT$
*
* Additional copyrights may follow
@@ -527,9 +527,16 @@
{
mca_btl_openib_device_t *device = (mca_btl_openib_device_t*)reg_data;
mca_btl_openib_reg_t *openib_reg = (mca_btl_openib_reg_t*)reg;
+enum ibv_access_flags access_flag = IBV_ACCESS_LOCAL_WRITE |
+IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ;
-openib_reg->mr = ibv_reg_mr(device->ib_pd, base, size,
IBV_ACCESS_LOCAL_WRITE |
-IBV_ACCESS_REMOTE_WRITE | IBV_ACCESS_REMOTE_READ);
+#if defined(HAVE_IBV_ACCESS_SO)
+if (reg->flags & MCA_MPOOL_FLAGS_SO_MEM) {
+access_flag |= IBV_ACCESS_SO;
+}
+#endif
+
+openib_reg->mr = ibv_reg_mr(device->ib_pd, base, size, access_flag);
if (NULL == openib_reg->mr) {
return OMPI_ERR_OUT_OF_RESOURCE;
Modified: trunk/ompi/mca/btl/openib/btl_openib_endpoint.c
==
--- trunk/ompi/mca/btl/openib/btl_openib_endpoint.c (original)
+++ trunk/ompi/mca/btl/openib/btl_openib_endpoint.c 2011-02-16 00:37:22 EST
(Wed, 16 Feb 2011)
@@ -16,7 +16,7 @@
* Copyright (c) 2006-2007 Voltaire All rights reserved.
* Copyright (c) 2006-2009 Mellanox Technologies, Inc. All rights reserved.
* Copyright (c) 2010 IBM Corporation. All rights reserved.
- * Copyright (c) 2010 Oracle and/or its affiliates. All rights reserved
+ * Copyright (c) 2010-2011 Oracle and/or its affiliates. All rights reserved
*
* $COPYRIGHT$
*
@@ -911,6 +911,7 @@
char *buf;
mca_btl_openib_recv_frag_t *headers_buf;
int i;
+uint32_t flag = MCA_MPOOL_FLAGS_CACHE_BYPASS;
/* Set local rdma pointer to 1 temporarily so other threads will not try
* to enter the function */
@@ -925,11 +926,25 @@
if(NULL == headers_buf)
goto unlock_rdma_local;
+#if defined(HAVE_IBV_ACCESS_SO)
+/* Solaris implements the Relaxed Ordering feature defined in the
+ PCI Specification. With this in mind any memory region which
+ relies on a buffer being written in a specific order, for
+ example the eager rdma connections created in this routinue,
+ must set a strong order flag when registering the memory for
+ rdma operations.
+
+ The following flag will be interpreted and the appropriate
+ steps will be taken when the memory is registered in
+ openib_reg_mr(). */
+flag |= MCA_MPOOL_FLAGS_SO_MEM;
+#endif
+
buf = (char *)
openib_btl->super.btl_mpool->mpool_alloc(openib_btl->super.btl_mpool,
openib_btl->eager_rdma_frag_size *
mca_btl_openib_component.eager_rdma_num,
mca_btl_openib_component.buffer_alignment,
-MCA_MPOOL_FLAGS_CACHE_BYPASS,
+flag,
(mca_mpool_base_registration_t**)&endpoint->eager_rdma_local.reg);
if(!buf)
Modified: trunk/ompi/mca/btl/openib/configure.m4
==
--- trunk/ompi/mca/btl/openib/configure.m4 (original)
+++ trunk/ompi/mca/btl/openib/configure.m4 2011-02-16 00:37:22 EST (Wed,
16 Feb 2011)
@@ -12,6 +12,7 @@
# All rights reserved.
# Copyright (c) 2007-2010 Cisco Systems, Inc. All rights reserved.
# Copyright (c) 2008 Mellanox Technologies. All rights reserved.
+# Copyright (c) 2011 Oracle and/or its affiliates. All rights reserved.
# $COPYRIGHT$
#
# Additional copyrights
Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
On Feb 16, 2011, at 8:29 AM, Don Kerr wrote: > Yes this is Solaris only. OFED has not bought back the IBV_ACCESS_SO flag. > Not sure they ever will. It should be sufficient to AC_CHECK_DECLS then -- no need for the additional Solaris check. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
I considered that but I wanted to guard against future OFED inclusion. Removing the Solaris check is easy enough. On 02/16/11 08:49, Jeff Squyres wrote: On Feb 16, 2011, at 8:29 AM, Don Kerr wrote: Yes this is Solaris only. OFED has not bought back the IBV_ACCESS_SO flag. Not sure they ever will. It should be sufficient to AC_CHECK_DECLS then -- no need for the additional Solaris check.
Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
If OFED includes that constant, wouldn't we want to use it? PCI ordering is PCI ordering (i.e., unreliable) on all hardware -- or am I wrong? On Feb 16, 2011, at 8:59 AM, Don Kerr wrote: > I considered that but I wanted to guard against future OFED inclusion. > Removing the Solaris check is easy enough. > > > On 02/16/11 08:49, Jeff Squyres wrote: >> On Feb 16, 2011, at 8:29 AM, Don Kerr wrote: >>> Yes this is Solaris only. OFED has not bought back the IBV_ACCESS_SO flag. >>> Not sure they ever will. >> It should be sufficient to AC_CHECK_DECLS then -- no need for the additional >> Solaris check. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24395
You would think but I did not want to speculate what OFED might do. I'm fine skipping the Solaris check, if OFED does include things may have to change at that point anyway. On 02/16/11 09:41, Jeff Squyres wrote: If OFED includes that constant, wouldn't we want to use it? PCI ordering is PCI ordering (i.e., unreliable) on all hardware -- or am I wrong? On Feb 16, 2011, at 8:59 AM, Don Kerr wrote: I considered that but I wanted to guard against future OFED inclusion. Removing the Solaris check is easy enough. On 02/16/11 08:49, Jeff Squyres wrote: On Feb 16, 2011, at 8:29 AM, Don Kerr wrote: Yes this is Solaris only. OFED has not bought back the IBV_ACCESS_SO flag. Not sure they ever will. It should be sufficient to AC_CHECK_DECLS then -- no need for the additional Solaris check.
[OMPI devel] Minor OMPI SVN configuration change
We have made a minor change in OMPI's SVN configuration: If you access the SVN repo over https, you *must* authenticate. Previously, you could authenticate or not; SVN would serve up what was appropriate in either case (there's one "private" tree in the SVN repo for not-yet-published academic work). Anonymous, read-only access to SVN is still provided over http. I doubt that this change will affect most people, because most people check out via https only if they plan to commit -- meaning that they have a write-enabled SVN account. Let me know if you have any questions / problems with this change. -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
Re: [OMPI devel] --enable-opal-multi-threads
I just filed CMR 2728 about this. We should fix the README for v1.5.2 so that it can get out the door, and then apply r22841 for v1.5.3. On Feb 15, 2011, at 4:53 PM, Eugene Loh wrote: > Ralph Castain wrote: >> On Feb 15, 2011, at 9:24 AM, Eugene Loh wrote: >> >> >>> Ralph Castain wrote: >>> >>> On Feb 14, 2011, at 9:26 PM, Abhishek Kulkarni wrote: > On Mon, 14 Feb 2011, Ralph Castain wrote: > > >> If the ability to turn "on" thread support is missing from 1.5, then >> that is an error. >> >> > No, it was changed from "--enable-mpi-threads" to > "--enable-opal-multi-threads" on the trunk in r22841 [1]. > > If the changeset has not been brought over to v1.5, it indeed looks like > an anachronism in the README. > > [1] > https://svn.open-mpi.org/trac/ompi/changeset/22841 > > > My point is that it isn't an anachronism in the README, but an error in 1.5 - it needs to have the ability to turn on thread safety. >>> I'm not sure if we're making progress here. So far as I can tell, the v1.5 >>> README talks about --enable-opal-multi-threads. This option does not >>> otherwise appear in v1.5, but only in the trunk. So, to make the v1.5 >>> README consistent with the v1.5 source code (as opposed to talking about >>> features that will appear in unspecified future releases), either: >>> >>> *) the comment should be removed from the README, or >>> >>> *) opal-multi-threads should be CMRed to v1.5 >>> >>> >> My point is that option 2 needs to be done - not sure how much clearer I can >> be :-) >> >> > Are we talking about CMRing such a change (r22841 perhaps?) to 1.5.2? If so, > is there already such a CMR? Or, should I (or someone better suited for the > job) file such a CMR? Which changesets? Candidate for reviewer? > > Or, are we talking about CMRing to 1.5.3? If so, the comment in question > should be pulled out of the 1.5.2 README. > > I understand opal-multi-threads is A Good Thing. I'm just against touting it > in the README file when it doesn't yet exist in the bits (equivalently, > putting it in the 1.5 README when the functionality is only in the trunk). >> On Feb 14, 2011, at 5:36 PM, Eugene Loh wrote: >> >> >>> In the v1.5 README, I see this: >>> >>> --enable-opal-multi-threads >>> Enables thread lock support in the OPAL and ORTE layers. Does >>> not enable MPI_THREAD_MULTIPLE - see above option for that feature. >>> This is currently disabled by default. >>> >>> I don't otherwise find opal-multi-threads at all in this branch. It >>> seems to me, for such an option, one needs to move to the trunk. >>> >>> Is this an error (anachronism) in the v1.5 README? >>> >>> > ___ > devel mailing list > de...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/devel -- Jeff Squyres jsquy...@cisco.com For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
