Re: [OMPI devel] [TIPC BTL] test programmes

[Xin He]

thank you all for replying me and giving me useful suggestions. I know 
where to start now. :-)



/Xin

On 08/02/2011 12:03 AM, Eugene Loh wrote:

   NAS Parallel Benchmarks are self-verifying.

Another option is the MPI Testing Tool
http://www.open-mpi.org/projects/mtt/ but it might be more trouble than
it's worth.

(INCIDENTALLY, THERE ARE TRAC TROUBLES WITH THE THREE LINKS AT THE
BOTTOM OF THAT PAGE!  COULD SOMEONE TAKE A LOOK?)

If you do decide to explore MTT,
http://www.open-mpi.org/projects/mtt/svn.php tells you how to do a
Subversion checkout.  It's a test harness.  For the tests themselves,
look in mtt/trunk/samples/*-template.ini for examples of what tests to
run.  Whether you want to pursue this route depends on whether you're
serious about doing lots of testing.

On 08/01/11 17:13, Jeff Squyres wrote:

Additionally, you might want to download an run a bunch of common MPI 
benchmarks, such as:

- Netpipe
- Intel MPI Benchmarks (IMB)
- SKaMPI
- HPL (Linpack)
- ...etc.

On Aug 1, 2011, at 8:12 AM, Chris Samuel wrote:

On Mon, 1 Aug 2011 09:47:00 PM Xin He wrote:

Do any of you guys have any testing programs that I should
run to test if it really works?

How about a real MPI program which has test data to check
it's running OK ?  Gromacs is open source and has a self-test
mechanism run via "make test" IIRC.

I think HPL (Linpack) also checks the data from its run..

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] "The MPI_Init() function was called before MPI_INIT was invoked."

[Josh Hursey]

Thanks for the bug report. We should probably cleanup the reporting of
this type of error. I think someone was looking into the "*** The
MPI_Init() function was called before MPI_INIT was invoked." issue
since it came up in a different bug report.

But I'll file a ticket on this so when we get back into the code we
can try to address it.
   https://svn.open-mpi.org/trac/ompi/ticket/2841

Thanks,
Josh


On Fri, Jul 22, 2011 at 8:30 PM, Paul H. Hargrove  wrote:
> The output below resulted from an attempt to start a job w/checkpoint
> support when the BLCR kernel modules were not loaded on then node ("pilot
> error").  The mistake is mine, but I am reporting this because there appears
> to be more going on in the output than probable should be -  the following 2
> lines in particular struck me as almost humorous, but clearly incorrect:
>
> *** The MPI_Init() function was called before MPI_INIT was invoked.
> *** This is disallowed by the MPI standard.
>
> Below is the command and full output.  This is OMPI-1.5.3 on Linux/x86.
>
> -Paul
>
> $ mpirun --prefix $HOME/obj-pcp-j/cr_mpirun-j-5+6/INSTALL -host pcp-j-6
> --mca btl ^openib -am ft-enable-cr -np 1 ./ring
> --
> It looks like opal_init failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during opal_init; some of which are due to configuration or
> environment problems.  This failure appears to be an internal failure;
> here's some additional information (which may only be relevant to an
> Open MPI developer):
>
>  opal_cr_init() failed failed
>  --> Returned value -1 instead of OPAL_SUCCESS
> --
> [pcp-j-6:29247] [[INVALID],INVALID] ORTE_ERROR_LOG: Error in file
> ../../../openmpi-1.5.3/orte/runtime/orte_init.c at line 79
> *** The MPI_Init() function was called before MPI_INIT was invoked.
> *** This is disallowed by the MPI standard.
> *** Your MPI job will now abort.
> [pcp-j-6:29247] Abort before MPI_INIT completed successfully; not able to
> guarantee that all other processes were killed!
> --
> It looks like MPI_INIT failed for some reason; your parallel process is
> likely to abort.  There are many reasons that a parallel process can
> fail during MPI_INIT; some of which are due to configuration or environment
> problems.  This failure appears to be an internal failure; here's some
> additional information (which may only be relevant to an Open MPI
> developer):
>
>  ompi_mpi_init: orte_init failed
>  --> Returned "Error" (-1) instead of "Success" (0)
> --
> --
> mpirun noticed that the job aborted, but has no info as to the process
> that caused that situation.
> --
>
>
> --
> Paul H. Hargrove                          phhargr...@lbl.gov
> Future Technologies Group
> HPC Research Department                   Tel: +1-510-495-2352
> Lawrence Berkeley National Laboratory     Fax: +1-510-486-6900
>
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>



-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey

Re: [OMPI devel] Using BLCR tools to checkpoint Open MPI applications

[Josh Hursey]

Eric,

Thanks for the great work on this integration. I filed a ticket for
the problem areas that you highlighted with the Open MPI side of the
integration so we do not lose track of them.
   https://svn.open-mpi.org/trac/ompi/ticket/2842

Hopefully we will get some cycles to address these issues in the near term.

Thanks,
Josh

On Wed, Jul 27, 2011 at 3:52 PM, Eric Roman  wrote:
>
> Dear Open MPI Developers,
>
> We've been working on using Torque's checkpoint/restart support, along with 
> BLCR
> and Open MPI's C/R support, to perform C/R on parallel jobs running under
> Torque.  The main issue here is that Open MPI requires the use of
> ompi-checkpoint and ompi-restart commands to checkpoint the application, but
> Torque uses cr_checkpoint and cr_restart to checkpoint job scripts, so an
> adapter is required between the two interfaces.  I've written a small program,
> called cr_mpirun, that meets this purpose.
>
> This code is now available on the BLCR web site that should enable you to use
> BLCR cr_checkpoint and cr_restart commands to checkpoint Open MPI 
> applications.
> You can download it at the following URL:
>
> https://upc-bugs.lbl.gov/blcr-dist/cr_mpirun/cr_mpirun-210.tar.gz
>
> This code can be used fairly reliably to invoke cr_checkpoint and cr_restart 
> on
> Open MPI applications.  In turn, this enables you to use Torque's
> checkpoint/restart support on parallel jobs.  I've tested mainly with qhold 
> and
> qrls, but have also experimented with using Maui's preemptee and preemptor
> classes.
>
> This release is intended as a development release, meaning that this release 
> is
> not suitable for general production use, but should be used for testing.  
> There
> are a number of issues that need to be worked out, and we need feedback from
> Torque and Open MPI developers, and from users interested in testing or filing
> bug reports.
>
> There is a list of known issues documented in the BUGS file in the release.
> There are HOWTO files in the release that describe the implementation,
> workarounds for current problems, and usage of cr_mpirun.
>
> Thanks for your interest.
>
> Sincerely,
> Eric Roman
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel
>
>



-- 
Joshua Hursey
Postdoctoral Research Associate
Oak Ridge National Laboratory
http://users.nccs.gov/~jjhursey

Re: [OMPI devel] RFC: CUDA register sm and openib host memory

[Jeff Squyres (jsquyres)]

Rolf -

Can you send a cumulative SVN diff against the SVN HEAD?

Sent from my phone. No type good. 

On Jul 28, 2011, at 5:52 PM, "Rolf vandeVaart"  wrote:

> WHAT: Add CUDA registration of host memory in sm and openib BTLs.
> 
>  
> 
> TIMEOUT: 8/4/2011
> 
>  
> 
> DETAILS: In order to improve performance of sending GPU device memory,
> 
> we need to register the host memory with the CUDA framework.  These
> 
> changes allow that to happen.  These changes are somewhat different
> 
> from what I proposed a while ago and I think a lot cleaner.  There is
> 
> a new memory pool flag that indicates whether a piece of memory
> 
> should be registered.  This allows us to register the sm memory and
> 
> the pre-posted openib memory.
> 
>  
> 
> The CUDA specific code is in the ompi/mca/common/cuda directory.
> 
>  
> 
> Do not look at the configure.m4 code, as that is still not done.
> 
>  
> 
> Here a link to the proposed changes:
> 
> https://bitbucket.org/rolfv/ompi-cuda-register
> 
>  
> 
> Here is a list of files that would change.
> 
> M   VERSION
> 
> M   configure.ac
> 
> M   ompi/mca/btl/openib/btl_openib_component.c
> 
> M   ompi/mca/btl/openib/Makefile.am
> 
> M   ompi/mca/mpool/sm/Makefile.am
> 
> M   ompi/mca/mpool/sm/mpool_sm_module.c
> 
> M   ompi/mca/mpool/mpool.h
> 
> M   ompi/mca/pml/ob1/pml_ob1_sendreq.h
> 
> A   ompi/mca/common/cuda
> 
> A   ompi/mca/common/cuda/configure.m4
> 
> A   ompi/mca/common/cuda/common_cuda.c
> 
> A   ompi/mca/common/cuda/help-mpi-common-cuda.txt
> 
> A   ompi/mca/common/cuda/Makefile.am
> 
> A   ompi/mca/common/cuda/common_cuda.h
> 
> M   ompi/class/ompi_free_list.c
> 
>  
> 
>  
> 
>  
> 
> This email message is for the sole use of the intended recipient(s) and may 
> contain confidential information.  Any unauthorized review, use, disclosure 
> or distribution is prohibited.  If you are not the intended recipient, please 
> contact the sender by reply email and destroy all copies of the original 
> message.
> ___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel

Re: [OMPI devel] RFC: CUDA register sm and openib host memory

[Rolf vandeVaart]

Yes, here it is attached.

Rolf

rvandeva...@nvidia.com
781-275-5358

From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org] On Behalf 
Of Jeff Squyres (jsquyres)
Sent: Tuesday, August 02, 2011 10:33 AM
To: Open MPI Developers
Cc: Open MPI Developers
Subject: Re: [OMPI devel] RFC: CUDA register sm and openib host memory

Rolf -

Can you send a cumulative SVN diff against the SVN HEAD?

Sent from my phone. No type good.

On Jul 28, 2011, at 5:52 PM, "Rolf vandeVaart" 
mailto:rvandeva...@nvidia.com>> wrote:
WHAT: Add CUDA registration of host memory in sm and openib BTLs.

TIMEOUT: 8/4/2011

DETAILS: In order to improve performance of sending GPU device memory,
we need to register the host memory with the CUDA framework.  These
changes allow that to happen.  These changes are somewhat different
from what I proposed a while ago and I think a lot cleaner.  There is
a new memory pool flag that indicates whether a piece of memory
should be registered.  This allows us to register the sm memory and
the pre-posted openib memory.

The CUDA specific code is in the ompi/mca/common/cuda directory.

Do not look at the configure.m4 code, as that is still not done.

Here a link to the proposed changes:
https://bitbucket.org/rolfv/ompi-cuda-register

Here is a list of files that would change.
M   VERSION
M   configure.ac
M   ompi/mca/btl/openib/btl_openib_component.c
M   ompi/mca/btl/openib/Makefile.am
M   ompi/mca/mpool/sm/Makefile.am
M   ompi/mca/mpool/sm/mpool_sm_module.c
M   ompi/mca/mpool/mpool.h
M   ompi/mca/pml/ob1/pml_ob1_sendreq.h
A   ompi/mca/common/cuda
A   ompi/mca/common/cuda/configure.m4
A   ompi/mca/common/cuda/common_cuda.c
A   ompi/mca/common/cuda/help-mpi-common-cuda.txt
A   ompi/mca/common/cuda/Makefile.am
A   ompi/mca/common/cuda/common_cuda.h
M   ompi/class/ompi_free_list.c




This email message is for the sole use of the intended recipient(s) and may 
contain confidential information.  Any unauthorized review, use, disclosure or 
distribution is prohibited.  If you are not the intended recipient, please 
contact the sender by reply email and destroy all copies of the original 
message.

___
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel


cuda-patch
Description: cuda-patch

Re: [OMPI devel] [OMPI svn-full] svn:open-mpi r24977

[Jeff Squyres (jsquyres)]

Te question that needs to be answered in the readme is: when should one se 
openib/ob1 vs. Mxm?  Users will need to know thus. 

Also see the part in the readme about te different PMLs - u might want to write 
more there. 

Sent from my phone. No type good. 

On Aug 2, 2011, at 10:30 AM, "mi...@osl.iu.edu"  wrote:

> Author: miked
> Date: 2011-08-02 10:30:11 EDT (Tue, 02 Aug 2011)
> New Revision: 24977
> URL: https://svn.open-mpi.org/trac/ompi/changeset/24977
> 
> Log:
> code and readme  updates, some refactoring
> Text files modified: 
>   trunk/NEWS   | 1
>  
>   trunk/README | 5 ++ 
>  
>   trunk/ompi/mca/mtl/mxm/mtl_mxm_cancel.c  | 4 +- 
>  
>   trunk/ompi/mca/mtl/mxm/mtl_mxm_probe.c   |16    
>  
>   trunk/ompi/mca/mtl/mxm/mtl_mxm_recv.c|54 
> ++--
>   trunk/ompi/mca/mtl/mxm/mtl_mxm_request.h | 2
>  
>   trunk/ompi/mca/mtl/mxm/mtl_mxm_send.c|77 
> --- 
>   7 files changed, 74 insertions(+), 85 deletions(-)
> 
> Modified: trunk/NEWS
> ==
> --- trunk/NEWS(original)
> +++ trunk/NEWS2011-08-02 10:30:11 EDT (Tue, 02 Aug 2011)
> @@ -62,6 +62,7 @@
>   OPAL levels - intended for use when configuring without MPI support
> - Modified paffinity system to provide warning when bindings result in
>   being "bound to all", which is equivalent to "not bound"
> +- Added Mellanox MTL layer implementation (mxm)
> 
> 
> 1.5.3
> 
> Modified: trunk/README
> ==
> --- trunk/README(original)
> +++ trunk/README2011-08-02 10:30:11 EDT (Tue, 02 Aug 2011)
> @@ -509,6 +509,9 @@
> or
> shell$ mpirun --mca pml cm ...
> 
> +- MXM MTL is an transport layer utilizing various Mellanox proprietary
> +  technologies and providing better scalability and performance for large 
> scale jobs
> +
> - Myrinet MX (and Open-MX) support is shared between the 2 internal
>   devices, the MTL and the BTL.  The design of the BTL interface in
>   Open MPI assumes that only naive one-sided communication
> @@ -707,7 +710,7 @@
> --with-mxm=
>   Specify the directory where the Mellanox MXM library and
>   header files are located.  This option is generally only necessary
> -  if the InfiniPath headers and libraries are not in default
> +  if the MXM headers and libraries are not in default
>   compiler/linker search paths.
> 
>   MXM is the support library for Mellanox network adapters.
> 
> Modified: trunk/ompi/mca/mtl/mxm/mtl_mxm_cancel.c
> ==
> --- trunk/ompi/mca/mtl/mxm/mtl_mxm_cancel.c(original)
> +++ trunk/ompi/mca/mtl/mxm/mtl_mxm_cancel.c2011-08-02 10:30:11 EDT (Tue, 
> 02 Aug 2011)
> @@ -18,9 +18,9 @@
> mxm_error_t err;
> mca_mtl_mxm_request_t *mtl_mxm_request = (mca_mtl_mxm_request_t*) 
> mtl_request;
> 
> -err = mxm_req_cancel(&mtl_mxm_request->mxm_request);
> +err = mxm_req_cancel(mtl_mxm_request->mxm_base_request);
> if (MXM_OK == err) {
> -err = mxm_req_test(&mtl_mxm_request->mxm_request);
> +err = mxm_req_test(mtl_mxm_request->mxm_base_request);
> if (MXM_OK == err) {
> mtl_request->ompi_req->req_status._cancelled = true;
> 
> mtl_mxm_request->super.completion_callback(&mtl_mxm_request->super);
> 
> Modified: trunk/ompi/mca/mtl/mxm/mtl_mxm_probe.c
> ==
> --- trunk/ompi/mca/mtl/mxm/mtl_mxm_probe.c(original)
> +++ trunk/ompi/mca/mtl/mxm/mtl_mxm_probe.c2011-08-02 10:30:11 EDT (Tue, 
> 02 Aug 2011)
> @@ -18,21 +18,21 @@
> int *flag, struct ompi_status_public_t *status)
> {
> mxm_error_t err;
> -mxm_req_t req;
> +mxm_recv_req_t req;
> 
> -req.state  = MXM_REQ_NEW;
> -req.mq = (mxm_mq_h)comm->c_pml_comm;
> -req.tag= tag;
> -req.tag_mask   = (tag == MPI_ANY_TAG) ? 0 : 0xU;
> -req.conn   = (src == MPI_ANY_SOURCE) ? NULL : 
> ompi_mtl_mxm_conn_lookup(comm, src);
> +req.base.state  = MXM_REQ_NEW;
> +req.base.mq = (mxm_mq_h)comm->c_pml_comm;
> +req.tag= tag;
> +req.tag_mask   = (tag == MPI_ANY_TAG) ? 0 : 0xU;
> +req.base.conn   = (src == MPI_ANY_SOURCE) ? NULL : 
> ompi_mtl_mxm_conn_lookup(comm, src);
> 
> err = mxm_req_probe(&req);
> if (MXM_OK == err) {
> *flag = 1;
> if (MPI_STATUS_IGNORE != status) {
> -status->MPI_SOURCE = *(int *)mxm_conn_get_context(req.conn);
> +status->MPI_SOURCE = *(int *)mxm_c

Re: [OMPI devel] RFC: CUDA register sm and openib host memory

[George Bosilca]

Rolf,

Can we have some more details on how this will improve performance of sending 
GPU device memory? I fail to see how registering the backend shared memory file 
with CUDA is supposed to do anything at all, as this memory is internal to Open 
MPI and not supposed to be visible at any other level.

  Thanks,
george.

On Jul 28, 2011, at 23:52 , Rolf vandeVaart wrote:

> DETAILS: In order to improve performance of sending GPU device memory,
> we need to register the host memory with the CUDA framework.  These
> changes allow that to happen.  These changes are somewhat different
> from what I proposed a while ago and I think a lot cleaner.  There is
> a new memory pool flag that indicates whether a piece of memory
> should be registered.  This allows us to register the sm memory and
> the pre-posted openib memory.

Re: [OMPI devel] RFC: CUDA register sm and openib host memory

[Rolf vandeVaart]

Hi George:

In the current implementation, to send CUDA device memory, we move it through 
internal host buffers.  In other words, we force the usage of the send 
protocols.
If the host buffers are CUDA registered, then when we call cuMemcpy to move the 
data into those host buffers, the cuMemcpy performs better if the memory is 
registered, and cannot be paged out.
This also may allow future optimizations, for example, asynchronous copies, 
that require the host memory we are copying into to be CUDA registered.

Rolf

>-Original Message-
>From: devel-boun...@open-mpi.org [mailto:devel-boun...@open-mpi.org]
>On Behalf Of George Bosilca
>Sent: Tuesday, August 02, 2011 10:50 AM
>To: Open MPI Developers
>Subject: Re: [OMPI devel] RFC: CUDA register sm and openib host memory
>
>Rolf,
>
>Can we have some more details on how this will improve performance of
>sending GPU device memory? I fail to see how registering the backend shared
>memory file with CUDA is supposed to do anything at all, as this memory is
>internal to Open MPI and not supposed to be visible at any other level.
>
>  Thanks,
>george.
>
>On Jul 28, 2011, at 23:52 , Rolf vandeVaart wrote:
>
>> DETAILS: In order to improve performance of sending GPU device memory,
>> we need to register the host memory with the CUDA framework.  These
>> changes allow that to happen.  These changes are somewhat different
>> from what I proposed a while ago and I think a lot cleaner.  There is
>> a new memory pool flag that indicates whether a piece of memory should
>> be registered.  This allows us to register the sm memory and the
>> pre-posted openib memory.
>
>___
>devel mailing list
>de...@open-mpi.org
>http://www.open-mpi.org/mailman/listinfo.cgi/devel
---
This email message is for the sole use of the intended recipient(s) and may 
contain
confidential information.  Any unauthorized review, use, disclosure or 
distribution
is prohibited.  If you are not the intended recipient, please contact the 
sender by
reply email and destroy all copies of the original message.
---

[OMPI devel] [patch] add explicit IT instructions in ARM assembly

[Leif Lindholm]

In the Linaro 11.05 Linux metadistribution, and hence most likely in 
Ubuntu 11.04, the GNU assembler no longer defaults to 
-mimplicit-it=thumb, causing the build to fail when compiling on this 
platform.


The attached trivial patch adds explicit IT instructions where required, 
permitting a successful build. Patch created against r24936, but 
validated also against subsequent ones up to and including r24961.


/
Leif

p.s.
In case someone is curious about what I'm talking about above, there is 
a pretty decent description of the IT instruction here:

http://infocenter.arm.com/help/topic/com.arm.doc.dui0489d/Cjabicci.htmlIndex: opal/asm/base/ARM.asm
===
--- opal/asm/base/ARM.asm	(revision 24936)
+++ opal/asm/base/ARM.asm	(working copy)
@@ -73,6 +73,7 @@
LSYM(7)
ldrexd  r4, r5, [r0]
cmp r4, r2
+   it  eq
cmpeq   r5, r3
bne REFLSYM(8)
strexd  r1, r6, r7, [r0]
@@ -91,6 +92,7 @@
LSYM(9)
ldrexd  r4, r5, [r0]
cmp r4, r2
+   it  eq
cmpeq   r5, r3
bne REFLSYM(10)
strexd  r1, r6, r7, [r0]
@@ -111,6 +113,7 @@
LSYM(11)
ldrexd  r4, r5, [r0]
cmp r4, r2
+   it  eq
cmpeq   r5, r3
bne REFLSYM(12)
dmb
Index: opal/include/opal/sys/arm/atomic.h
===
--- opal/include/opal/sys/arm/atomic.h	(revision 24936)
+++ opal/include/opal/sys/arm/atomic.h	(working copy)
@@ -142,6 +142,7 @@
__asm__ __volatile__ (
  "1:  ldrexd  %0, %H0, [%2]   \n"
  "cmp %0, %3  \n"
+ "it  eq  \n"
  "cmpeq   %H0, %H3\n"
  "bne 2f  \n"
  "strexd  %1, %4, %H4, [%2]   \n"

Re: [OMPI devel] [patch] add explicit IT instructions in ARM assembly

[Jeff Squyres]

As far as I'm concerned, the ARM ASM is your stuff, so I have no issue with 
just committing this directly for you.

https://svn.open-mpi.org/trac/ompi/changeset/24979
https://svn.open-mpi.org/trac/ompi/ticket/2843


On Aug 2, 2011, at 12:51 PM, Leif Lindholm wrote:

> In the Linaro 11.05 Linux metadistribution, and hence most likely in Ubuntu 
> 11.04, the GNU assembler no longer defaults to -mimplicit-it=thumb, causing 
> the build to fail when compiling on this platform.
> 
> The attached trivial patch adds explicit IT instructions where required, 
> permitting a successful build. Patch created against r24936, but validated 
> also against subsequent ones up to and including r24961.
> 
> /
>   Leif
> 
> p.s.
> In case someone is curious about what I'm talking about above, there is a 
> pretty decent description of the IT instruction here:
> http://infocenter.arm.com/help/topic/com.arm.doc.dui0489d/Cjabicci.html___
> devel mailing list
> de...@open-mpi.org
> http://www.open-mpi.org/mailman/listinfo.cgi/devel


-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/

[OMPI devel] MXM question

[Jeff Squyres]

Is MXM for InfiniBand only?

E.g., what happens if I have the MXM MTL component available on a machine with 
iWARP or RoCE devices?

-- 
Jeff Squyres
jsquy...@cisco.com
For corporate legal information go to:
http://www.cisco.com/web/about/doing_business/legal/cri/