I applied the fix committed on master and described in
https://github.com/open-mpi/ompi/issues/268
on 1.8.3 and 1.8.4rc1 and this seems to have fixed my problems. I can
include my PSM based mtt results in the main mtt database if desired.
Adrian
On Tue, Nov 11, 2014 at 07:42:24PM +0100, Adrian Reber wrote:
> Using the intel test suite I can reproduce it for example with:
>
> $ mpirun --np 2 --map-by ppr:1:node `pwd`/src/MPI_Allgatherv_c
> MPITEST info (0): Starting MPI_Allgatherv() test
> MPITEST info (0): Node spec MPITEST_comm_sizes[6]=2 too large, using 1
> MPITEST info (0): Node spec MPITEST_comm_sizes[22]=2 too large, using 1
> MPITEST info (0): Node spec MPITEST_comm_sizes[32]=2 too large, using 1
>
> MPI_Allgatherv_c:9230 terminated with signal 11 at PC=7fc4ced4b150
> SP=7fff45aa2fb0. Backtrace:
> /lib64/libpsm_infinipath.so.1(ips_proto_connect+0x630)[0x7fc4ced4b150]
> /lib64/libpsm_infinipath.so.1(ips_ptl_connect+0x3a)[0x7fc4ced4219a]
> /lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x3e7)[0x7fc4ced3a727]
> /opt/bwhpc/common/mpi/openmpi/1.8.4-gnu-4.8/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1f3)[0x7fc4cf902303]
> /opt/bwhpc/common/mpi/openmpi/1.8.4-gnu-4.8/lib/libmpi.so.1(ompi_comm_get_rprocs+0x49a)[0x7fc4cf7cbc2a]
> /opt/bwhpc/common/mpi/openmpi/1.8.4-gnu-4.8/lib/libmpi.so.1(PMPI_Intercomm_create+0x2f2)[0x7fc4cf7fb602]
> /lustre/lxfs/work/ws/es_test01-open_mpi-0/ompi-tests/intel_tests/src/MPI_Allgatherv_c[0x40f5bf]
> /lustre/lxfs/work/ws/es_test01-open_mpi-0/ompi-tests/intel_tests/src/MPI_Allgatherv_c[0x40edf4]
> /lustre/lxfs/work/ws/es_test01-open_mpi-0/ompi-tests/intel_tests/src/MPI_Allgatherv_c[0x401c80]
> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7fc4cf1a8af5]
> /lustre/lxfs/work/ws/es_test01-open_mpi-0/ompi-tests/intel_tests/src/MPI_Allgatherv_c[0x401a89]
> -------------------------------------------------------
> Primary job terminated normally, but 1 process returned
> a non-zero exit code.. Per user-direction, the job has been aborted.
> -------------------------------------------------------
>
>
> On Tue, Nov 11, 2014 at 10:26:52AM -0800, Ralph Castain wrote:
> > I think it would help understand this if you isolated it down to a single
> > test that is failing, rather than just citing an entire test suite. For
> > example, we know that the many-to-one test is never going to pass,
> > regardless of transport. We also know that the dynamic tests will fail with
> > PSM as they are not supported by that transport.
> >
> > So could you find one test that doesn’t pass, and give us some info on that
> > one?
> >
> >
> > > On Nov 11, 2014, at 10:04 AM, Adrian Reber <adr...@lisas.de> wrote:
> > >
> > > Some more information about our PSM troubles.
> > >
> > > Using 1.6.5 the test suite still works. It fails with 1.8.3 and
> > > 1.8.4rc1. As long as all processes are running on one node it also
> > > works. As soon as one process is running on a second node it fails with
> > > the previously described errors. I also tried the 1.8 release and it has
> > > the same error. Another way to trigger it with only two processes is:
> > >
> > > mpirun --np 2 --map-by ppr:1:node mpi_test_suite -t "environment"
> > >
> > > Some change introduced between 1.6.5 and 1.8 broke this test case with
> > > PSM. I have not yet been able to upgrade PSM to 3.3 but it seems more
> > > Open MPI related than PSM.
> > >
> > > Intel MPI (4.1.1) has also no troubles running the test cases.
> > >
> > > Adrian
> > >
> > > On Mon, Nov 10, 2014 at 09:12:41PM +0000, Friedley, Andrew wrote:
> > >> Hi Adrian,
> > >>
> > >> Yes, I suggest trying either RH support or Intel's support at
> > >> ibsupp...@intel.com. They might have seen this problem before. Since
> > >> you're running the RHEL versions of PSM and related software, one thing
> > >> you could try is IFS. I think I was running IFS 7.3.0, so that's a
> > >> difference between your setup and mine. At the least, it may help
> > >> support nail down the issue.
> > >>
> > >> Andrew
> > >>
> > >>> -----Original Message-----
> > >>> From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Adrian
> > >>> Reber
> > >>> Sent: Monday, November 10, 2014 12:39 PM
> > >>> To: Open MPI Developers
> > >>> Subject: Re: [OMPI devel] 1.8.3 and PSM errors
> > >>>
> > >>> Andrew,
> > >>>
> > >>> thanks for looking into this. I was able to reproduce this error on
> > >>> RHEL 7 with
> > >>> PSM provided by RHEL:
> > >>>
> > >>> infinipath-psm-3.2-2_ga8c3e3e_open.2.el7.x86_64
> > >>> infinipath-psm-devel-3.2-2_ga8c3e3e_open.2.el7.x86_64
> > >>>
> > >>> $ mpirun -np 32 mpi_test_suite -t "environment"
> > >>>
> > >>> mpi_test_suite:4877 terminated with signal 11 at PC=7f5a2f4a2150
> > >>> SP=7fff9e0ce770. Backtrace:
> > >>> /lib64/libpsm_infinipath.so.1(ips_proto_connect+0x630)[0x7f5a2f4a2150]
> > >>> /lib64/libpsm_infinipath.so.1(ips_ptl_connect+0x3a)[0x7f5a2f49919a]
> > >>> /lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x3e7)[0x7f5a2f491727]
> > >>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>> 4.8/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1f3)[0x7f5a30054cf3]
> > >>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>> 4.8/lib/libmpi.so.1(ompi_comm_get_rprocs+0x49a)[0x7f5a2ff221da]
> > >>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>> 4.8/lib/libmpi.so.1(PMPI_Intercomm_create+0x2f2)[0x7f5a2ff51832]
> > >>> mpi_test_suite[0x469420]
> > >>> mpi_test_suite[0x441d8e]
> > >>> /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f5a2f8ffaf5]
> > >>> mpi_test_suite[0x405349]
> > >>>
> > >>> Source RPM : infinipath-psm-3.2-2_ga8c3e3e_open.2.el7.src.rpm
> > >>> Build Date : Tue 04 Mar 2014 02:45:41 AM CET Build Host : x86-
> > >>> 025.build.eng.bos.redhat.com Relocations : /usr
> > >>> Packager : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
> > >>> Vendor : Red Hat, Inc.
> > >>> URL :
> > >>> http://www.openfabrics.org/downloads/infinipath-psm/infinipath-
> > >>> psm-3.2-2_ga8c3e3e_open.tar.gz
> > >>> Summary : QLogic PSM Libraries
> > >>>
> > >>> Is this supposed to work? Or is this something Red Hat has to fix?
> > >>>
> > >>> Adrian
> > >>>
> > >>> On Mon, Oct 27, 2014 at 10:22:08PM +0000, Friedley, Andrew wrote:
> > >>>> Hi Adrian,
> > >>>>
> > >>>> I'm unable to reproduce here with OMPI v1.8.3 (I assume you're doing
> > >>>> this
> > >>> with one 8-core node):
> > >>>>
> > >>>> $ mpirun -np 32 -mca pml cm -mca mtl psm ./mpi_test_suite -t
> > >>> "environment"
> > >>>> (Rank:0) tst_test_array[0]:Status
> > >>>> (Rank:0) tst_test_array[1]:Request_Null
> > >>>> (Rank:0) tst_test_array[2]:Type_dup
> > >>>> (Rank:0) tst_test_array[3]:Get_version Number of failed tests:0
> > >>>>
> > >>>> Works with various np from 8 to 32. Your original case:
> > >>>>
> > >>>> $ mpirun -np 32 ./mpi_test_suite -t "All,^io,^one-sided"
> > >>>>
> > >>>> Runs for a while and eventually hits send cancellation errors.
> > >>>>
> > >>>> Any chance you could try updating your infinipath libraries?
> > >>>>
> > >>>> Andrew
> > >>>>
> > >>>>> -----Original Message-----
> > >>>>> From: devel [mailto:devel-boun...@open-mpi.org] On Behalf Of Adrian
> > >>>>> Reber
> > >>>>> Sent: Monday, October 27, 2014 9:11 AM
> > >>>>> To: Open MPI Developers
> > >>>>> Subject: Re: [OMPI devel] 1.8.3 and PSM errors
> > >>>>>
> > >>>>> This is a simpler test setup:
> > >>>>>
> > >>>>> On 8 core machines this works:
> > >>>>>
> > >>>>> $ mpirun -np 8 mpi_test_suite -t "environment"
> > >>>>> [...]
> > >>>>> Number of failed tests:0
> > >>>>>
> > >>>>> Using 9 or more cores it fails:
> > >>>>>
> > >>>>> $ mpirun -np 9 mpi_test_suite -t "environment"
> > >>>>>
> > >>>>> mpi_test_suite:20293 terminated with signal 11 at PC=2b6d107fa9a4
> > >>>>> SP=7fff06431a70. Backtrace:
> > >>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b6d107
> > >>>>> fa9a
> > >>>>> 4]
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b6d107e
> > >>>>> b1
> > >>>>> 72]
> > >>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1a4)[0x2b6d0fa6e384]
> > >>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_comm_get_rprocs+0x2fa)[0x2b6d0f93376a]
> > >>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Intercomm_create+0x332)[0x2b6d0f963d42]
> > >>>>> mpi_test_suite[0x46cd00]
> > >>>>> mpi_test_suite[0x44434c]
> > >>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6d10047d5d]
> > >>>>> mpi_test_suite[0x4058e9]
> > >>>>> -------------------------------------------------------
> > >>>>> Primary job terminated normally, but 1 process returned a non-zero
> > >>>>> exit code.. Per user-direction, the job has been aborted.
> > >>>>> -------------------------------------------------------
> > >>>>>
> > >>>>> mpi_test_suite:11212 terminated with signal 11 at PC=2b2c27d0d9a4
> > >>>>> SP=7ffff5020430. Backtrace:
> > >>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b2c27d
> > >>>>> 0d9a
> > >>>>> 4]
> > >>>>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b2c27cf
> > >>>>> e17
> > >>>>> 2]
> > >>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl_psm_add_procs+0x1a4)[0x2b2c26f81384]
> > >>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_comm_get_rprocs+0x2fa)[0x2b2c26e4676a]
> > >>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Intercomm_create+0x332)[0x2b2c26e76d42]
> > >>>>> mpi_test_suite[0x46cd00]
> > >>>>> mpi_test_suite[0x44434c]
> > >>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b2c2755ad5d]
> > >>>>> mpi_test_suite[0x4058e9]
> > >>>>> --------------------------------------------------------------------
> > >>>>> ------ mpirun detected that one or more processes exited with
> > >>>>> non-zero status, thus causing the job to be terminated. The first
> > >>>>> process to do so was:
> > >>>>>
> > >>>>> Process name: [[47415,1],0]
> > >>>>> Exit code: 1
> > >>>>> --------------------------------------------------------------------
> > >>>>> ------
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Mon, Oct 27, 2014 at 08:27:17AM -0700, Ralph Castain wrote:
> > >>>>>> I’m afraid I can’t quite decipher from all this what actually
> > >>>>>> fails. Of course,
> > >>>>> PSM doesn’t support dynamic operations like comm_spawn or
> > >>>>> connect_accept, so if you are running those tests that just won’t
> > >>>>> work. Is that the heart of the problem here?
> > >>>>>>
> > >>>>>>
> > >>>>>>> On Oct 27, 2014, at 1:40 AM, Adrian Reber <adr...@lisas.de> wrote:
> > >>>>>>>
> > >>>>>>> Running Open MPI 1.8.3 with PSM does not seem to work right now
> > >>> at all.
> > >>>>>>> I am getting the same errors also on trunk from my newly set up MTT.
> > >>>>>>> Before trying to debug this I just wanted to make sure this is
> > >>>>>>> not a configuration error. I have following PSM packages installed:
> > >>>>>>>
> > >>>>>>> infinipath-devel-3.1.1-363.1140_rhel6_qlc.noarch
> > >>>>>>> infinipath-libs-3.1.1-363.1140_rhel6_qlc.x86_64
> > >>>>>>> infinipath-3.1.1-363.1140_rhel6_qlc.x86_64
> > >>>>>>>
> > >>>>>>> with 1.6.5 I do not see PSM errors and the test suite fails much
> > >>>>>>> later:
> > >>>>>>>
> > >>>>>>> P2P tests Many-to-one with MPI_Iprobe (MPI_ANY_SOURCE)
> > >>> (21/48),
> > >>>>> comm
> > >>>>>>> Intracomm merged of the Halved Intercomm (13/13), type
> > >>>>>>> MPI_TYPE_MIX_ARRAY (28/29) P2P tests Many-to-one with
> > >>> MPI_Iprobe
> > >>>>>>> (MPI_ANY_SOURCE) (21/48), comm Intracomm merged of the Halved
> > >>>>>>> Intercomm (13/13), type MPI_TYPE_MIX_LB_UB (29/29)
> > >>>>>>> n050304:5.0.Cannot cancel send requests (req=0x2ad8ba881f80) P2P
> > >>>>>>> tests Many-to-one with Isend and Cancellation (22/48), comm
> > >>>>>>> MPI_COMM_WORLD (1/13), type MPI_CHAR (1/29)
> > >>> n050304:2.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b25143fbd88) n050302:7.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b4d95eb0f80) n050301:4.0.Cannot
> > >>>>>>> cancel send requests (req=0x2adf03e14f80) n050304:4.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ad877257ed8) n050301:6.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ba47634af80) n050304:8.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ae8ac16cf80) n050302:3.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ab81dcb4d88) n050303:4.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b9ef4ef8f80) n050303:2.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ab0f03f9f80) n050302:9.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b214f9ebed8) n050301:2.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b31302d4f80) n050302:4.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b0581bd3f80) n050301:8.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ae53776bf80) n050303:6.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b13eeb78f80) n050304:7.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b4e99715f80) n050304:9.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b10429c2f80) n050304:3.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b9196f5fe30) n050304:6.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b30d6c69ed8) n050301:9.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b93c9e04f80) n050303:9.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ab4d6ce0f80) n050301:5.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b6ad851ef80) n050303:3.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b8ef52a0f80) n050301:3.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b277a4aff80) n050303:7.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ba570fa9f80) n050301:7.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ba707dfbf80) n050302:2.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b90f2e51e30) n050303:5.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b1250ba8f80) n050302:8.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b22e0129ed8) n050303:8.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b6609792f80) n050302:6.0.Cannot
> > >>>>>>> cancel send requests (req=0x2b2b6081af80) n050302:5.0.Cannot
> > >>>>>>> cancel send requests (req=0x2ab24f6f1f80)
> > >>>>>>> ----------------------------------------------------------------
> > >>>>>>> ----
> > >>>>>>> ------ mpirun has exited due to process rank 14 with PID 4496 on
> > >>>>>>> node n050303 exiting improperly. There are two reasons this could
> > >>> occur:
> > >>>>>>>
> > >>>>>>> 1. this process did not call "init" before exiting, but others
> > >>>>>>> in the job did. This can cause a job to hang indefinitely while
> > >>>>>>> it waits for all processes to call "init". By rule, if one
> > >>>>>>> process calls "init", then ALL processes must call "init" prior to
> > >>> termination.
> > >>>>>>>
> > >>>>>>> 2. this process called "init", but exited without calling
> > >>>>>>> "finalize".
> > >>>>>>> By rule, all processes that call "init" MUST call "finalize"
> > >>>>>>> prior to exiting or it will be considered an "abnormal termination"
> > >>>>>>>
> > >>>>>>> This may have caused other processes in the application to be
> > >>>>>>> terminated by signals sent by mpirun (as reported here).
> > >>>>>>> ----------------------------------------------------------------
> > >>>>>>> ----
> > >>>>>>> ------
> > >>>>>>> [adrian@n050304 mpi_test_suite]$
> > >>>>>>>
> > >>>>>>> and this are my PSM errors with 1.8.3:
> > >>>>>>>
> > >>>>>>> [adrian@n050304 mpi_test_suite]$ mpirun -np 32 mpi_test_suite
> > >>>>>>> -t
> > >>>>> "All,^io,^one-sided"
> > >>>>>>>
> > >>>>>>> mpi_test_suite:8904 terminated with signal 11 at PC=2b08466239a4
> > >>>>> SP=7ffff03c6e30. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16905 terminated with signal 11 at
> > >>>>>>> PC=2ae4cad209a4
> > >>>>> SP=7fffceefa730. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3171 terminated with signal 11 at PC=2b57daafe9a4
> > >>>>> SP=7fff5c4b3af0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16906 terminated with signal 11 at
> > >>>>>>> PC=2b4c9fa019a4
> > >>>>> SP=7fffe916c330. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3172 terminated with signal 11 at PC=2b6dde92e9a4
> > >>>>> SP=7fff04cf1730. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16907 terminated with signal 11 at
> > >>>>>>> PC=2ad6eb8589a4
> > >>>>> SP=7fffc30d02f0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3173 terminated with signal 11 at PC=2b2e4aec89a4
> > >>>>> SP=7fffa054e230. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16908 terminated with signal 11 at
> > >>>>>>> PC=2b4e6e5589a4
> > >>>>> SP=7fff68c7a1f0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3174 terminated with signal 11 at PC=2b7049b279a4
> > >>>>> SP=7fff99a49f70. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16909 terminated with signal 11 at
> > >>>>>>> PC=2b252219d9a4
> > >>>>> SP=7fff72a0c6b0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3175 terminated with signal 11 at PC=2ac8d5caf9a4
> > >>>>> SP=7fff6d7a63f0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16910 terminated with signal 11 at
> > >>>>>>> PC=2b7f83fc49a4
> > >>>>> SP=7fffb95b89b0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3176 terminated with signal 11 at PC=2b11438da9a4
> > >>>>> SP=7fffe626f270. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16903 terminated with signal 11 at
> > >>>>>>> PC=2ac5249249a4
> > >>>>> SP=7fff8874af30. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3177 terminated with signal 11 at PC=2ab6154549a4
> > >>>>> SP=7fffbf6ff430. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:16904 terminated with signal 11 at
> > >>>>>>> PC=2ad0265099a4
> > >>>>> SP=7fff89fea470. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:3178 terminated with signal 11 at PC=2b606b1a79a4
> > >>>>> SP=7fff20240db0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4458 terminated with signal 11 at PC=2b593ef029a4
> > >>>>> SP=7fff4f48b470. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4459 terminated with signal 11 at PC=2b06dde559a4
> > >>>>> SP=7fffd771a4f0. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4460 terminated with signal 11 at PC=2ba7904cb9a4
> > >>>>> SP=7fff9694c8b0. Backtrace:
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ab
> > >>>>>>> 6154
> > >>>>>>> 549a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ab61544
> > >>>>>>> 5172]
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4461 terminated with signal 11 at PC=2b26799fd9a4
> > >>>>> SP=7fff70f69eb0. Backtrace:
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b1
> > >>>>>>> 1438
> > >>>>>>> da9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b11438c
> > >>>>>>> b172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b1142b4e384]
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4462 terminated with signal 11 at PC=2b15418e19a4
> > >>>>> SP=7fff858425b0. Backtrace:
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ab6146c8384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ab61458d76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ab6145bdd42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ab614ca1d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4463 terminated with signal 11 at PC=2b43082919a4
> > >>>>> SP=7fff2ea8a530. Backtrace:
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4464 terminated with signal 11 at PC=2adc01fe89a4
> > >>>>> SP=7fff0de9d4b0. Backtrace:
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b1142a1376a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b1142a43d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>>
> > >>>>>>> mpi_test_suite:4465 terminated with signal 11 at PC=2b477a1819a4
> > >>>>> SP=7fffd33831b0. Backtrace:
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b1143127d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b4
> > >>>>>>> 3082
> > >>>>>>> 919a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b430828
> > >>>>>>> 2172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b4
> > >>>>>>> 77a1
> > >>>>>>> 819a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b477a17
> > >>>>>>> 2172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b47793f5384]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b2
> > >>>>>>> 6799
> > >>>>>>> fd9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b26799e
> > >>>>>>> e172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b2678c71384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b47792ba76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b47792ead42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b4307505384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b43073ca76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b43073fad42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b2678b3676a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b2678b66d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4307aded5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b47799ced5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b267924ad5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b2
> > >>>>>>> e4ae
> > >>>>>>> c89a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b2e4aeb
> > >>>>>>> 9172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ac
> > >>>>>>> 8d5c
> > >>>>>>> af9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ac8d5ca
> > >>>>>>> 0172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b5
> > >>>>>>> 7daa
> > >>>>>>> fe9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b57daae
> > >>>>>>> f172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b57d9d72384]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b0
> > >>>>>>> 8466
> > >>>>>>> 239a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b084661
> > >>>>>>> 4172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b0845897384]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b5
> > >>>>>>> 93ef
> > >>>>>>> 029a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b593eef
> > >>>>>>> 3172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b7
> > >>>>>>> 049b
> > >>>>>>> 279a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b7049b1
> > >>>>>>> 8172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b4
> > >>>>>>> e6e5
> > >>>>>>> 589a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b4e6e54
> > >>>>>>> 9172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b084575c76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b084578cd42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b0845e70d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b1
> > >>>>>>> 5418
> > >>>>>>> e19a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b15418d
> > >>>>>>> 2172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b2e4a13c384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b2e4a00176a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b2e4a031d42]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b4e6d7cc384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b593e176384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b593e03b76a]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b6
> > >>>>>>> 06b1
> > >>>>>>> a79a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b606b19
> > >>>>>>> 8172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b606a41b384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b4e6d69176a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b1540b55384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b1540a1a76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b57d9c3776a]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b0
> > >>>>>>> 6dde
> > >>>>>>> 559a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b06dde4
> > >>>>>>> 6172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b06dd0c9384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b06dcf8e76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b06dcfbed42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ac8d4f23384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ac8d4de876a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b4e6d6c1d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4e6dda5d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b1540a4ad42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b154112ed5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b6
> > >>>>>>> dde9
> > >>>>>>> 2e9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b6dde91
> > >>>>>>> f172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ad
> > >>>>>>> 6eb8
> > >>>>>>> 589a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ad6eb84
> > >>>>>>> 9172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b593e06bd42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b593e74fd5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b606a2e076a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b606a310d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b606a9f4d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b06dd6a2d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b7048d9b384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b7048c6076a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b7048c90d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ac
> > >>>>>>> 5249
> > >>>>>>> 249a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ac52491
> > >>>>>>> 5172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ac523b98384]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ad
> > >>>>>>> c01f
> > >>>>>>> e89a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2adc01fd
> > >>>>>>> 9172]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b2e4a715d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ad6eaacc384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ad6ea99176a]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ba
> > >>>>>>> 7904
> > >>>>>>> cb9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ba7904b
> > >>>>>>> c172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b6dddba2384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b6ddda6776a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b6ddda97d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ac523a5d76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ac523a8dd42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2adc0125c384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2adc0112176a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2adc01151d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b57d9c67d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b57da34bd5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac524171d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ba78f73f384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ba78f60476a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ac8d4e18d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ac8d54fcd5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ad6ea9c1d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ad6eb0a5d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2adc01835d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b6dde17bd5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ba78f634d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ba78fd18d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7049374d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b2
> > >>>>>>> 5221
> > >>>>>>> 9d9a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b252218
> > >>>>>>> e172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ae
> > >>>>>>> 4cad
> > >>>>>>> 209a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ae4cad1
> > >>>>>>> 1172]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2ad
> > >>>>>>> 0265
> > >>>>>>> 099a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2ad0264f
> > >>>>>>> a172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ae4c9f94384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ae4c9e5976a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b2521411384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b25212d676a]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b7
> > >>>>>>> f83f
> > >>>>>>> c49a4]
> > >>>>>>>
> > >>>>>
> > >>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b7f83fb
> > >>>>>>> 5172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ae4c9e89d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ae4ca56dd5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(ips_proto_connect+0x334)[0x2b4
> > >>>>>>> c9fa
> > >>>>>>> 019a4]
> > >>>>>>> /usr/lib64/libpsm_infinipath.so.1(__psm_ep_connect+0x692)[0x2b4c
> > >>>>>>> 9f9f
> > >>>>>>> 2172]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b4c9ec75384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b4c9eb3a76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2ad02577d384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2ad02564276a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2ad025672d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2ad025d56d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b2521306d42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b25219ead5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_mtl
> > >>>>>>> _psm_add_procs+0x1a4)[0x2b7f83238384]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(ompi_com
> > >>>>>>> m_get_rprocs+0x2fa)[0x2b7f830fd76a]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b7f8312dd42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /opt/bwhpc/common/mpi/openmpi/1.8.3-gnu-
> > >>>>> 4.9/lib/libmpi.so.1(MPI_Inte
> > >>>>>>> rcomm_create+0x332)[0x2b4c9eb6ad42]
> > >>>>>>> mpi_test_suite[0x46cd00]
> > >>>>>>> mpi_test_suite[0x44434c]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b4c9f24ed5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b7f83811d5d]
> > >>>>>>> mpi_test_suite[0x4058e9]
> > >>>>>>> -------------------------------------------------------
> > >>>>>>> Primary job terminated normally, but 1 process returned a
> > >>>>>>> non-zero exit code.. Per user-direction, the job has been aborted.
> > >>>>>>> -------------------------------------------------------
> > >>>>>>> ----------------------------------------------------------------
> > >>>>>>> ----
> > >>>>>>> ------ mpirun detected that one or more processes exited with
> > >>>>>>> non-zero status, thus causing the job to be terminated. The
> > >>>>>>> first process to do so was:
> > >>>>>>>
> > >>>>>>> Process name: [[9290,1],0]
> > >>>>>>> Exit code: 1
> > >>>>>>> ----------------------------------------------------------------
> > >>>>>>> ----
> > >>>>>>> ------ _______________________________________________
> > >>>>>>> devel mailing list
> > >>>>>>> de...@open-mpi.org
> > >>>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >>>>>>> Link to this post:
> > >>>>>>> http://www.open-
> > >>> mpi.org/community/lists/devel/2014/10/16093.php
> > >>>>>>
> > >>>>>> _______________________________________________
> > >>>>>> devel mailing list
> > >>>>>> de...@open-mpi.org
> > >>>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >>>>>> Link to this post:
> > >>>>>> http://www.open-mpi.org/community/lists/devel/2014/10/16099.php
> > >>>>> _______________________________________________
> > >>>>> devel mailing list
> > >>>>> de...@open-mpi.org
> > >>>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >>>>> Link to this post: http://www.open-
> > >>>>> mpi.org/community/lists/devel/2014/10/16100.php
> > >>>> _______________________________________________
> > >>>> devel mailing list
> > >>>> de...@open-mpi.org
> > >>>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >>>> Link to this post:
> > >>>> http://www.open-mpi.org/community/lists/devel/2014/10/16103.php
> > >>>
> > >>> devel mailing list
> > >>> de...@open-mpi.org
> > >>> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >>> Link to this post: http://www.open-
> > >>> mpi.org/community/lists/devel/2014/11/16277.php
> > >> _______________________________________________
> > >> devel mailing list
> > >> de...@open-mpi.org
> > >> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > >> Link to this post:
> > >> http://www.open-mpi.org/community/lists/devel/2014/11/16278.php
> > > _______________________________________________
> > > devel mailing list
> > > de...@open-mpi.org <mailto:de...@open-mpi.org>
> > > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > > <http://www.open-mpi.org/mailman/listinfo.cgi/devel>
> > > Link to this post:
> > > http://www.open-mpi.org/community/lists/devel/2014/11/16289.php
> > > <http://www.open-mpi.org/community/lists/devel/2014/11/16289.php>
>
> > _______________________________________________
> > devel mailing list
> > de...@open-mpi.org
> > Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> > Link to this post:
> > http://www.open-mpi.org/community/lists/devel/2014/11/16290.php
> _______________________________________________
> devel mailing list
> de...@open-mpi.org
> Subscription: http://www.open-mpi.org/mailman/listinfo.cgi/devel
> Link to this post:
> http://www.open-mpi.org/community/lists/devel/2014/11/16291.php
Adrian
--
Adrian Reber <adr...@lisas.de> http://lisas.de/~adrian/
Typo in the code
