We have been getting hangs and failures of openmpi-1.3.X
on an 8-core FC11 box. Details:
Machine:
Linux octet.carys.home 2.6.30.5-43.fc11.x86_64 #1 SMP Thu Aug 27
21:39:52 EDT 2009 x86_64 x86_64 x86_64 GNU/Linux
with 2 quad-core cpus.
Compiler:
g++ (GCC) 4.4.1 20090725 (Red Hat 4.4.1-2)
The code is later in this email.
OpenMPI configured with
../configure --prefix=/usr/local/openmpi-1.3.3
--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3/lib
used to compile and execute:
/usr/local/openmpi-1.3.3/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.3.3/bin/mpirun -np 8 ompi1.3.3-bug
hangs on various steps: 10, 39, 125. Top shows the code
using all resources:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND 5716 cary 20 0 98.6m 4396 2940 R 100.3 0.0
0:33.58 ompi1.3.3-bug 5707 cary 20 0 98.7m 4592 3060 R 99.9
0.0 0:33.57 ompi1.3.3-bug 5710 cary 20 0 98.6m 4388 2932
R 99.9 0.0 0:33.61 ompi1.3.3-bug 5713 cary 20 0 98.6m
4392 2936 R 99.9 0.0 0:33.61 ompi1.3.3-bug 5719 cary 20
0 98.6m 4396 2940 R 99.9 0.0 0:33.60 ompi1.3.3-bug 5722
cary 20 0 98.6m 4384 2928 R 99.9 0.0 0:33.61
ompi1.3.3-bug 5728 cary 20 0 98.6m 4408 2952 R 99.9 0.0
0:33.53 ompi1.3.3-bug 5725 cary 20 0 98.6m 4404 2948 R
99.6 0.0 0:33.57 ompi1.3.3-bug
OpenMPI configured with
../configure --prefix=/usr/local/openmpi-1.2.9
--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.2.9/lib
used to compile and execute:
/usr/local/openmpi-1.2.9/bin/mpicxx ompi1.3.3-bug.cxx -o ompi1.3.3-bug
/usr/local/openmpi-1.2.9/bin/mpirun -np 8 ompi1.3.3-bug
works just fine.
So then I started looking for unitialized variables. I rebuilt
OpenMPI, configured with
../configure --prefix=/usr/local/openmpi-1.3.3-valg
--with-valgrind=/contrib/valgrind
--with-wrapper-ldflags=-Wl,-rpath,/usr/local/openmpi-1.3.3-valg/lib
and used to to compile and execute:
/usr/local/openmpi-1.3.3-valg/bin/mpicxx ompi1.3.3-bug.cxx -o
ompi1.3.3-bug
/usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind ompi1.3.3-bug
and one can get valgrind to segfault, as shown at the very end of this
email. Other times it hangs on step 1 or another step.
John Cary
THE CODE
$ cat ompi1.3.3-bug.cxx
/**
* A simple test program to demonstrate a problem in OpenMPI 1.3
*/
// mpi includes
#include <mpi.h>
// std includes
#include <iostream>
#include <vector>
// useful hashdefine
#define ARRAY_SIZE 250
/**
* Main driver
*/
int main(int argc, char** argv) {
// Initialize MPI
MPI_Init(&argc, &argv);
int rk, sz;
MPI_Comm_rank(MPI_COMM_WORLD, &rk);
MPI_Comm_size(MPI_COMM_WORLD, &sz);
// Create some data to pass around
std::vector<double> d(ARRAY_SIZE);
// Initialize to some values if we aren't rank 0
if ( rk )
for ( unsigned i = 0; i < ARRAY_SIZE; ++i )
d[i] = 2*i + 1;
// Loop until this breaks
unsigned t = 0;
while ( 1 ) {
MPI_Status s;
if ( rk )
MPI_Send( &d[0], d.size(), MPI_DOUBLE, 0, 3, MPI_COMM_WORLD );
else
for ( int i = 1; i < sz; ++i )
MPI_Recv( &d[0], d.size(), MPI_DOUBLE, i, 3, MPI_COMM_WORLD, &s );
MPI_Barrier(MPI_COMM_WORLD);
std::cout << "Transmission " << ++t << " completed." << std::endl;
}
// Finalize MPI
MPI_Finalize();
}
THE VALGRIND RUN
octet.cary$ /usr/local/openmpi-1.3.3-valg/bin/mpirun -np 8 valgrind
ompi1.3.3-bug
==902== Memcheck, a memory error detector
==902== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==902== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==902== Command: ompi1.3.3-bug
==902==
==905== Memcheck, a memory error detector
==905== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==905== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==905== Command: ompi1.3.3-bug
==905==
==908== Memcheck, a memory error detector
==908== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==908== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==908== Command: ompi1.3.3-bug
==908==
==911== Memcheck, a memory error detector
==911== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==911== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==911== Command: ompi1.3.3-bug
==911==
==914== Memcheck, a memory error detector
==914== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==914== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==914== Command: ompi1.3.3-bug
==914==
==917== Memcheck, a memory error detector
==917== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==917== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==917== Command: ompi1.3.3-bug
==917==
==920== Memcheck, a memory error detector
==920== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==920== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==920== Command: ompi1.3.3-bug
==920==
==923== Memcheck, a memory error detector
==923== Copyright (C) 2002-2009, and GNU GPL'd, by Julian Seward et al.
==923== Using Valgrind-3.5.0 and LibVEX; rerun with -h for copyright info
==923== Command: ompi1.3.3-bug
==923==
==917== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==902== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==905== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==914== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==911== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==920== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==923== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==908== Warning: client syscall munmap tried to modify addresses
0xffffffffffffffff-0xffe
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== Conditional jump or move depends on uninitialised value(s)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== Conditional jump or move depends on uninitialised value(s)
==911== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== Conditional jump or move depends on uninitialised value(s)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==917== Conditional jump or move depends on uninitialised value(s)
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
==917==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x8C1F94D: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x8C1FA22: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x8C1FA93: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==917== Conditional jump or move depends on uninitialised value(s)
==917== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==917== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==917== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==917== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==917== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==917== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==917==
==908== Conditional jump or move depends on uninitialised value(s)
==908== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==908== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==908== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==908== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==908== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==908== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==908==
==923== Conditional jump or move depends on uninitialised value(s)
==923== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==923== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==923== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==923== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==923==
==920== Conditional jump or move depends on uninitialised value(s)
==920== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==920== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==920== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==920== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==920== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==920== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==920==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x8C1F0FC: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==905== Conditional jump or move depends on uninitialised value(s)
==905== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==905== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==905== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==905== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==905== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==905== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==905==
==902== Conditional jump or move depends on uninitialised value(s)
==902== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==902== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==902== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==902== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==902== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==902== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==902==
==911== Conditional jump or move depends on uninitialised value(s)
==911== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==911== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==911== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==911== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==911== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==911== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==911==
==914== Conditional jump or move depends on uninitialised value(s)
==914== at 0x7BDFD64: mca_mpool_sm_alloc (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_mpool_sm.so)
==914== by 0x4E3EBDF: ompi_free_list_grow (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E3EC5F: ompi_free_list_resize (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x8C1F630: mca_btl_sm_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==914== by 0x8816759: mca_bml_r2_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_bml_r2.so)
==914== by 0x8400274: mca_pml_ob1_add_procs (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_pml_ob1.so)
==914== by 0x4E5B048: ompi_mpi_init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x4E79F65: PMPI_Init (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==914== by 0x408F99: main (in /home/research/cary/ompi1.3.3-bug)
==914==
[octet:00923] *** Process received signal ***
[octet:00923] Signal: Segmentation fault (11)
[octet:00923] Signal code: Address not mapped (1)
[octet:00923] Failing at address: 0x40
[octet:00923] [ 0] /lib64/libpthread.so.0 [0x3825c0eee0]
[octet:00923] [ 1]
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so [0x8c21009]
[octet:00923] [ 2]
/usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0(opal_progress+0x5a)
[0x5324c9a]
[octet:00923] [ 3] /usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0
[0x4e59d45]
[octet:00923] [ 4]
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9ccad2a]
[octet:00923] [ 5]
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so [0x9cd349e]
[octet:00923] [ 6]
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0(MPI_Barrier+0x72)
[0x4e6fa32]
[octet:00923] [ 7] ompi1.3.3-bug(main+0x1ee) [0x409152]
[octet:00923] [ 8] /lib64/libc.so.6(__libc_start_main+0xfd)
[0x382501ea2d]
[octet:00923] [ 9] ompi1.3.3-bug [0x408ea9]
[octet:00923] *** End of error message ***
==923==
==923== Process terminating with default action of signal 11 (SIGSEGV)
==923== Access not within mapped region at address 0x40
==923== at 0x8C21009: mca_btl_sm_component_progress (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_btl_sm.so)
==923== by 0x5324C99: opal_progress (in
/usr/local/openmpi-1.3.3-valg/lib/libopen-pal.so.0.0.0)
==923== by 0x4E59D44: ompi_request_default_wait_all (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x9CCAD29: ompi_coll_tuned_sendrecv_actual (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
==923== by 0x9CD349D:
ompi_coll_tuned_barrier_intra_recursivedoubling (in
/usr/local/openmpi-1.3.3-valg/lib/openmpi/mca_coll_tuned.so)
==923== by 0x4E6FA31: PMPI_Barrier (in
/usr/local/openmpi-1.3.3-valg/lib/libmpi.so.0.0.0)
==923== by 0x409151: main (in /home/research/cary/ompi1.3.3-bug)
==923== If you believe this happened as a result of a stack
==923== overflow in your program's main thread (unlikely but
==923== possible), you can try to increase the size of the
==923== main thread stack using the --main-stacksize= flag.
==923== The main thread stack size used in this run was 10485760.
==923==
==923== HEAP SUMMARY:
==923== in use at exit: 2,062,803 bytes in 2,652 blocks
==923== total heap usage: 10,244 allocs, 7,592 frees, 15,824,029
bytes allocated
==923==
==923== LEAK SUMMARY:
==923== definitely lost: 1,700 bytes in 29 blocks
==923== indirectly lost: 2,400 bytes in 1 blocks
==923== possibly lost: 2,384 bytes in 22 blocks
==923== still reachable: 2,056,319 bytes in 2,600 blocks
==923== suppressed: 0 bytes in 0 blocks
==923== Rerun with --leak-check=full to see details of leaked memory
==923==
==923== For counts of detected and suppressed errors, rerun with: -v
==923== Use --track-origins=yes to see where uninitialised values come
from
==923== ERROR SUMMARY: 5 errors from 5 contexts (suppressed: 7 from 7)
--------------------------------------------------------------------------
mpirun noticed that process rank 7 with PID 923 on node
octet.carys.home exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------
