Temporary workaround: -mca btl ^vader On Feb 8, 2014, at 10:11 AM, Ralph Castain <r...@open-mpi.org> wrote:
> Sorry to say, some recent commit has broken the trunk: > > rhc@bend002 examples]$ mpirun -n 3 ./hello_c > [bend001:22289] *** Process received signal *** > [bend001:22289] Signal: Segmentation fault (11) > [bend001:22289] Signal code: Invalid permissions (2) > [bend001:22289] Failing at address: 0x7f354daaa000 > [bend001:22290] *** Process received signal *** > [bend001:22290] Signal: Segmentation fault (11) > [bend001:22290] Signal code: Invalid permissions (2) > [bend001:22290] Failing at address: 0x7fa819d81000 > [bend001:22289] [ 0] /lib64/libpthread.so.0[0x38e320f710] > [bend001:22289] [ 1] /lib64/libc.so.6[0x38e26845ad] > [bend001:22289] [ 2] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7f3549924b0b] > [bend001:22289] [ 3] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7f354db62a21] > [bend001:22289] [ 4] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7f354a1cfc2c] > [bend001:22289] [ 5] [bend001:22290] [ 0] /lib64/libpthread.so.0[0x38e320f710] > [bend001:22290] [ 1] /lib64/libc.so.6[0x38e26845ad] > [bend001:22290] [ 2] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7fa815bfbb0b] > [bend001:22290] [ 3] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7f354db6189e] > [bend001:22289] [ 6] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7f35492c3cc3] > [bend001:22289] [ 7] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7f354db88261] > [bend001:22289] [ 8] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7f354dafbc7b] > [bend001:22289] [ 9] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7fa819e39a21] > [bend001:22290] [ 4] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7fa8164a6c2c] > [bend001:22290] [ 5] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7fa819e3889e] > [bend001:22290] [ 6] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7fa81559acc3] > [bend001:22290] [ 7] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7fa819e5f261] > [bend001:22290] [ 8] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7f354db2f156] > [bend001:22289] [10] ./hello_c[0x400806] > [bend001:22289] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d] > [bend001:22289] [12] ./hello_c[0x400719] > [bend001:22289] *** End of error message *** > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7fa819dd2c7b] > [bend001:22290] [ 9] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7fa819e06156] > [bend001:22290] [10] ./hello_c[0x400806] > [bend001:22290] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d] > [bend001:22290] [12] ./hello_c[0x400719] > [bend001:22290] *** End of error message *** > [bend001:22291] *** Process received signal *** > [bend001:22291] Signal: Segmentation fault (11) > [bend001:22291] Signal code: Invalid permissions (2) > [bend001:22291] Failing at address: 0x7f498fc96000 > [bend001:22291] [ 0] /lib64/libpthread.so.0[0x38e320f710] > [bend001:22291] [ 1] /lib64/libc.so.6[0x38e26845ad] > [bend001:22291] [ 2] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_btl_vader.so(+0x3b0b)[0x7f498795db0b] > [bend001:22291] [ 3] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_btl_base_select+0x1cc)[0x7f498fd4ea21] > [bend001:22291] [ 4] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x27)[0x7f498c3bbc2c] > [bend001:22291] [ 5] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_bml_base_init+0xe2)[0x7f498fd4d89e] > [bend001:22291] [ 6] > /home/common/openmpi/build/svn-trunk/lib/openmpi/mca_pml_ob1.so(+0x7cc3)[0x7f49872fccc3] > [bend001:22291] [ 7] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(mca_pml_base_select+0x29c)[0x7f498fd74261] > [bend001:22291] [ 8] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(ompi_mpi_init+0x685)[0x7f498fce7c7b] > [bend001:22291] [ 9] > /home/common/openmpi/build/svn-trunk/lib/libmpi.so.0(MPI_Init+0x185)[0x7f498fd1b156] > [bend001:22291] [10] ./hello_c[0x400806] > [bend001:22291] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x38e261ed1d] > [bend001:22291] [12] ./hello_c[0x400719] > [bend001:22291] *** End of error message *** > -------------------------------------------------------------------------- > mpirun noticed that process rank 0 with PID 22289 on node bend001 exited on > signal 11 (Segmentation fault). > -------------------------------------------------------------------------- > 3 total processes killed (some possibly by mpirun during cleanup) > [rhc@bend002 examples]$ > > Nathan: can you please take a look? > > Ralph >
