This is the a knowing issue, https://svn.open-mpi.org/trac/ompi/ticket/2087 Maybe it's priority should be raised up. Lenny.
On Wed, Dec 30, 2009 at 12:13 PM, Daniel Spångberg <dani...@mkem.uu.se>wrote: > Dear OpenMPI list, > > I have used the dynamic rules for collectives to be able to select one > specific algorithm. With the latest versions of openmpi this seems to be > broken. Just enabling coll_tuned_use_dynamic_rules causes the code to > segfault. However, I do not provide a file with rules, since I just want to > modify the behavior of one routine. > > I have tried the below example code on openmpi 1.3.2, 1.3.3, 1.3.4, and > 1.4. It *works* on 1.3.2, 1.3.3, but segfaults on 1.3.4 and 1.4. I have > confirmed this on Scientific Linux 5.2, and 5.4. I have also successfully > reproduced the crash using version 1.4 running on debian etch. All running > on amd64, compiled from source without other options to configure than > --prefix. The crash occurs whether I use the intel 11.1 compiler (via env > CC) or gcc. It also occurs no matter the btl is set to openib,self tcp,self > sm,self or combinations of those. See below for ompi_info and other info. I > have tried MPI_Alltoall, MPI_Alltoallv, and MPI_Allreduce which behave the > same. > > #include <stdlib.h> > #include <mpi.h> > > > int main(int argc, char **argv) > { > int rank,size; > char *buffer, *buffer2; > > MPI_Init(&argc,&argv); > > MPI_Comm_size(MPI_COMM_WORLD,&size); > MPI_Comm_rank(MPI_COMM_WORLD,&rank); > > buffer=calloc(100*size,1); > buffer2=calloc(100*size,1); > > MPI_Alltoall(buffer,100,MPI_BYTE,buffer2,100,MPI_BYTE,MPI_COMM_WORLD); > > MPI_Finalize(); > return 0; > } > > Demonstrated behaviour: > > $ ompi_info > Package: Open MPI daniels@arthur Distribution > Open MPI: 1.4 > Open MPI SVN revision: r22285 > Open MPI release date: Dec 08, 2009 > Open RTE: 1.4 > Open RTE SVN revision: r22285 > Open RTE release date: Dec 08, 2009 > OPAL: 1.4 > OPAL SVN revision: r22285 > OPAL release date: Dec 08, 2009 > Ident string: 1.4 > Prefix: > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install > Configured architecture: x86_64-unknown-linux-gnu > Configure host: arthur > Configured by: daniels > Configured on: Tue Dec 29 16:54:37 CET 2009 > Configure host: arthur > Built by: daniels > Built on: Tue Dec 29 17:04:36 CET 2009 > Built host: arthur > C bindings: yes > C++ bindings: yes > Fortran77 bindings: yes (all) > Fortran90 bindings: yes > Fortran90 bindings size: small > C compiler: gcc > C compiler absolute: /usr/bin/gcc > C++ compiler: g++ > C++ compiler absolute: /usr/bin/g++ > Fortran77 compiler: gfortran > Fortran77 compiler abs: /usr/bin/gfortran > Fortran90 compiler: gfortran > Fortran90 compiler abs: /usr/bin/gfortran > C profiling: yes > C++ profiling: yes > Fortran77 profiling: yes > Fortran90 profiling: yes > C++ exceptions: no > Thread support: posix (mpi: no, progress: no) > Sparse Groups: no > Internal debug support: no > MPI parameter check: runtime > Memory profiling support: no > Memory debugging support: no > libltdl support: yes > Heterogeneous support: no > mpirun default --prefix: no > MPI I/O support: yes > MPI_WTIME support: gettimeofday > Symbol visibility support: yes > FT Checkpoint support: no (checkpoint thread: no) > MCA backtrace: execinfo (MCA v2.0, API v2.0, Component v1.4) > MCA memory: ptmalloc2 (MCA v2.0, API v2.0, Component v1.4) > MCA paffinity: linux (MCA v2.0, API v2.0, Component v1.4) > > MCA carto: auto_detect (MCA v2.0, API v2.0, Component v1.4) > MCA carto: file (MCA v2.0, API v2.0, Component v1.4) > MCA maffinity: first_use (MCA v2.0, API v2.0, Component v1.4) > MCA timer: linux (MCA v2.0, API v2.0, Component v1.4) > MCA installdirs: env (MCA v2.0, API v2.0, Component v1.4) > MCA installdirs: config (MCA v2.0, API v2.0, Component v1.4) > MCA dpm: orte (MCA v2.0, API v2.0, Component v1.4) > MCA pubsub: orte (MCA v2.0, API v2.0, Component v1.4) > MCA allocator: basic (MCA v2.0, API v2.0, Component v1.4) > MCA allocator: bucket (MCA v2.0, API v2.0, Component v1.4) > MCA coll: basic (MCA v2.0, API v2.0, Component v1.4) > MCA coll: hierarch (MCA v2.0, API v2.0, Component v1.4) > MCA coll: inter (MCA v2.0, API v2.0, Component v1.4) > MCA coll: self (MCA v2.0, API v2.0, Component v1.4) > MCA coll: sm (MCA v2.0, API v2.0, Component v1.4) > MCA coll: sync (MCA v2.0, API v2.0, Component v1.4) > MCA coll: tuned (MCA v2.0, API v2.0, Component v1.4) > MCA io: romio (MCA v2.0, API v2.0, Component v1.4) > MCA mpool: fake (MCA v2.0, API v2.0, Component v1.4) > MCA mpool: rdma (MCA v2.0, API v2.0, Component v1.4) > MCA mpool: sm (MCA v2.0, API v2.0, Component v1.4) > MCA pml: cm (MCA v2.0, API v2.0, Component v1.4) > MCA pml: csum (MCA v2.0, API v2.0, Component v1.4) > MCA pml: ob1 (MCA v2.0, API v2.0, Component v1.4) > MCA pml: v (MCA v2.0, API v2.0, Component v1.4) > MCA bml: r2 (MCA v2.0, API v2.0, Component v1.4) > MCA rcache: vma (MCA v2.0, API v2.0, Component v1.4) > MCA btl: self (MCA v2.0, API v2.0, Component v1.4) > MCA btl: sm (MCA v2.0, API v2.0, Component v1.4) > MCA btl: tcp (MCA v2.0, API v2.0, Component v1.4) > MCA topo: unity (MCA v2.0, API v2.0, Component v1.4) > MCA osc: pt2pt (MCA v2.0, API v2.0, Component v1.4) > MCA osc: rdma (MCA v2.0, API v2.0, Component v1.4) > MCA iof: hnp (MCA v2.0, API v2.0, Component v1.4) > MCA iof: orted (MCA v2.0, API v2.0, Component v1.4) > MCA iof: tool (MCA v2.0, API v2.0, Component v1.4) > MCA oob: tcp (MCA v2.0, API v2.0, Component v1.4) > MCA odls: default (MCA v2.0, API v2.0, Component v1.4) > MCA ras: slurm (MCA v2.0, API v2.0, Component v1.4) > MCA rmaps: load_balance (MCA v2.0, API v2.0, Component v1.4) > MCA rmaps: rank_file (MCA v2.0, API v2.0, Component v1.4) > MCA rmaps: round_robin (MCA v2.0, API v2.0, Component v1.4) > MCA rmaps: seq (MCA v2.0, API v2.0, Component v1.4) > MCA rml: oob (MCA v2.0, API v2.0, Component v1.4) > MCA routed: binomial (MCA v2.0, API v2.0, Component v1.4) > MCA routed: direct (MCA v2.0, API v2.0, Component v1.4) > MCA routed: linear (MCA v2.0, API v2.0, Component v1.4) > MCA plm: rsh (MCA v2.0, API v2.0, Component v1.4) > MCA plm: slurm (MCA v2.0, API v2.0, Component v1.4) > MCA filem: rsh (MCA v2.0, API v2.0, Component v1.4) > MCA errmgr: default (MCA v2.0, API v2.0, Component v1.4) > MCA ess: env (MCA v2.0, API v2.0, Component v1.4) > MCA ess: hnp (MCA v2.0, API v2.0, Component v1.4) > MCA ess: singleton (MCA v2.0, API v2.0, Component v1.4) > MCA ess: slurm (MCA v2.0, API v2.0, Component v1.4) > MCA ess: tool (MCA v2.0, API v2.0, Component v1.4) > MCA grpcomm: bad (MCA v2.0, API v2.0, Component v1.4) > MCA grpcomm: basic (MCA v2.0, API v2.0, Component v1.4) > > $ mpicc -O2 -o bug_openmpi_1.4_test bug_openmpi_1.4_test.c > $ ldd ./bug_openmpi_1.4_test > libmpi.so.0 => > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0 > (0x00002b33fa57e000) > libopen-rte.so.0 => > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libopen-rte.so.0 > (0x00002b33fa821000) > libopen-pal.so.0 => > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libopen-pal.so.0 > (0x00002b33faa6b000) > libdl.so.2 => /lib64/libdl.so.2 (0x00000032c7400000) > libnsl.so.1 => /lib64/libnsl.so.1 (0x00000032cfe00000) > libutil.so.1 => /lib64/libutil.so.1 (0x00000032d4a00000) > libm.so.6 => /lib64/libm.so.6 (0x00000032c7000000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00000032c7800000) > libc.so.6 => /lib64/libc.so.6 (0x00000032c6c00000) > /lib64/ld-linux-x86-64.so.2 (0x00000032c5c00000) > $ mpirun -mca btl tcp,self -mca coll_tuned_use_dynamic_rules 0 -np 8 > ./bug_openmpi_1.4_test > $ mpirun -mca btl tcp,self -mca coll_tuned_use_dynamic_rules 1 -np 8 > ./bug_openmpi_1.4_test > [girasole:27510] *** Process received signal *** > [girasole:27510] Signal: Segmentation fault (11) > [girasole:27510] Signal code: (128) > [girasole:27510] Failing at address: (nil) > [girasole:27503] *** Process received signal *** > [girasole:27503] Signal: Segmentation fault (11) > [girasole:27503] Signal code: (128) > [girasole:27503] Failing at address: (nil) > [girasole:27510] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27510] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2ae2b29fbeb5] > [girasole:27510] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2ae2b29fa8ca] > [girasole:27510] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2ae2ae76bbff] > [girasole:27510] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27510] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27510] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27510] *** End of error message *** > [girasole:27503] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27503] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b534b1b6eb5] > [girasole:27503] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b534b1b58ca] > [girasole:27503] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2b5346f26bff] > [girasole:27503] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27503] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27503] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27503] *** End of error message *** > [girasole:27505] *** Process received signal *** > [girasole:27505] Signal: Segmentation fault (11) > [girasole:27505] Signal code: (128) > [girasole:27505] Failing at address: (nil) > [girasole:27509] *** Process received signal *** > [girasole:27509] Signal: Segmentation fault (11) > [girasole:27509] Signal code: (128) > [girasole:27509] Failing at address: (nil) > [girasole:27505] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27505] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2ab662aa0eb5] > [girasole:27505] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2ab662a9f8ca] > [girasole:27505] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2ab65e810bff] > [girasole:27505] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27505] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27505] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27505] *** End of error message *** > [girasole:27507] *** Process received signal *** > [girasole:27507] Signal: Segmentation fault (11) > [girasole:27507] Signal code: (128) > [girasole:27507] Failing at address: (nil) > [girasole:27509] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27509] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b7dc1863eb5] > [girasole:27509] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b7dc18628ca] > [girasole:27509] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2b7dbd5d3bff] > [girasole:27509] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27509] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27509] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27509] *** End of error message *** > [girasole:27507] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27507] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b09eb873eb5] > [girasole:27507] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b09eb8728ca] > [girasole:27507] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2b09e75e3bff] > [girasole:27507] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27507] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27507] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27507] *** End of error message *** > [girasole:27504] *** Process received signal *** > [girasole:27504] Signal: Segmentation fault (11) > [girasole:27504] Signal code: (128) > [girasole:27504] Failing at address: (nil) > [girasole:27506] *** Process received signal *** > [girasole:27506] Signal: Segmentation fault (11) > [girasole:27506] Signal code: (128) > [girasole:27506] Failing at address: (nil) > [girasole:27504] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27504] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b6fde1afeb5] > [girasole:27504] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b6fde1ae8ca] > [girasole:27504] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2b6fd9f1fbff] > [girasole:27504] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27504] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27504] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27504] *** End of error message *** > -------------------------------------------------------------------------- > mpirun noticed that process rank 7 with PID 27510 on node girasole exited > on signal 11 (Segmentation fault). > -------------------------------------------------------------------------- > [girasole:27506] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27506] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b66f2908eb5] > [girasole:27506] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b66f29078ca] > [girasole:27506] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2b66ee678bff] > [girasole:27506] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27506] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27506] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27506] *** End of error message *** > [girasole:27508] *** Process received signal *** > [girasole:27508] Signal: Segmentation fault (11) > [girasole:27508] Signal code: (128) > [girasole:27508] Failing at address: (nil) > [girasole:27508] [ 0] /lib64/libpthread.so.0 [0x32c780de80] > [girasole:27508] [ 1] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b89b09a1eb5] > [girasole:27508] [ 2] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/openmpi/mca_coll_tuned.so > [0x2b89b09a08ca] > [girasole:27508] [ 3] > /home/daniels/src/MISC/openmpi-1.4/openmpi-1.4_install/lib/libmpi.so.0(MPI_Alltoall+0x15f) > [0x2b89ac711bff] > [girasole:27508] [ 4] ./bug_openmpi_1.4_test(main+0x97) [0x4009b7] > [girasole:27508] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) > [0x32c6c1d8b4] > [girasole:27508] [ 6] ./bug_openmpi_1.4_test [0x400869] > [girasole:27508] *** End of error message *** > > > Best regards, > > -- > Daniel Spångberg > Materialkemi > Uppsala Universitet > _______________________________________________ > users mailing list > us...@open-mpi.org > http://www.open-mpi.org/mailman/listinfo.cgi/users >