Dear OMPI community,
We observe a segmentation fault in our regression testing. Our initial investigation shows that It happens for any 1.8.x release and with any PML/BTL/MTL combo on two processes, when running the MPICH one-sided test, accumulate-fence test, attached to this report with the following command line: $mpirun -np 2 --bind-to core --display-map --map-by node -mca pml ob1 -mca btl self,openib ../test/mpi/rma/accfence1 The initial trace is: Data for JOB [16088,1] offset 0 ======================== JOB MAP ======================== Data for node: vegas15 Num slots: 16 Max slots: 0 Num procs: 1 Process OMPI jobid: [16088,1] App: 0 Process rank: 0 Data for node: vegas16 Num slots: 16 Max slots: 0 Num procs: 1 Process OMPI jobid: [16088,1] App: 0 Process rank: 1 ============================================================= [vegas16:22098] *** Process received signal *** [vegas16:22098] Signal: Segmentation fault (11) [vegas16:22098] Signal code: Address not mapped (1) [vegas16:22098] Failing at address: 0x34 [vegas16:22098] [ 0] /lib64/libpthread.so.0[0x3f6e80f710] [vegas16:22098] [ 1] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_int_free+0x188)[0x7ffff772baa2] [vegas16:22098] [ 2] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(opal_memory_ptmalloc2_free+0x98)[0x7ffff772a1f5] [vegas16:22098] [ 3] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libopen-pal.so.6(+0xd6f59)[0x7ffff7728f59] [vegas16:22098] [ 4] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(+0x2f884)[0x7ffff7c92884] [vegas16:22098] [ 5] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_attr_delete_all+0x2eb)[0x7ffff7c92dbe] [vegas16:22098] [ 6] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_comm_free+0x6a)[0x7ffff7c99336] [vegas16:22098] [ 7] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/openmpi/mca_osc_rdma.so(ompi_osc_rdma_free+0x921)[0x7ffff32ab3bc] [vegas16:22098] [ 8] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(ompi_win_free+0x24)[0x7ffff7cc0c87] [vegas16:22098] [ 9] /labhome/alinas/workspace/ompi/openmpi-1.8.4/install/lib/libmpi.so.1(MPI_Win_free+0xb8)[0x7ffff7d2b702] [vegas16:22098] [10] /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402447] [vegas16:22098] [11] /lib64/libc.so.6(__libc_start_main+0xfd)[0x3f6e41ed1d] [vegas16:22098] [12] /labhome/alinas/workspace/mpich/mpich-mellanox/test/mpi/rma/accfence1[0x402119] [vegas16:22098] *** End of error message *** And subsequent investigation of the core file generates the following hints: (gdb) bt #0 0x00007ffff7722a96 in opal_memory_ptmalloc2_int_free (av=0x7ffff796b320, mem=0x7125a0) at malloc.c:4402 #1 0x00007ffff77211f5 in opal_memory_ptmalloc2_free (mem=0x7125a0) at malloc.c:3511 #2 0x00007ffff771ff59 in opal_memory_linux_free_hook (__ptr=0x7125a0, caller=0x7ffff769a8f6) at hooks.c:709 #3 0x00007ffff769a8f6 in opal_datatype_destruct (datatype=0x7123b0) at opal_datatype_create.c:59 #4 0x00007ffff3346ad0 in opal_obj_run_destructors (object=0x7123b0) at ../../../../opal/class/opal_object.h:448 #5 0x00007ffff334af68 in process_acc (module=0x70e370, source=0, acc_header=0x70fef0) at osc_rdma_data_move.c:1184 #6 0x00007ffff334c752 in process_frag (module=0x70e370, frag=0x70fee0) at osc_rdma_data_move.c:1576 #7 0x00007ffff334cafb in ompi_osc_rdma_callback (request=0x700b80) at osc_rdma_data_move.c:1656 #8 0x00007ffff3db3770 in ompi_request_complete (request=0x700b80, with_signal=true) at ../../../../ompi/request/request.h:402 #9 0x00007ffff3db3f11 in recv_request_pml_complete (recvreq=0x700b80) at pml_ob1_recvreq.h:181 #10 0x00007ffff3db5019 in mca_pml_ob1_recv_frag_callback_match (btl=0x7ffff41d9c20, tag=65 'A', des=0x7fffffffd210, cbdata=0x0) at pml_ob1_recvfrag.c:243 #11 0x00007ffff3fd6c4b in mca_btl_sm_component_progress () at btl_sm_component.c:1087 #12 0x00007ffff7678d66 in opal_progress () at runtime/opal_progress.c:187 #13 0x00007ffff3dabb44 in opal_condition_wait (c=0x7ffff7ffa120, m=0x7ffff7ffa160) at ../../../../opal/threads/condition.h:78 #14 0x00007ffff3dabcc6 in ompi_request_wait_completion (req=0x7fffffffd410) at ../../../../ompi/request/request.h:381 #15 0x00007ffff3dac9da in mca_pml_ob1_recv (addr=0x7fffffffd9ec, count=1, datatype=0x7ffff7fe25c0, src=0, tag=-24, comm=0x70dac0, status=0x0) at pml_ob1_irecv.c:109 #16 0x00007ffff2cd2868 in ompi_coll_tuned_scatter_intra_basic_linear (sbuf=0x0, scount=1, sdtype=0x7ffff7fe25c0, rbuf=0x7fffffffd9ec, rcount=1, rdtype=0x7ffff7fe25c0, root=0, comm=0x70dac0, module=0x70fa20) at coll_tuned_scatter.c:231 #17 0x00007ffff2cbbd75 in ompi_coll_tuned_scatter_intra_dec_fixed (sbuf=0x0, scount=1, sdtype=0x7ffff7fe25c0, rbuf=0x7fffffffd9ec, rcount=1, rdtype=0x7ffff7fe25c0, root=0, comm=0x70dac0, module=0x70fa20) at coll_tuned_decision_fixed.c:769 #18 0x00007ffff3b9c16e in mca_coll_basic_reduce_scatter_block_intra (sbuf=0x70e220, rbuf=0x7fffffffd9ec, rcount=1, dtype=0x7ffff7fe25c0, op=0x60d180, comm=0x70dac0, module=0x70f230) at coll_basic_reduce_scatter_block.c:102 #19 0x00007ffff334eecc in ompi_osc_rdma_fence (assert=0, win=0x70e260) at osc_rdma_active_target.c:140 #20 0x00007ffff7d2a1b5 in PMPI_Win_fence (assert=0, win=0x70e260) at pwin_fence.c:59 #21 0x0000000000402405 in main () It looks to be a clear case of memory corruption hidden somewhere in the OSC code. Nathan, can you please have a look? Thanks, Alina.