Hi, The following commit broke trunk in jenkins: >>>Per the OMPI developer conference, remove the last vestiges of OMPI_USE_PROGRESS_THREADS
*22:15:09* + LD_LIBRARY_PATH=/scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib*22:15:09* + OMPI_MCA_scoll_fca_enable=1*22:15:09* + OMPI_MCA_scoll_fca_np=0*22:15:09* + OMPI_MCA_pml=ob1*22:15:09* + OMPI_MCA_btl=sm,self,openib*22:15:09* + OMPI_MCA_spml=yoda*22:15:09* + OMPI_MCA_memheap_mr_interleave_factor=8*22:15:09* + OMPI_MCA_memheap=ptmalloc*22:15:09* + OMPI_MCA_btl_openib_if_include=mlx4_0:1*22:15:09* + OMPI_MCA_rmaps_base_dist_hca=mlx4_0*22:15:09* + OMPI_MCA_memheap_base_hca_name=mlx4_0*22:15:09* + OMPI_MCA_rmaps_base_mapping_policy=dist:mlx4_0*22:15:09* + MXM_RDMA_PORTS=mlx4_0:1*22:15:09* + SHMEM_SYMMETRIC_HEAP_SIZE=1024M*22:15:09* + timeout -s SIGSEGV 3m /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/bin/oshrun -np 8 /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/examples/hello_shmem*22:15:09* [vegas12:08101] *** Process received signal ****22:15:09* [vegas12:08101] Signal: Segmentation fault (11)*22:15:09* [vegas12:08101] Signal code: Address not mapped (1)*22:15:09* [vegas12:08101] Failing at address: (nil)*22:15:09* [vegas12:08101] [ 0] /lib64/libpthread.so.0[0x3937c0f500]*22:15:09* [vegas12:08101] [ 1] /usr/lib64/libibverbs.so.1(ibv_destroy_comp_channel+0x16)[0x7ffff3785f46]*22:15:09* [vegas12:08101] [ 2] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_btl_openib.so(+0xdf02)[0x7ffff3fc1f02]*22:15:09* [vegas12:08101] [ 3] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_btl_openib.so(+0xf161)[0x7ffff3fc3161]*22:15:09* [vegas12:08101] [ 4] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_btl_openib.so(+0x12b41)[0x7ffff3fc6b41]*22:15:09* [vegas12:08101] [ 5] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(mca_btl_base_select+0x117)[0x7ffff7a29807]*22:15:09* [vegas12:08101] [ 6] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7ffff41ed7e2]*22:15:09* [vegas12:08101] [ 7] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(mca_bml_base_init+0x99)[0x7ffff7a29009]*22:15:09* [vegas12:08101] [ 8] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_pml_ob1.so(+0x58b5)[0x7ffff2f528b5]*22:15:09* [vegas12:08101] [ 9] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(mca_pml_base_select+0x1e0)[0x7ffff7a3c590]*22:15:09* [vegas12:08101] [10] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(ompi_mpi_init+0x455)[0x7ffff7a06bf5]*22:15:09* [vegas12:08101] [11] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/liboshmem.so.0(oshmem_shmem_init+0xfd)[0x7ffff7ca66dd]*22:15:09* [vegas12:08101] [12] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/liboshmem.so.0(shmem_init+0x28)[0x7ffff7ca9328]*22:15:09* [vegas12:08101] [13] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/examples/hello_shmem[0x40077d]*22:15:09* [vegas12:08101] [14] /lib64/libc.so.6(__libc_start_main+0xfd)[0x393741ecdd]*22:15:09* [vegas12:08101] [15] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/examples/hello_shmem[0x4006a9]*22:15:09* [vegas12:08101] *** End of error message ****22:15:09* [vegas12][[28889,1],2][btl_openib_component.c:909:device_destruct] Failed to cancel OpenIB progress thread*22:15:09* [vegas12][[28889,1],5][btl_openib_component.c:909:device_destruct] Failed to cancel OpenIB progress thread*22:15:09* [vegas12:08099] *** Process received signal ****22:15:09* [vegas12:08099] Signal: Segmentation fault (11)*22:15:09* [vegas12:08099] Signal code: Address not mapped (1)*22:15:09* [vegas12:08099] Failing at address: (nil)*22:15:09* [vegas12:08099] [ 0] /lib64/libpthread.so.0[0x3937c0f500]*22:15:09* [vegas12:08099] [ 1] /usr/lib64/libibverbs.so.1(ibv_destroy_comp_channel+0x16)[0x7ffff3785f46]*22:15:09* [vegas12:08099] [ 2] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_btl_openib.so(+0xdf02)[0x7ffff3fc1f02]*22:15:09* [vegas12:08099] [ 3] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_btl_openib.so(+0xf161)[0x7ffff3fc3161]*22:15:09* [vegas12:08099] [ 4] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_btl_openib.so(+0x12b41)[0x7ffff3fc6b41]*22:15:09* [vegas12:08099] [ 5] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(mca_btl_base_select+0x117)[0x7ffff7a29807]*22:15:09* [vegas12:08099] [ 6] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_bml_r2.so(mca_bml_r2_component_init+0x12)[0x7ffff41ed7e2]*22:15:09* [vegas12:08099] [ 7] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(mca_bml_base_init+0x99)[0x7ffff7a29009]*22:15:09* [vegas12:08099] [ 8] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/openmpi/mca_pml_ob1.so(+0x58b5)[0x7ffff2f528b5]*22:15:09* [vegas12:08099] [ 9] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(mca_pml_base_select+0x1e0)[0x7ffff7a3c590]*22:15:09* [vegas12:08099] [10] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/libmpi.so.0(ompi_mpi_init+0x455)[0x7ffff7a06bf5]*22:15:09* [vegas12:08099] [11] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/liboshmem.so.0(oshmem_shmem_init+0xfd)[0x7ffff7ca66dd]*22:15:09* [vegas12:08099] [12] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/oshm_install2/lib/liboshmem.so.0(shmem_init+0x28)[0x7ffff7ca9328]*22:15:09* [vegas12:08099] [13] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/examples/hello_shmem[0x40077d]*22:15:09* [vegas12:08099] [14] /lib64/libc.so.6(__libc_start_main+0xfd)[0x393741ecdd]*22:15:09* [vegas12:08099] [15] /scrap/jenkins/scrap/workspace/hpc-ompi-shmem/label/hpc-test-node/examples/hello_shmem[0x4006a9]*22:15:09* [vegas12:08099] *** End of error message ****22:15:09* ------------------------------------------------------------
