[OMPI devel] CUDA kernels in OpenMPI
I'm trying to build a CUDA kernel into OpenMPI (because I'm experimenting with an Allreduce collective with data in GPU buffers, and I want the GPU to do the reduction). This involves writing a '.cu' file, and compiling this to '.o' with the NVIDIA CUDA compiler 'nvcc'; and also writing some of OpenMPI in C++ to interface with the CUDA kernel. I've revised the Makefile.am in the relevant directory to look like (there is more, but this is the relevant section) sources = \ coll_ibm.h \ coll_ibm_component.c \ coll_ibm_module.c \ coll_ibm_allgather.c \ coll_ibm_allgatherv.c \ coll_ibm_allreduce.c \ coll_ibm_alltoall.c \ coll_ibm_alltoallv.c \ coll_ibm_barrier.c \ coll_ibm_bcast.c \ coll_ibm_exscan.c \ coll_ibm_gather.c \ coll_ibm_gatherv.c \ coll_ibm_reduce.c \ coll_ibm_reduce_scatter.c \ coll_ibm_reduce_scatter_block.c \ coll_ibm_scan.c \ coll_ibm_scatter.c \ coll_ibm_scatterv.c \ allreduce_overlap.cc \ comm_gpu.cu SUFFIXES = .cu comm_gpu.lo: comm_gpu.cu /usr/local/cuda/bin/nvcc -gencode arch=compute_60,code=sm_60 -lcuda -O3 --compiler-options "-O2 -fopenmp -mcpu=power8 -fPIC" -c comm_gpu.cu mv comm_gpu.o comm_gpu.lo The C++ compile runs OK; but the attempt to run 'nvcc' is ignored; the build completed, but no comm_gpu.lo is generated, and as far as I can tell, the library for this directory is built without the comm_gpu.lo object. What do I need to put in the Makefile.am to get this to build properly ? T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - Active Storage Fabrics - IBM System BlueGene IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/profile/view?id=60628729 ResearchGate https://www.researchgate.net/profile/T_Ward2 IBM System BlueGene Research IBM System BlueGene Marketing IBM Resources for Global Servants IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today New Lamps For Old - Diskless Remote Boot Linux from National Center for High-Performance Computing, Taiwan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
[OMPI devel] CUDA kernels in OpenMPI
Here is the complete Makefile so far. I have it in directory ompi/mca/coll/ibm , which contains an implementation of an IBM-written collectives library. It won't work as-is, because I don't know how to use 'libtool' which is presumably needed to do the compile. If anybody can show me a rule using libtool to do a compile with 'nvcc', I would be most grateful. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - Active Storage Fabrics - IBM System BlueGene IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/profile/view?id=60628729 ResearchGate https://www.researchgate.net/profile/T_Ward2 IBM System BlueGene Research IBM System BlueGene Marketing IBM Resources for Global Servants IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today New Lamps For Old - Diskless Remote Boot Linux from National Center for High-Performance Computing, Taiwan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Makefile.am Description: Binary data ___ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
[OMPI devel] CUDA kernels in OpenMPI
It looks like the mailing system deleted the attachment, so here it is inline # # Copyright (c) 2004-2005 The Trustees of Indiana University and Indiana # University Research and Technology # Corporation. All rights reserved. # Copyright (c) 2004-2005 The University of Tennessee and The University # of Tennessee Research Foundation. All rights # reserved. # Copyright (c) 2004-2009 High Performance Computing Center Stuttgart, # University of Stuttgart. All rights reserved. # Copyright (c) 2004-2005 The Regents of the University of California. # All rights reserved. # Copyright (c) 2010 Cisco Systems, Inc. All rights reserved. # Copyright (c) 2012 Sandia National Laboratories. All rights reserved. # Copyright (c) 2013 Los Alamos National Security, LLC. All rights # reserved. # Copyright (c) 2016 IBM Corporation. All rights reserved. # $COPYRIGHT$ # # Additional copyrights may follow # # $HEADER$ # AM_CPPFLAGS = $(coll_ibm_CPPFLAGS) sources = \ coll_ibm.h \ coll_ibm_component.c \ coll_ibm_module.c \ coll_ibm_allgather.c \ coll_ibm_allgatherv.c \ coll_ibm_allreduce.c \ coll_ibm_alltoall.c \ coll_ibm_alltoallv.c \ coll_ibm_barrier.c \ coll_ibm_bcast.c \ coll_ibm_exscan.c \ coll_ibm_gather.c \ coll_ibm_gatherv.c \ coll_ibm_reduce.c \ coll_ibm_reduce_scatter.c \ coll_ibm_reduce_scatter_block.c \ coll_ibm_scan.c \ coll_ibm_scatter.c \ coll_ibm_scatterv.c \ comm_gpu.cu \ allreduce_overlap.cc SUFFIXES = .cu # #comm_gpu.lo: comm_gpu.cu # /usr/local/cuda/bin/nvcc -gencode arch=compute_60,code=sm_60 -lcuda -O3 --compiler-options "-O2 -fopenmp -mcpu=power8 -fPIC" -c comm_gpu.cu # mv comm_gpu.o comm_gpu.lo %.lo : %.cu /usr/local/cuda/bin/nvcc -gencode arch=compute_60,code=sm_60 -lcuda -O3 --compiler-options "-O2 -fopenmp -mcpu=power8 -fPIC" -c $< mv $*.o .libs/ touch $*.lo # Make the output library in this directory, and name it either # mca__.la (for DSO builds) or libmca__.la # (for static builds). if MCA_BUILD_ompi_coll_ibm_DSO component_noinst = component_install = mca_coll_ibm.la else component_noinst = libmca_coll_ibm.la component_install = endif mcacomponentdir = $(ompilibdir) mcacomponent_LTLIBRARIES = $(component_install) mca_coll_ibm_la_SOURCES = $(sources) if WANT_COLL_IBM_WITH_PAMI mca_coll_ibm_la_LIBADD = $(coll_ibm_LIBS) \ $(OMPI_TOP_BUILDDIR)/ompi/mca/common/pami/libmca_common_pami.la else mca_coll_ibm_la_LIBADD = $(coll_ibm_LIBS) endif mca_coll_ibm_la_LDFLAGS = -module -avoid-version $(coll_ibm_LDFLAGS) noinst_LTLIBRARIES = $(component_noinst) libmca_coll_ibm_la_SOURCES =$(sources) libmca_coll_ibm_la_LIBADD = $(coll_ibm_LIBS) libmca_coll_ibm_la_LDFLAGS = -module -avoid-version $(coll_ibm_LDFLAGS) T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - Active Storage Fabrics - IBM System BlueGene IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/profile/view?id=60628729 ResearchGate https://www.researchgate.net/profile/T_Ward2 IBM System BlueGene Research IBM System BlueGene Marketing IBM Resources for Global Servants IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today New Lamps For Old - Diskless Remote Boot Linux from National Center for High-Performance Computing, Taiwan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
Re: [OMPI devel] CUDA kernels in OpenMPI
I added the following lines to my 'Makefile.am' in the directory with the CUDA sources .cu.lo: /usr/local/cuda/bin/nvcc -gencode arch=compute_60,code=sm_60 -O3 --cuda -c $< mv -f $*.cu.cpp.ii $*.ii libtool --mode=compile $(CXX) $(CXXFLAGS) -c $*.ii and added the CUDA sources to my 'SOURCES' variable. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - Active Storage Fabrics - IBM System BlueGene IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/profile/view?id=60628729 ResearchGate https://www.researchgate.net/profile/T_Ward2 IBM System BlueGene Research IBM System BlueGene Marketing IBM Resources for Global Servants IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today New Lamps For Old - Diskless Remote Boot Linux from National Center for High-Performance Computing, Taiwan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
[OMPI devel] Fix for MPI_SIZEOF when building OMPI with gfortran 4.8
I have put up a pull request https://github.com/open-mpi/ompi/pull/3822 to supply a fix for the fact that MPI_SIZEOF doesn't build when building OMPI with gfortran 4.8 (on RHEL 7). My team lead hopes that Jeff Squyres will review the patch. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - Active Storage Fabrics - IBM System BlueGene IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/profile/view?id=60628729 ResearchGate https://www.researchgate.net/profile/T_Ward2 IBM System BlueGene Research IBM System BlueGene Marketing IBM Resources for Global Servants IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today New Lamps For Old - Diskless Remote Boot Linux from National Center for High-Performance Computing, Taiwan Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
[OMPI devel] Fw: Is OMPI_ENABLE_HETEROGENEOUS_SUPPORT still supported ?
I want to build a version of OpenMPI v4 to run on a cluster where some ranks are IBM POWER (little endian) and other ranks are x86_64. The OpenMPI wiki says that I need to define OMPI_ENABLE_HETEROGENEOUS_SUPPORT so that structure padding will be inserted as appropriate to make this work; but a quick 'grep' in the source code finds that this macro isn't used anywhere. Can I build OpenMPI for such a cluster nowadays ? T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU ___ devel mailing list devel@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/devel
[OMPI devel] hcoll missing libsharp
One of my fellow developers at IBM is having problems building OpenMPI 4. The 'configure' command is failing trying to set up the mca hcoll '.so'. The error message is to the effect that libsharp.so is missing. Looking by hand, the sharp library is in the expected place in the MOFED install. Is there some option to 'configure' that he is missing ? T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
'configure' ends with --- MCA component coll:hcoll (m4 configuration macro) checking for MCA component coll:hcoll compile mode... dso checking hcoll/api/hcoll_api.h usability... yes checking hcoll/api/hcoll_api.h presence... yes checking for hcoll/api/hcoll_api.h... yes looking for library in lib checking for library containing hcoll_get_version... no looking for library in lib64 checking for library containing hcoll_get_version... no configure: error: HCOLL support requested but not found. Aborting make[3]: *** [config.status] Error 1 and I find the following in 'config.log' : main () ^ /tmp/cciCRCpI.o: In function `main': conftest.c:(.text.startup+0x7): undefined reference to `hcoll_get_version' collect2: error: ld returned 1 exit status .. /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/bin/ld: warning: libsharp_coll.so.2, needed by /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib/libhcoll.so, not found (try using -rpath or -rpath-link) /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/bin/ld: warning: libsharp.so.2, needed by /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib/libhcoll.so, not found (try using -rpath or -rpath-link) /lib64/libibnetdisc.so.5: undefined reference to `ibd_mkey' /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib/libhcoll.so: undefined reference to `sharp_coll_strerror' /lib64/libibnetdisc.so.5: undefined reference to `get_mkey_by_portid' /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib/libhcoll.so: undefined reference to `sharp_coll_dereg_mr' /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib/libhcoll.so: undefined reference to `sharp_coll_req_test' T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
The 'configure' command is hidden a few levels of Makefile down; it may take me a while to isolate it. I will send another note when I have all the information requested. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
I have put the tarball requested on my web site here http://tjcw.freeshell.org/ompi-output.tar.bz2 ; it is too large to be posted to the mailing list. It has a 'typescript' from running configure, and the config.log file. My 'configure' command was ../configure --prefix=/install/u/tjcw/workspace/ompi_install --enable-shared --enable-mca-no-build=btl-uct --disable-static --enable-mpi-fortran=usempi --disable-libompitrace --enable-script-wrapper-compilers --enable-wrapper-rpath --with-hcoll=/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll --with-ucx=/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies --with-ucx-libdir=/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/lib I was trying to build the 'master' branch of ompi, at commit e1e6d8b85e2a27e4981566d8b110730857314c54 . The 'gcc' command which fails is at line 132820 of 'config.log' T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
The sharp libs are in /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib . I will an appropriate LDFLAGS to the configure command. Thanks ! Yes, I expected to be using /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/bin/ld ; for some builds we build our own 'binutils' rather than using the binutils in the linux distribution. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
Adding an appropriate LDFLAGS= didn't help; the revised tarball is here http://tjcw.freeshell.org/ompi-output-2.tar.bz2 . Do I need to specify '-lsharp' to the link command ? If so, how do I do that ? T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
-bash-4.2$ ls -l /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2 lrwxrwxrwx 1 tjcw tjcw 22 Oct 14 08:58 /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2 -> libsharp_coll.so.2.0.0 -bash-4.2$ ls -lL /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2 -rwxr-xr-x 1 tjcw tjcw 407576 Oct 14 08:58 /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2 -bash-4.2$ ldd /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/libsharp_coll.so.2 linux-vdso.so.1 => (0x7ffee0506000) libsharp.so.2 => /install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib/../lib/libsharp.so.2 (0x7f4a84de1000) libdl.so.2 => /lib64/libdl.so.2 (0x7f4a84bdd000) libibverbs.so.1 => /lib64/libibverbs.so.1 (0x7f4a849c4000) libibcm.so.1 => /lib64/libibcm.so.1 (0x7f4a847bf000) librdmacm.so.1 => /lib64/librdmacm.so.1 (0x7f4a845a3000) libibumad.so.3 => /lib64/libibumad.so.3 (0x7f4a84399000) libz.so.1 => /lib64/libz.so.1 (0x7f4a84183000) libpthread.so.0 => /lib64/libpthread.so.0 (0x7f4a83f67000) librt.so.1 => /lib64/librt.so.1 (0x7f4a83d5f000) libc.so.6 => /lib64/libc.so.6 (0x7f4a83992000) /lib64/ld-linux-x86-64.so.2 (0x7f4a85206000) libnl-route-3.so.200 => /lib64/libnl-route-3.so.200 (0x7f4a83725000) libnl-3.so.200 => /lib64/libnl-3.so.200 (0x7f4a83504000) libm.so.6 => /lib64/libm.so.6 (0x7f4a83202000) -bash-4.2$ No, I don't know what ibd_mkey and get_mkey_by_portid are, either. AIX has a command that will tell you which shared object resolves a symbol from a built shared object, but I don't think linux has an equvalent command ... T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
Setting LD_LIBRARY_PATH didn't help; I got the same error. Is the problem because of my MOFED level ? It may be that libsharp.so is in a different directory, or that libhcoll.so depends on libhsharp.so in a different way, than with other levels of MOFED. I just tried building ompi v4.0.x , and it has the same 'configure' problem for me as master. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
I'm using a MOFED from file MLNX_OFED_LINUX-4.0-0.0.8.2-rhel7.3-x86_64.tgz , this on a machine running RHEL 7.6 . Should I be using a newer MOFED ? T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
In my last posting, I had a typo in my LD_LIBRARY_PATH setting. With this fixed, now I get configure:279563: gcc -std=gnu99 -std=gnu99 -o conftest -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -mcx16 -pthread -I/install/u/tjcw/workspace/ompi/build/opal/mca/event/libevent2022/libevent/include -I/install/u/tjcw/workspace/ompi/opal/mca/event/libevent2022/libevent -I/install/u/tjcw/workspace/ompi/opal/mca/event/libevent2022/libevent/include -I/install/u/tjcw/workspace/ompi/build/opal/mca/hwloc/hwloc201/hwloc/include -I/install/u/tjcw/workspace/ompi/opal/mca/hwloc/hwloc201/hwloc/include -I/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/include -L/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/sharp/lib -L/install/u/tjcw/workspace/ibm_smpi_toucan_ucx/ompibase/dependencies/mofed_400/opt/mellanox/hcoll/lib64 conftest.c -lhcoll -lrt -lm -lutil -lz >&5 //usr/lib64/libibnetdisc.so.5: undefined reference to `ibd_mkey' //usr/lib64/libibnetdisc.so.5: undefined reference to `get_mkey_by_portid' //usr/lib64/libibnetdisc.so.5: undefined reference to `is_gi_supported' //usr/lib64/libibnetdisc.so.5: undefined reference to `ibd_mkey_mgr' //usr/lib64/libibnetdisc.so.5: undefined reference to `is_mlnx_ext_port_info_supported' collect2: error: ld returned 1 exit status i.e. it's only missing symbols from libnetdisc.so , but more symbols than it was missing before. My next try will be to install MOFED 4.7 T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
I set up MOFED 4.7.1 , and now the configure complete successfully (without needing to set LD_LIBRARY_PATH or add LDFLAGS=-L...). But the 'make' fails; the last lines of the output are CCLD mca_coll_hcoll.la /bin/ld: cannot find -ludev collect2: error: ld returned 1 exit status make[2]: *** [mca_coll_hcoll.la] Error 1 make[2]: Leaving directory `/smpi_dev/tjcw/workspace/ompi/build/ompi/mca/coll/hcoll' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory `/smpi_dev/tjcw/workspace/ompi/build/ompi' make: *** [all-recursive] Error 1 [f8n02][/smpi_dev/tjcw/workspace/ompi/build]> My system (RHEL 7,6) has a /usr/lib64/libudev.so.1 , but doesn't have a /usr/lib64/libudev.so symbolic link. Tarball of the configure and build logs is here http://tjcw.freeshell.org/ompi-output-3.tar.bz2 . I was building commitb6c4d5c1180ab39beae3a4d0793b149e83192109 of the 'master' branch. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
Re: [OMPI devel] hcoll missing libsharp
Thanks ! OpenMPI builds successfully for me now. T J (Chris) Ward, IBM Research. Scalable Data-Centric Computing - IBM Spectrum MPI IBM United Kingdom Ltd., Hursley Park, Winchester, Hants, SO21 2JN 011-44-1962-818679 LinkedIn https://www.linkedin.com/in/tjcward/ ResearchGate https://www.researchgate.net/profile/Thomas_Ward16 IBM Research -- Data Centric Systems IBM Supercomputer Marketing IBM Branded Products IBM Branded Swag UNIX in the Cloud - Find A Place Where There's Room To Grow, with the original Open Standard. Free Trial Here Today Protein Folding by Supercomputer - BlueMatter Molecular Dynamics package. Data Tables - In-memory key-value store package. Linux on Windows - Virtualisation package. On the Lighthouse ! Unless stated otherwise above: IBM United Kingdom Limited - Registered in England and Wales with number 741598. Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
