Here is some additional information about when using `--with-hwloc=/usr` seems necessary.
Using the tarball for the released, stable 13bb410b52becbfa140f5791bd50d580 /sw/src/arcts/ompi/openmpi-1.10.7.tar.gz and with $ ./configure --prefix=/sw/arcts/centos7/intel_14_0_2/openmpi/1.10.7 --mandir=/sw/arcts/centos7/intel_14_0_2/openmpi/1.10.7/share/man --with-slurm --without-tm --with-verbs --with-hwloc --disable-dlopen --enable-shared CC=icc CXX=icpc FC=ifort F77=ifort config.log reports configure:67748: checking hwloc.h usability configure:67748: icc -std=gnu99 -c -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -restrict -Qoption,cpp,--extended_float_types -pthread conftest.c >&5 configure:67748: $? = 0 configure:67748: result: yes configure:67748: checking hwloc.h presence configure:67748: icc -E conftest.c configure:67748: $? = 0 configure:67748: result: yes configure:67748: checking for hwloc.h configure:67748: result: yes but make reports Making all in mca/hwloc make[2]: Entering directory `/tmp/bennet/build/openmpi-1.10.7/opal/mca/hwloc' CC base/hwloc_base_frame.lo CC base/hwloc_base_util.lo CC base/hwloc_base_dt.lo CC base/hwloc_base_maffinity.lo In file included from ../../../opal/mca/hwloc/hwloc.h(134), from base/hwloc_base_frame.c(23): ../../../opal/mca/hwloc/external/external.h(20): catastrophic error: cannot open source file "/include/hwloc.h" #include MCA_hwloc_external_header ^ compilation aborted for base/hwloc_base_frame.c (code 4) It looks to me like configure is not prepending `/usr` to the path to `/include/hwloc.h` when one uses the bare `--with-hwloc` on the configure line, and therefore using `--with-hwloc=/usr` is called for if one wants it included. Thanks, -- bennet Note, this test fails, but it isn't directly related to finding hwloc.h. configure:67840: result: looking for library without search path configure:67842: checking for library containing hwloc_topology_init configure:67873: icc -std=gnu99 -o conftest -O3 -DNDEBUG -finline-functions -fno-strict-aliasing -restrict -Qoption,cpp,--extended_float_types -pthread conftest.c -lutil >&5 /tmp/iccu8nO7c.o: In function `main': conftest.c:(.text+0x35): undefined reference to `hwloc_topology_init' configure:67873: $? = 1 with too much more output to include here. Seems to be the same situation with the last available nightly build, as well. bcea63d634d05c0f5a821ce75a1eb2b2 openmpi-v1.10-201705170239-5e373bf.tar.gz On Sun, Feb 24, 2019 at 8:11 AM Bennet Fauber <ben...@umich.edu> wrote: > > HI, Gilles, > > With respect to your comment about not using --FOO=/usr.... It is bad > practice, sure, and it should be unnecessary, but we have had at least > one instance where it is also necessary for the requested feature to > actually work. The case I am thinking of was, in particular, OpenMPI > 1.10.2, where OMPI did not properly bind processes to cores unless we > built --with-hwloc=/usr. > > When it wasn't used, it mostly ran fine, but for one program, it would > occasionally result in 'hopping processes' and very poor performance. > After rebuilding --with-hwloc=/usr, that is no longer a problem. It > was difficult to pin down, because it did not seem to be an issue all > the time, but maybe one in five dragging performance. > > So, while it may be the case that it should not be necessary and not > be good practice, it may also be the case that sometimes it does seem > to be necessary. > > -- bennet > > On Sun, Feb 24, 2019 at 5:21 AM Gilles Gouaillardet > <gilles.gouaillar...@gmail.com> wrote: > > > > Passant, > > > > The fix is included in PMIx 2.2.2 > > > > The bug is in a public header file, so you might indeed have to > > rebuild the SLURM plugin for PMIx. > > I did not check the SLURM sources though, so assuming PMIx was built > > as a shared library, there is still a chance > > it might work even if you do not rebuild the SLURM plugin. I'd rebuild > > at least the SLURM plugin for PMIx to be on the safe side though. > > > > Cheers, > > > > Gilles > > > > On Sun, Feb 24, 2019 at 4:07 PM Passant A. Hafez > > <passant.ha...@kaust.edu.sa> wrote: > > > > > > Thanks Gilles. > > > > > > So do we have to rebuild Slurm after applying this patch? > > > > > > Another question, is this fix included in the PMIx 2.2.2 > > > https://github.com/pmix/pmix/releases/tag/v2.2.2 ? > > > > > > > > > > > > > > > All the best, > > > > > > > > > ________________________________________ > > > From: users <users-boun...@lists.open-mpi.org> on behalf of Gilles > > > Gouaillardet <gilles.gouaillar...@gmail.com> > > > Sent: Sunday, February 24, 2019 4:09 AM > > > To: Open MPI Users > > > Subject: Re: [OMPI users] Building PMIx and Slurm support > > > > > > Passant, > > > > > > you have to manually download and apply > > > https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch > > > to PMIx 2.2.1 > > > that should likely fix your problem. > > > > > > As a side note, it is a bad practice to configure --with-FOO=/usr > > > since it might have some unexpected side effects. > > > Instead, you can replace > > > > > > configure --with-slurm --with-pmix=/usr --with-pmi=/usr > > > --with-libevent=/usr > > > > > > with > > > > > > configure --with-slurm --with-pmix=external --with-pmi > > > --with-libevent=external > > > > > > to be on the safe side I also invite you to pass --with-hwloc=external > > > to the configure command line > > > > > > > > > Cheers, > > > > > > Gilles > > > > > > On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez > > > <passant.ha...@kaust.edu.sa> wrote: > > > > > > > > Hello Gilles, > > > > > > > > Here are some details: > > > > > > > > Slurm 18.08.4 > > > > > > > > PMIx 2.2.1 (as shown in /usr/include/pmix_version.h) > > > > > > > > Libevent 2.0.21 > > > > > > > > srun --mpi=list > > > > srun: MPI types are... > > > > srun: none > > > > srun: openmpi > > > > srun: pmi2 > > > > srun: pmix > > > > srun: pmix_v2 > > > > > > > > Open MPI versions tested: 4.0.0 and 3.1.2 > > > > > > > > > > > > For each installation to be mentioned a different MPI Hello World > > > > program was compiled. > > > > Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun > > > > --mpi=pmix program > > > > > > > > File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with > > > > configure options: > > > > --with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr > > > > and configure log: > > > > Libevent support: external > > > > PMIx support: External (2x) > > > > > > > > File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with > > > > configure options: > > > > --with-slurm --with-pmix > > > > and configure log: > > > > Libevent support: internal (external libevent version is less that > > > > internal version 2.0.22) > > > > PMIx support: Internal > > > > > > > > Tested also different installations for 3.1.2 and got errors similar to > > > > 400ext_2x2.out > > > > (NOT-SUPPORTED in file event/pmix_event_registration.c at line 101) > > > > > > > > > > > > > > > > > > > > > > > > All the best, > > > > -- > > > > Passant A. Hafez | HPC Applications Specialist > > > > KAUST Supercomputing Core Laboratory (KSL) > > > > King Abdullah University of Science and Technology > > > > Building 1, Al-Khawarizmi, Room 0123 > > > > Mobile : +966 (0) 55-247-9568 > > > > Mobile : +20 (0) 106-146-9644 > > > > Office : +966 (0) 12-808-0367 > > > > > > > > ________________________________________ > > > > From: users <users-boun...@lists.open-mpi.org> on behalf of Gilles > > > > Gouaillardet <gilles.gouaillar...@gmail.com> > > > > Sent: Saturday, February 23, 2019 5:17 PM > > > > To: Open MPI Users > > > > Subject: Re: [OMPI users] Building PMIx and Slurm support > > > > > > > > Hi, > > > > > > > > PMIx has cross-version compatibility, so as long as the PMIx library > > > > used by SLURM is compatible with the one (internal or external) used > > > > by Open MPI, you should be fine. > > > > If you want to minimize the risk of cross-version incompatibility, > > > > then I encourage you to use the same (and hence external) PMIx that > > > > was used to build SLURM with Open MPI. > > > > > > > > Can you tell a bit more than "it didn't work" ? > > > > (Open MPI version, PMIx version used by SLURM, PMIx version used by > > > > Open MPI, error message, ...) > > > > > > > > Cheers, > > > > > > > > Gilles > > > > > > > > On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez > > > > <passant.ha...@kaust.edu.sa> wrote: > > > > > > > > > > > > > > > Good day everyone, > > > > > > > > > > I've trying to build and use the PMIx support for Open MPI but I > > > > > tried many things that I can list if needed, but with no luck. > > > > > I was able to test the PMIx client but when I used OMPI specifying > > > > > srun --mpi=pmix it didn't work. > > > > > > > > > > So if you please advise me with the versions of each PMIx and Open > > > > > MPI that should be working well with Slurm 18.08, it'd be great. > > > > > > > > > > Also, what is the difference between using internal vs external PMIx > > > > > installations? > > > > > > > > > > > > > > > > > > > > All the best, > > > > > > > > > > -- > > > > > > > > > > Passant A. Hafez | HPC Applications Specialist > > > > > KAUST Supercomputing Core Laboratory (KSL) > > > > > King Abdullah University of Science and Technology > > > > > Building 1, Al-Khawarizmi, Room 0123 > > > > > Mobile : +966 (0) 55-247-9568 > > > > > Mobile : +20 (0) 106-146-9644 > > > > > Office : +966 (0) 12-808-0367 > > > > > _______________________________________________ > > > > > users mailing list > > > > > users@lists.open-mpi.org > > > > > https://lists.open-mpi.org/mailman/listinfo/users > > > > _______________________________________________ > > > > users mailing list > > > > users@lists.open-mpi.org > > > > https://lists.open-mpi.org/mailman/listinfo/users > > > > _______________________________________________ > > > > users mailing list > > > > users@lists.open-mpi.org > > > > https://lists.open-mpi.org/mailman/listinfo/users > > > _______________________________________________ > > > users mailing list > > > users@lists.open-mpi.org > > > https://lists.open-mpi.org/mailman/listinfo/users > > > _______________________________________________ > > > users mailing list > > > users@lists.open-mpi.org > > > https://lists.open-mpi.org/mailman/listinfo/users > > _______________________________________________ > > users mailing list > > users@lists.open-mpi.org > > https://lists.open-mpi.org/mailman/listinfo/users _______________________________________________ users mailing list users@lists.open-mpi.org https://lists.open-mpi.org/mailman/listinfo/users