On Sun, Mar 3, 2019 at 10:57 PM Daniel Letai <d...@letai.org.il> wrote:
Hello,
I have built the following stack :
centos 7.5 (gcc 4.8.5-28, libevent 2.0.21-4)
MLNX_OFED_LINUX-4.5-1.0.1.0-rhel7.5-x86_64.tgz built with --all
--without-32bit (this includes ucx 1.5.0)
hwloc from centos 7.5 : 1.11.8-4.el7
pmix 3.1.2
slurm 18.08.5-2 built --with-ucx --with-pmix
openmpi 4.0.0 : configure --with-slurm --with-pmix=external
--with-pmi --with-libevent=external --with-hwloc=external
--with-knem=/opt/knem-1.1.3.90mlnx1 --with-hcoll=/opt/mellanox/hcoll
The configure part succeeds, however 'make' errors out with:
ext3x.c: In function 'ext3x_value_unload':
ext3x.c:1109:10: error: 'PMIX_MODEX' undeclared (first use in this
function)
And same for 'PMIX_INFO_ARRAY'
However, both are declared in the
opal/mca/pmix/pmix3x/pmix/include/pmix_common.h file.
opal/mca/pmix/ext3x/ext3x.c does include pmix_common.h but as a
system include #include <pmix_common> , while ext3x.h includes it as
a local include #include "pmix_common". Neither seem to pull from
the correct path.
Regards,
Dani_L.
On 2/24/19 3:09 AM, Gilles Gouaillardet wrote:
Passant,
you have to manually download and apply
https://github.com/pmix/pmix/commit/2e2f4445b45eac5a3fcbd409c81efe318876e659.patch
to PMIx 2.2.1
that should likely fix your problem.
As a side note, it is a bad practice to configure --with-FOO=/usr
since it might have some unexpected side effects.
Instead, you can replace
configure --with-slurm --with-pmix=/usr --with-pmi=/usr
--with-libevent=/usr
with
configure --with-slurm --with-pmix=external --with-pmi
--with-libevent=external
to be on the safe side I also invite you to pass --with-hwloc=external
to the configure command line
Cheers,
Gilles
On Sun, Feb 24, 2019 at 1:54 AM Passant A. Hafez
<passant.ha...@kaust.edu.sa> wrote:
Hello Gilles,
Here are some details:
Slurm 18.08.4
PMIx 2.2.1 (as shown in /usr/include/pmix_version.h)
Libevent 2.0.21
srun --mpi=list
srun: MPI types are...
srun: none
srun: openmpi
srun: pmi2
srun: pmix
srun: pmix_v2
Open MPI versions tested: 4.0.0 and 3.1.2
For each installation to be mentioned a different MPI Hello World
program was compiled.
Jobs were submitted by sbatch, 2 node * 2 tasks per node then srun
--mpi=pmix program
File 400ext_2x2.out (attached) is for OMPI 4.0.0 installation with
configure options:
--with-slurm --with-pmix=/usr --with-pmi=/usr --with-libevent=/usr
and configure log:
Libevent support: external
PMIx support: External (2x)
File 400int_2x2.out (attached) is for OMPI 4.0.0 installation with
configure options:
--with-slurm --with-pmix
and configure log:
Libevent support: internal (external libevent version is less that
internal version 2.0.22)
PMIx support: Internal
Tested also different installations for 3.1.2 and got errors similar
to 400ext_2x2.out
(NOT-SUPPORTED in file event/pmix_event_registration.c at line 101)
All the best,
--
Passant A. Hafez | HPC Applications Specialist
KAUST Supercomputing Core Laboratory (KSL)
King Abdullah University of Science and Technology
Building 1, Al-Khawarizmi, Room 0123
Mobile : +966 (0) 55-247-9568
Mobile : +20 (0) 106-146-9644
Office : +966 (0) 12-808-0367
________________________________________
From: users <users-boun...@lists.open-mpi.org> on behalf of Gilles
Gouaillardet <gilles.gouaillar...@gmail.com>
Sent: Saturday, February 23, 2019 5:17 PM
To: Open MPI Users
Subject: Re: [OMPI users] Building PMIx and Slurm support
Hi,
PMIx has cross-version compatibility, so as long as the PMIx library
used by SLURM is compatible with the one (internal or external) used
by Open MPI, you should be fine.
If you want to minimize the risk of cross-version incompatibility,
then I encourage you to use the same (and hence external) PMIx that
was used to build SLURM with Open MPI.
Can you tell a bit more than "it didn't work" ?
(Open MPI version, PMIx version used by SLURM, PMIx version used by
Open MPI, error message, ...)
Cheers,
Gilles
On Sat, Feb 23, 2019 at 9:46 PM Passant A. Hafez
<passant.ha...@kaust.edu.sa> wrote:
Good day everyone,
I've trying to build and use the PMIx support for Open MPI but I
tried many things that I can list if needed, but with no luck.
I was able to test the PMIx client but when I used OMPI specifying
srun --mpi=pmix it didn't work.
So if you please advise me with the versions of each PMIx and Open
MPI that should be working well with Slurm 18.08, it'd be great.
Also, what is the difference between using internal vs external PMIx
installations?
All the best,
--
Passant A. Hafez | HPC Applications Specialist
KAUST Supercomputing Core Laboratory (KSL)
King Abdullah University of Science and Technology
Building 1, Al-Khawarizmi, Room 0123
Mobile : +966 (0) 55-247-9568
Mobile : +20 (0) 106-146-9644
Office : +966 (0) 12-808-0367
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://lists.open-mpi.org/mailman/listinfo/users
