Hi Nathan
thanks for this suggestion. I have understood that now all is managed by
the UCX layer. Am I wrong ?
These options do not seams to work with my openMPI 5.0.5 build. But I've
built OpenMPI on the cluster front-end and it had no HBA at this time.
I've added one this evening (an old spare one I had) and the software to
manage it, may be should I rebuild OpenMPI ?
bash-5.1$ mpirun hostname
kareline-0-0.localcluster.priv
kareline-0-1.localcluster.priv
bash-5.1$ mpirun --mca pml cm --mca mtl psm2 osu_bw
--------------------------------------------------------------------------
A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find.
Host: kareline-0-0
Framework: mtl
Component: psm2
--------------------------------------------------------------------------
--------------------------------------------------------------------------
No components were able to be opened in the pml framework.
This typically means that either no components of this type were
installed, or none of the installed components can be loaded.
Sometimes this means that shared libraries required by these
components are unable to be found/loaded.
Host: kareline-0-0
Framework: pml
--------------------------------------------------------------------------
bash-5.1$ which mpirun
/opt/GCC14/OpenMPI/5.0.5/bin/mpirun
bash-5.1$
Patrick
Le 30/09/2024 à 18:26, Nathan Hjelm a écrit :
If this is a QLogic system why not try psm2 (--mca pml cm --mca mtl
psm2)? Not sure how good UCX support is over these systems and psm2 is
the vendor's library.
Not sure what the right link is to the current version but found this
version:
GitHub - cornelisnetworks/opa-psm2
GitHub - cornelisnetworks/opa-psm2
github.com
<https://github.com/cornelisnetworks/opa-psm2>
-Nathan
On Sep 30, 2024, at 10:18 AM, Patrick Begou via users
<users@lists.open-mpi.org> wrote:
Hi,
I'm working on refreshing an old cluster with Almalinux 9 (instead of
CentOS6 😕) and building a fresh OpenMPI 5.0.5 environment. I've
reached the step where OpenMPI begins to work with ucx 1.17 and Pmix
5.0.3 but not totally. Nodes are using a Qlogic QDR HBA with a
managed Qlogic switch (40Gb/s) and 1Gb/s ethernet and I've a limited
knowledge with all the software stack required now with ucx for this
hardware.
This is the output of osu_bw test between 2 nodes (in slurm context)
bash-5.1$ mpirun --mca pml ucx --mca osc ucx --mca scoll ucx --mca
atomic ucx osu_bw # OSU MPI Bandwidth Test v7.4 # Datatype: MPI_CHAR.
# Size Bandwidth (MB/s) 1 0.30
2 0.59 4 1.16
8 2.33 16 4.78
32 9.46 64 18.80
128 36.21 256 69.61
512 142.48 1024 256.41
2048 498.27 4096 719.19
8192 1010.86 16384 1416.17
32768 1935.44 65536 2509.17
131072 2786.79 262144 2401.26
524288 500.32 1048576 854.12
2097152 3114.28 4194304 1830.78
The options come from
https://docs.open-mpi.org/en/main/tuning-apps/networking/ib-and-roce.html,
without them it uses the slow ethernet 1Gb/s interface. Running the
osu_bibw test is worse as soon as the size of the messages increase
like if some congestion occurs. # OSU MPI Bi-Directional Bandwidth
Test v7.4 # Datatype: MPI_CHAR. # Size Bandwidth (MB/s) 1 0.52 2 1.04
4 2.08 8 4.18 16 8.37 32 16.76 64 33.11 128 65.93 256 130.89 512
248.77 1024 492.23 2048 1024.23 4096 1622.98 8192 2352.29 16384
1724.83 32768 2309.67 65536 2538.13 131072 2586.15 262144 95.93
524288 42.83 1048576 63.14 2097152 78.81 4194304 129.66
1) I've built ucx 1.17.0 with the gcc 11.4 provided by the OS as I
need a thread safe version (suggested by Gilles Gouaillardet when I
was building UCX for OpenMPI 4.04 on another cluster with HDR100 and
have some performances troubles) ../ucx/contrib/configure-release
--enable-mt
2) I've built a fresh version of PMIX 5.0.3 with the gcc 11.4
provided by the OS without specific options: prefix=/usr
build_srpm=yes build_multiple=yes ./buildrpm.sh
../../pmix-5.0.3.tar.bz2 3) slurm is built with PMIX and UCX with the
gcc 11.4 provided by the OS
3) Then I've built OpenMPI with a fresh install of gcc 14.2 (to have
a correct version of the fortran module) Configure command line:
'--enable-mpirun-prefix-by-default'
'--prefix=/opt/GCC14/OpenMPI/5.0.5' '--enable-mpi1-compatibility'
'--with-slurm'
PATH and LD_LIBRARY_PATH are set via the module environment tool.
Using the old deployment of this cluster (same Qlogiq HBA and IB
switch) based on openMPI 3.1.3rc1 with openib and gcc 7.3, it works
fine. Configure command line:
'--prefix=/share/apps/GCC73/openmpi/31-patch'
'--enable-mpirun-prefix-by-default' '--disable-dlopen'
'--enable-mpi-cxx' '--without-slurm' '--enable-mpi-thread-multiple'
# OSU MPI Bi-Directional Bandwidth Test v7.4 # Datatype: MPI_CHAR. #
Size Bandwidth (MB/s) 1 1.93 .... 1048576 6034.23 2097152 6028.31
4194304 6033.63
The basic Almalinux packages deployed to manage the infiniband
network are: - kernel-lt => required for the ib_qib module that is
not available with Almalinux9 - kernel-lt-devel - infiniband-diags -
libibumad - rdma-core - ib_qib-ibverbs
My ucx threadsafe packages deployed: -
ucx-threadsafe-1.17.0-1.el9.x86_64 -
ucx-threadsafe-devel-1.17.0-1.el9.x86_64 -
ucx-threadsafe-ib-1.17.0-1.el9.x86_64 -
ucx-threadsafe-rdmacm-1.17.0-1.el9.x86_64 -
ucx-threadsafe-cma-1.17.0-1.el9.x86_64
May be I'm wrong there too.
Thanks all for your help. Patrick
