Hej Ole,
To muligheder: 1) Prøv at sætte disse tre environment variable: export OMPI_MCA_osc='^ucx' export OMPI_MCA_pml='^ucx' 'export OMPI_MCA_btl='^openib' 2) Prøv at bygge med eb --filter-deps=UCX OpenMPI-4.0.5-GCC-10.2.0.eb --force og derefter sætte export OMPI_MCA_btl=^openib Mvh Jakob > On 6 Dec 2021, at 15.04, Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk> wrote: > > Hi Bart, > > Thanks for your recommendations! We had already tried this: > > export OMPI_MCA_osc='^ucx' > export OMPI_MCA_pml='^ucx' > > and unfortunately this increased the CPU time of our benchmark code (GPAW) by > about 30% compared to the same compute node without an Omni-Path adapter. So > this doesn't appear to be a viable solution. > > We had also tried to rebuild with: > > $ eb --filter-deps=UCX OpenMPI-4.0.5-GCC-10.2.0.eb --force > > but then the job error log files had some warnings: > >> -------------------------------------------------------------------------- >> By default, for Open MPI 4.0 and later, infiniband ports on a device >> are not used by default. The intent is to use UCX for these devices. >> You can override this policy by setting the btl_openib_allow_ib MCA parameter >> to true. >> Local host: d063 >> Local adapter: hfi1_0 >> Local port: 1 >> -------------------------------------------------------------------------- >> -------------------------------------------------------------------------- >> WARNING: There is at least non-excluded one OpenFabrics device found, >> but there are no active ports detected (or Open MPI was unable to use >> them). This is most certainly not what you wanted. Check your >> cables, subnet manager configuration, etc. The openib BTL will be >> ignored for this job. >> Local host: d063 >> -------------------------------------------------------------------------- >> [d063.nifl.fysik.dtu.dk:23605] 55 more processes have sent help message >> help-mpi-btl-openib.txt / ib port not selected >> [d063.nifl.fysik.dtu.dk:23605] Set MCA parameter "orte_base_help_aggregate" >> to 0 to see all help / error messages >> [d063.nifl.fysik.dtu.dk:23605] 55 more processes have sent help message >> help-mpi-btl-openib.txt / no active ports found > > These warnings did sound rather bad, so we didn't pursue this approach any > further. > > Do you have any other ideas about OMPI_* variables that we could try? Since > I'm not an MPI expert, complete commands and variables would be appreciated > :-) > > I would like to remind you that we're running AlmaLinux 8.5 with new versions > of libfabric etc. from the BaseOS. On CentOS 7.9 we never had any problems > with Omni-Path adapters. > > Thanks, > Ole > > On 12/3/21 15:08, Bart Oldeman wrote: >> Hi Ole, >> we found that UCX isn't very useful not performant on OmniPath, so if your >> compiled isn't used on both InfiniBand and OmniPath you can compile OpenMPI >> using "eb --filter-deps=UCX ..." >> Open MPI works well there either using libpsm2 directly (using the "cm" pml >> and "psm2" mtl), or via libfabric (using the same "cm" pml and the "ofi" mtl) >> We use the same Open MPI binaries on multiple clusters but set this on >> OmniPath: >> OMPI_MCA_btl='^openib' >> OMPI_MCA_osc='^ucx' >> OMPI_MCA_pml='^ucx' >> to disable UCX and openib at runtime. If you include UCX in EB's OpenMPI it >> will not compile in "openib" so the first one of those three would not be >> needed. >> Regards, >> Bart >> On Fri, 3 Dec 2021 at 07:29, Ole Holm Nielsen <ole.h.niel...@fysik.dtu.dk >> <mailto:ole.h.niel...@fysik.dtu.dk>> wrote: >> Hi Åke, >> On 12/3/21 08:27, Åke Sandgren wrote: >> >> On 02-12-2021 14:18, Åke Sandgren wrote: >> >>> On 12/2/21 2:06 PM, Ole Holm Nielsen wrote: >> >>>> These are updated observations of running OpenMPI codes with an >> >>>> Omni-Path network fabric on AlmaLinux 8.5:: >> >>>> >> >>>> Using the foss-2021b toolchain and OpenMPI/4.1.1-GCC-11.2.0 my >> trivial >> >>>> MPI test code works correctly: >> >>>> >> >>>> $ ml OpenMPI >> >>>> $ ml >> >>>> >> >>>> Currently Loaded Modules: >> >>>> 1) GCCcore/11.2.0 9) >> hwloc/2.5.0-GCCcore-11.2.0 >> >>>> 2) zlib/1.2.11-GCCcore-11.2.0 10) OpenSSL/1.1 >> >>>> 3) binutils/2.37-GCCcore-11.2.0 11) >> >>>> libevent/2.1.12-GCCcore-11.2.0 >> >>>> 4) GCC/11.2.0 12) >> UCX/1.11.2-GCCcore-11.2.0 >> >>>> 5) numactl/2.0.14-GCCcore-11.2.0 13) >> >>>> libfabric/1.13.2-GCCcore-11.2.0 >> >>>> 6) XZ/5.2.5-GCCcore-11.2.0 14) >> PMIx/4.1.0-GCCcore-11.2.0 >> >>>> 7) libxml2/2.9.10-GCCcore-11.2.0 15) >> OpenMPI/4.1.1-GCC-11.2.0 >> >>>> 8) libpciaccess/0.16-GCCcore-11.2.0 >> >>>> >> >>>> $ mpicc mpi_test.c >> >>>> $ mpirun -n 2 a.out >> >>>> >> >>>> (null): There are 2 processes >> >>>> >> >>>> (null): Rank 1: d008 >> >>>> >> >>>> (null): Rank 0: d008 >> >>>> >> >>>> >> >>>> I also tried the OpenMPI/4.1.0-GCC-10.2.0 module, but this still >> gives >> >>>> the error messages: >> >>>> >> >>>> $ ml OpenMPI/4.1.0-GCC-10.2.0 >> >>>> $ ml >> >>>> >> >>>> Currently Loaded Modules: >> >>>> 1) GCCcore/10.2.0 3) >> binutils/2.35-GCCcore-10.2.0 5) >> >>>> numactl/2.0.13-GCCcore-10.2.0 7) >> libxml2/2.9.10-GCCcore-10.2.0 9) >> >>>> hwloc/2.2.0-GCCcore-10.2.0 11) >> UCX/1.9.0-GCCcore-10.2.0 13) >> >>>> PMIx/3.1.5-GCCcore-10.2.0 >> >>>> 2) zlib/1.2.11-GCCcore-10.2.0 4) >> GCC/10.2.0 6) >> >>>> XZ/5.2.5-GCCcore-10.2.0 8) >> libpciaccess/0.16-GCCcore-10.2.0 10) >> >>>> libevent/2.1.12-GCCcore-10.2.0 12) >> libfabric/1.11.0-GCCcore-10.2.0 14) >> >>>> OpenMPI/4.1.0-GCC-10.2.0 >> >>>> >> >>>> $ mpicc mpi_test.c >> >>>> $ mpirun -n 2 a.out >> >>>> [1638449983.577933] [d008:910356:0] ib_iface.c:966 UCX ERROR >> >>>> ibv_create_cq(cqe=4096) failed: Operation not supported >> >>>> [1638449983.577827] [d008:910355:0] ib_iface.c:966 UCX ERROR >> >>>> ibv_create_cq(cqe=4096) failed: Operation not supported >> >>>> [d008.nifl.fysik.dtu.dk:910355 >> <http://d008.nifl.fysik.dtu.dk:910355>] pml_ucx.c:273 Error: Failed >> to create >> >>>> UCP worker >> >>>> [d008.nifl.fysik.dtu.dk:910356 >> <http://d008.nifl.fysik.dtu.dk:910356>] pml_ucx.c:273 Error: Failed >> to create >> >>>> UCP worker >> >>>> >> >>>> (null): There are 2 processes >> >>>> >> >>>> (null): Rank 0: d008 >> >>>> >> >>>> (null): Rank 1: d008 >> >>>> >> >>>> Conclusion: The foss-2021b toolchain with >> OpenMPI/4.1.1-GCC-11.2.0 seems >> >>>> to be required on systems with an Omni-Path network fabric on >> AlmaLinux >> >>>> 8.5. Perhaps the newer UCX/1.11.2-GCCcore-11.2.0 is really what's >> >>>> needed, compared to UCX/1.9.0-GCCcore-10.2.0 from foss-2020b. >> >>>> >> >>>> Does anyone have comments on this? >> >>> >> >>> UCX is the problem here in combination with libfabric I think. >> Write a >> >>> hook that upgrades the version of UCX to 1.11-something if it's < >> >>> 1.11-ish, or just that specific version if you have older-and-working >> >>> versions. >> >> >> >> You are right that the nodes with Omni-Path have different libfabric >> >> packages which come from the EL8.5 BaseOS as well as the latest >> >> Cornelis/Intel Omni-Path drivers: >> >> >> >> $ rpm -qa | grep libfabric >> >> libfabric-verbs-1.10.0-2.x86_64 >> >> libfabric-1.12.1-1.el8.x86_64 >> >> libfabric-devel-1.12.1-1.el8.x86_64 >> >> libfabric-psm2-1.10.0-2.x86_64 >> >> >> >> The 1.12 packages are from EL8.5, and 1.10 packages are from Cornelis. >> >> >> >> Regarding UCX, I was first using the trusted foss-2020b toolchain >> which >> >> includes UCX/1.9.0-GCCcore-10.2.0. I guess that we shouldn't mess with >> >> the toolchains? >> >> >> >> The foss-2021b toolchain includes the newer UCX 1.11, which seems to >> >> solve this particular problem. >> >> >> >> Can we make any best practices recommendations from these >> observations? >> > >> > I didn't check properly, but UCX does not depend on libfabric, OpenMPI >> > does, so I'd write a hook that replaces libfabric < 1.12 with at least >> > 1.12.1. >> > Sometimes you just have to mess with the toolchains, and this looks >> like >> > one of those situations. >> > >> > Or as a test build your own OpenMPI-4.1.0 or 4.0.5 (that 2020b uses) >> > with an updated libfabric and check if that fixes the problem. And >> if it >> > does, write a hook that replaces libfabric. See the framework/contrib >> > for examples, I did that for UCX so there is code there to show you >> how. >> I don't feel qualified to mess around with modifying EB toolchains... >> The foss-2021b toolchain including OpenMPI/4.1.1-GCC-11.2.0 seems to >> solve >> the present problem. Do you think there are any disadvantages with >> asking >> users to go for foss-2021b? Of course we may need several modules to be >> upgraded from foss-2020b to foss-2021b. >> Another possibility may be the coming driver upgrade from Cornelis >> Networks to support the Omni-Path fabric on EL 8.4 and EL 8.5. I'm >> definitely going to check this when it becomes available. >> Thanks, >> Ole >> -- >> Dr. Bart E. Oldeman | bart.olde...@mcgill.ca <mailto:bart.olde...@mcgill.ca> >> | bart.olde...@calculquebec.ca <mailto:bart.olde...@calculquebec.ca> >> Scientific Computing Analyst / Analyste en calcul scientifique >> McGill HPC Centre / Centre de Calcul Haute Performance de McGill | >> http://www.hpc.mcgill.ca <http://www.hpc.mcgill.ca> >> Calcul Québec | http://www.calculquebec.ca <http://www.calculquebec.ca> >> Compute/Calcul Canada | http://www.computecanada.ca >> <http://www.computecanada.ca> >> Tel/Tél: 514-396-8926 | Fax/Télécopieur: 514-396-8934 >
