Hello Gilles,thanks for your response. I'm testing with 20 task, each using 8 threads. When using a single node or only few nodes, we do not see this either.
Attached is the used slurm script, which reports also the environment variables, and the output log from three different runs with srun, mpirun, and mpirun --mca ....
It is correct, that when running with mpirun we do not see this issue. The errors are only observed when running with "srun"
Moreover, I notice that fewer tests are performed when using mpirun.From that we can conclude that the issue is related to slurm-openmpi interaction.
Switching from srun to mpirun has also some negative implications w.r.t. to scheduling and also robustness. Therefore, we would like to start the job with srun.
Cheers, Alois Am 5/3/22 um 12:52 schrieb Gilles Gouaillardet via users:
Alois, Thanks for the report. FWIW, I am not seeing any errors on my Mac with Open MPI from brew (4.1.3) How many MPI tasks are you running? Can you please confirm you can evidence the error withmpirun -np <number_of_processes> ./mpi_test_suite -d MPI_TYPE_MIX_ARRAY -c 0 -t collectiveAlso, can you try the same command with mpirun --mca pml ob1 --mca btl tcp,self ... Cheers, GillesOn Tue, May 3, 2022 at 7:08 PM Alois Schlögl via users <users@lists.open-mpi.org> wrote:Within our cluster (debian10/slurm16, debian11/slurm20), with infiniband, and we have several instances of openmpi installed through the Lmod module system. When testing the openmpi installations with the mpi-test-suite 1.1 [1], it shows errors like these ... Rank:0) tst_test_array[45]:Allreduce Min/Max with MPI_IN_PLACE (Rank:0) tst_test_array[46]:Allreduce Sum (Rank:0) tst_test_array[47]:Alltoall Number of failed tests: 130 Summary of failed tests: ERROR class:P2P test:Ring Send Pack (7), comm Duplicated MPI_COMM_WORLD (4), type MPI_TYPE_MIX (27) number of values:1000 ERROR class:P2P test:Ring Send Pack (7), comm Duplicated MPI_COMM_WORLD (4), type MPI_TYPE_MIX_ARRAY (28) number of values:1000 ... when using openmpi/4.1.x (i tested with 4.1.1 and 4.1.3) The number of errors may vary, but the first errors are always about ERROR class:P2P test:Ring Send Pack (7), comm Duplicated MPI_COMM_WORLD When testing on openmpi/3.1.3, the tests runs successfully, and there are no failed tests. Typically, the openmpi/4.1.x installation is configured with ./configure --prefix=${PREFIX} \ --with-ucx=$UCX_HOME \ --enable-orterun-prefix-by-default \ --enable-mpi-cxx \ --with-hwloc \ --with-pmi \ --with-pmix \ --with-cuda=$CUDA_HOME \ --with-slurm but I've also tried different compilation options including w/ and w/o --enable-mpi1-compatibility, w/ and w/o ucx, using hwloc from the OS, or compiled from source. But I could not identify any pattern. Therefore, I'd like asking you what the issue might be. Specifically, I'm would like to know: - Am I right in assuming that mpi-test-suite [1] suitable for testing openmpi ? - what are possible causes for these type of errors ? - what would you recommend how to debug these issues ? Kind regards, Alois [1] https://github.com/open-mpi/mpi-test-suite/t
job-mpi-test3.sh
Description: application/shellscript
delta197
/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/ompi_info
running on 20*8 cores with 20 MPI-tasks and 8 threads
SHELL=/bin/bash
SLURM_JOB_USER=schloegl
SLURM_TASKS_PER_NODE=2(x10)
SLURM_JOB_UID=10103
SLURM_TASK_PID=50793
PKG_CONFIG_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib/pkgconfig:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib/pkgconfig:/mnt/nfs/clustersw/shared/cuda/11.2.2/pkgconfig
SLURM_LOCALID=0
SLURM_SUBMIT_DIR=/nfs/scistore16/jonasgrp/schloegl/slurm
HOSTNAME=delta197
LANGUAGE=en_US:en
SLURMD_NODENAME=delta197
_ModuleTable002_=ewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvaHdsb2MvMi43LjEubHVhIiwKZnVsbE5hbWUgPSAiaHdsb2MvMi43LjEiLApsb2FkT3JkZXIgPSAzLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImh3bG9jLzIuNy4xIiwKd1YgPSAiMDAwMDAwMDAyLjAwMDAwMDAwNy4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm9wZW5tcGkgPSB7CmZuID0gIi9tbnQvbmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvQ29yZS9vcGVubXBpLzQuMS4zZC5sdWEiLApmdWxsTmFtZSA9ICJvcGVubXBpLzQuMS4zZCIsCmxvYWRPcmRlciA9IDQsCnByb3BUID0g
MPICC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpicc
__LMOD_REF_COUNT_MODULEPATH=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/MPI/openmpi/4.1.3d:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Linux:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core:1;/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/modulefiles/Core:1
OMPI_MCA_btl=self,openib
_ModuleTable005_=c3cvRGViaWFuL2J1bGxzZXllL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIiwKfQo=
__LMOD_REF_COUNT_LD_RUN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1
SLURM_NODE_ALIASES=(null)
SLURM_CLUSTER_NAME=istscicomp
OMPI_MCA_btl_openib_allow_ib=1
SLURM_CPUS_ON_NODE=16
__LMOD_REF_COUNT__LMFILES_=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/ucx/1.12.1.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/cuda/11.2.2.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/hwloc/2.7.1.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/openmpi/4.1.3d.lua:1
SLURM_JOB_CPUS_PER_NODE=16(x10)
MPICXX=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpicxx
LMOD_DIR=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/libexec
MPIFC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpifort
PWD=/nfs/scistore16/jonasgrp/schloegl/slurm
SLURM_GTIDS=0
LOGNAME=schloegl
UCX_NET_DEVICES=ibp59s0
XDG_SESSION_TYPE=unspecified
SLURM_JOB_PARTITION=defaultp
MODULESHOME=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod
MANPATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/share/man:/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/share/man::
NUM_CORES=20*8
SLURM_JOB_NUM_NODES=10
OMPI_MCA_pml=ob1
OPENBLAS_NUM_THREADS=8
SLURM_JOBID=33082
SLURM_JOB_QOS=normal
MPI_HOME=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d
__LMOD_REF_COUNT_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/sbin:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/bin:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/bin:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/bin:1;/nfs/scistore16/jonasgrp/schloegl/bin:1;/usr/local/bin:1;/usr/bin:1;/bin:1;/usr/local/games:1;/usr/games:1
HOME=/nfs/scistore16/jonasgrp/schloegl
_ModuleTable_Sz_=5
__LMOD_REF_COUNT_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1
LANG=en_US.UTF-8
__LMOD_REF_COUNT_LOADEDMODULES=ucx/1.12.1:1;cuda/11.2.2:1;hwloc/2.7.1:1;openmpi/4.1.3d:1
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib/pkgconfig:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib/pkgconfig:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/pkgconfig:1
SLURM_PROCID=0
CUDA_DEVICE_ORDER=PCI_BUS_ID
MAN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/share/man:/mnt/nfs/clustersw/shared/cuda/11.2.2/share/man
__LMOD_REF_COUNT_CPATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/include:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/include:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/include:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/include:1
LMOD_SETTARG_FULL_SUPPORT=no
UCX_HOME=/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1
TMPDIR=/tmp
SLURM_CPUS_PER_TASK=8
SLURM_NTASKS=20
SLURM_TOPOLOGY_ADDR=sw11.delta197
LMOD_VERSION=8.6.12
_ModuleTable003_=e30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAib3Blbm1waS80LjEuM2QiLAp3ViA9ICIwMDAwMDAwMDQuMDAwMDAwMDAxLjAwMDAwMDAwMy4qZC4qemZpbmFsIiwKfSwKdWN4ID0gewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvdWN4LzEuMTIuMS5sdWEiLApmdWxsTmFtZSA9ICJ1Y3gvMS4xMi4xIiwKbG9hZE9yZGVyID0gMSwKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJ1Y3gvMS4xMi4xIiwKd1YgPSAiMDAwMDAwMDAxLjAwMDAwMDAxMi4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCn0sCm1wYXRoQSA9IHsKIi9tbnQv
MODULEPATH_ROOT=/mnt/nfs/clustersw/Debian/bullseye/modulefiles
SLURM_TOPOLOGY_ADDR_PATTERN=switch.node
XDG_SESSION_CLASS=background
LMOD_PKG=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod
SLURM_MEM_PER_NODE=2048
OMPI_MCA_opal_warn_on_missing_libcuda=0
SLURM_WORKING_CLUSTER=istscicomp:10.36.192.126:6817:9216:101
USER=schloegl
LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib
SLURM_NODELIST=delta[197-206]
OMP_MCA_mpi_param_check=YES
ENVIRONMENT=BATCH
LOADEDMODULES=ucx/1.12.1:cuda/11.2.2:hwloc/2.7.1:openmpi/4.1.3d
SLURM_JOB_ACCOUNT=itgrp
SLURM_PRIO_PROCESS=0
SLURM_NPROCS=20
LMOD_ROOT=/mnt/nfs/clustersw/Debian/bullseye/lmod
SHLVL=2
SLURM_NNODES=10
LD_RUN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib
BASH_ENV=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/init/bash
__LMOD_REF_COUNT_MAN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/share/man:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/share/man:1
LMOD_sys=Linux
DISTRIB_ID=Debian
__LMOD_REF_COUNT_MANPATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/share/man:1;/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/share/man:1
XDG_SESSION_ID=c36
SLURM_SUBMIT_HOST=gpu114
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpNUEkgPSAib3Blbm1waSIsCn0sCm1UID0gewpjdWRhID0gewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvY3VkYS8xMS4yLjIubHVhIiwKZnVsbE5hbWUgPSAiY3VkYS8xMS4yLjIiLApsb2FkT3JkZXIgPSAyLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImN1ZGEvMTEuMi4yIiwKd1YgPSAiXjAwMDAwMDExLjAwMDAwMDAwMi4wMDAwMDAwMDIuKnpmaW5hbCIsCn0sCmh3bG9jID0g
MPIEXEC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpiexec
MPIRUN=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpirun
DISTRIB_RELEASE=11
LD_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/extras/CUPTI/lib64:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/lib64
XDG_RUNTIME_DIR=/run/user/10103
SLURM_JOB_ID=33082
SLURM_NODEID=0
_ModuleTable004_=bmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvTVBJL29wZW5tcGkvNC4xLjNkIgosICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0xpbnV4IgosICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUiLCAiL21udC9uZnMvY2x1c3RlcnN3L0RlYmlhbi9idWxsc2V5ZS9sbW9kL2xtb2QvbW9kdWxlZmlsZXMvQ29yZSIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0xpbnV4Oi9tbnQvbmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvQ29yZTovbW50L25mcy9jbHVzdGVy
LMOD_FAMILY_MPI_VERSION=4.1.3d
OMP_NUM_THREADS=8
LMOD_FAMILY_MPI=openmpi
CUDA_HOME=/mnt/nfs/clustersw/shared/cuda/11.2.2
OMPI_MCA_mpi_cuda_support=1
SLURM_MPI_TYPE=pmix
__LMOD_REF_COUNT_C_INCLUDE_PATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:1
DISTRIB_CODENAME=bullseye
SLURM_CONF=/etc/slurm/slurm.conf
PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/sbin:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/bin:/mnt/nfs/clustersw/shared/cuda/11.2.2/bin:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/bin:/nfs/scistore16/jonasgrp/schloegl/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
SLURM_JOB_NAME=mpitest
MODULEPATH=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/MPI/openmpi/4.1.3d:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Linux:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core:/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/modulefiles/Core
SLURM_NTASKS_PER_NODE=2
_LMFILES_=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/ucx/1.12.1.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/cuda/11.2.2.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/hwloc/2.7.1.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/openmpi/4.1.3d.lua
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/10103/bus
LMOD_CMD=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/libexec/lmod
C_INCLUDE_PATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include
MKL_NUM_THREADS=8
MAIL=/var/mail/schloegl
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/extras/CUPTI/lib64:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/lib64:1
SLURM_JOB_GID=11114
SLURM_GET_USER_ENV=1
CPATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/include:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/include:/mnt/nfs/clustersw/shared/cuda/11.2.2/include:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/include
DISTRIB_DESCRIPTION=Debian GNU/Linux 11 (bullseye)
SLURM_JOB_NODELIST=delta[197-206]
BASH_FUNC_ml%%=() { eval $($LMOD_DIR/ml_cmd "$@")
}
BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
case "$-" in
*v*x*)
__lmod_sh_dbg='vx'
;;
*v*)
__lmod_sh_dbg='v'
;;
*x*)
__lmod_sh_dbg='x'
;;
esac;
fi;
if [ -n "${__lmod_sh_dbg:-}" ]; then
set +$__lmod_sh_dbg;
echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's
output" 1>&2;
fi;
_mlshopt="f";
case "$-" in
*f*)
unset _mlshopt
;;
esac;
if [ -n "${_mlshopt:-}" ]; then
set -$_mlshopt;
fi;
eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh);
__lmod_my_status=$?;
if [ -n "${_mlshopt:-}" ]; then
set +$_mlshopt;
fi;
unset _mlshopt;
if [ -n "${__lmod_sh_dbg:-}" ]; then
echo "Shell debugging restarted" 1>&2;
set -$__lmod_sh_dbg;
unset __lmod_sh_dbg;
fi;
return $__lmod_my_status
}
_=/usr/bin/env
==== SRUN
(Rank:0) tst_test_array[0]:Status
(Rank:0) tst_test_array[1]:Request_Null
(Rank:0) tst_test_array[2]:Type_dup
(Rank:0) tst_test_array[3]:Get_version
(Rank:0) tst_test_array[4]:Ring
(Rank:0) tst_test_array[5]:Ring Send Bottom
(Rank:0) tst_test_array[6]:Ring Send Pack
(Rank:0) tst_test_array[7]:Ring Isend
(Rank:0) tst_test_array[8]:Ring Ibsend
(Rank:0) tst_test_array[9]:Ring Irsend
(Rank:0) tst_test_array[10]:Ring Issend
(Rank:0) tst_test_array[11]:Ring Bsend
(Rank:0) tst_test_array[12]:Ring Rsend
(Rank:0) tst_test_array[13]:Ring Ssend
(Rank:0) tst_test_array[14]:Ring Sendrecv
(Rank:0) tst_test_array[15]:Ring same value
(Rank:0) tst_test_array[16]:Ring Persistent
(Rank:0) tst_test_array[17]:Direct Partner Intercomm
(Rank:0) tst_test_array[18]:Many-to-one
(Rank:0) tst_test_array[19]:Many-to-one with MPI_Probe (MPI_ANY_SOURCE)
(Rank:0) tst_test_array[20]:Many-to-one with MPI_Iprobe (MPI_ANY_SOURCE)
(Rank:0) tst_test_array[21]:Many-to-one with Isend and Cancellation
(Rank:0) tst_test_array[22]:Alltoall
(Rank:0) tst_test_array[23]:Alltoall Persistent
(Rank:0) tst_test_array[24]:Alltoall xIsend
(Rank:0) tst_test_array[25]:Alltoall Irsend
(Rank:0) tst_test_array[26]:Alltoall Issend
(Rank:0) tst_test_array[27]:Alltoall with MPI_Probe (MPI_ANY_SOURCE)
(Rank:0) tst_test_array[28]:Ring Send with cart comm
(Rank:0) tst_test_array[29]:Alltoall with topo comm
(Rank:0) tst_test_array[30]:Bcast
(Rank:0) tst_test_array[31]:Gather
(Rank:0) tst_test_array[32]:Allgather
(Rank:0) tst_test_array[33]:Allgather with MPI_IN_PLACE
(Rank:0) tst_test_array[34]:Scan sum
(Rank:0) tst_test_array[35]:Scatter
(Rank:0) tst_test_array[36]:Scatterv
(Rank:0) tst_test_array[37]:Scatterv with stride
(Rank:0) tst_test_array[38]:Reduce Min
(Rank:0) tst_test_array[39]:Reduce Max
(Rank:0) tst_test_array[40]:Reduce Min with MPI_IN_PLACE
(Rank:0) tst_test_array[41]:Reduce Max with MPI_IN_PLACE
(Rank:0) tst_test_array[42]:Allreduce Min
(Rank:0) tst_test_array[43]:Allreduce Max
(Rank:0) tst_test_array[44]:Allreduce Min/Max
(Rank:0) tst_test_array[45]:Allreduce Min/Max with MPI_IN_PLACE
(Rank:0) tst_test_array[46]:Allreduce Sum
(Rank:0) tst_test_array[47]:Alltoall
Number of failed tests: 106
Summary of failed tests:
ERROR class:P2P test:Ring Send Pack (7), comm Duplicated MPI_COMM_WORLD (4),
type MPI_TYPE_MIX (27) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Duplicated MPI_COMM_WORLD (4),
type MPI_TYPE_MIX_ARRAY (28) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Reversed MPI_COMM_WORLD (5), type
MPI_TYPE_MIX (27) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Reversed MPI_COMM_WORLD (5), type
MPI_TYPE_MIX_ARRAY (28) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Halved MPI_COMM_WORLD (6), type
MPI_TYPE_MIX (27) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Halved MPI_COMM_WORLD (6), type
MPI_TYPE_MIX_ARRAY (28) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Odd/Even split MPI_COMM_WORLD
(9), type MPI_TYPE_MIX (27) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Odd/Even split MPI_COMM_WORLD
(9), type MPI_TYPE_MIX_ARRAY (28) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_TYPE_MIX (27) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_TYPE_MIX_ARRAY (28) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm MPI_COMM_TYPE_SHARED comm (13),
type MPI_TYPE_MIX (27) number of values:1000
ERROR class:P2P test:Ring Send Pack (7), comm MPI_COMM_TYPE_SHARED comm (13),
type MPI_TYPE_MIX_ARRAY (28) number of values:1000
ERROR class:P2P test:Alltoall (23), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall (23), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall (23), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall (23), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall (23), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall (23), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm MPI_COMM_WORLD (1), type
MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm MPI_COMM_WORLD (1), type
MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm MPI_COMM_WORLD (1), type
MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm MPI_COMM_WORLD (1), type
MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm MPI_COMM_WORLD (1), type
MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm MPI_COMM_WORLD (1), type
MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Duplicated MPI_COMM_WORLD
(4), type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Duplicated MPI_COMM_WORLD
(4), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Duplicated MPI_COMM_WORLD
(4), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Duplicated MPI_COMM_WORLD
(4), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Duplicated MPI_COMM_WORLD
(4), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Duplicated MPI_COMM_WORLD
(4), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Reversed MPI_COMM_WORLD
(5), type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Reversed MPI_COMM_WORLD
(5), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Reversed MPI_COMM_WORLD
(5), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Reversed MPI_COMM_WORLD
(5), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Reversed MPI_COMM_WORLD
(5), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Reversed MPI_COMM_WORLD
(5), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Intracomm merged of the
Halved Inter_communicato (12), type MPI_CONTIGUOUS_INT (21) number of
values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Intracomm merged of the
Halved Inter_communicato (12), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Intracomm merged of the
Halved Inter_communicato (12), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Intracomm merged of the
Halved Inter_communicato (12), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Intracomm merged of the
Halved Inter_communicato (12), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Persistent (24), comm Intracomm merged of the
Halved Inter_communicato (12), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm MPI_COMM_WORLD (1), type
MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm MPI_COMM_WORLD (1), type
MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm MPI_COMM_WORLD (1), type
MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm MPI_COMM_WORLD (1), type
MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm MPI_COMM_WORLD (1), type
MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm MPI_COMM_WORLD (1), type
MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Duplicated MPI_COMM_WORLD (4),
type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Duplicated MPI_COMM_WORLD (4),
type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Duplicated MPI_COMM_WORLD (4),
type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Duplicated MPI_COMM_WORLD (4),
type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Duplicated MPI_COMM_WORLD (4),
type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Duplicated MPI_COMM_WORLD (4),
type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Reversed MPI_COMM_WORLD (5),
type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Reversed MPI_COMM_WORLD (5),
type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Reversed MPI_COMM_WORLD (5),
type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Reversed MPI_COMM_WORLD (5),
type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Reversed MPI_COMM_WORLD (5),
type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Reversed MPI_COMM_WORLD (5),
type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Irsend (26), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm MPI_COMM_WORLD (1), type
MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm MPI_COMM_WORLD (1), type
MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm MPI_COMM_WORLD (1), type
MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm MPI_COMM_WORLD (1), type
MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm MPI_COMM_WORLD (1), type
MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm MPI_COMM_WORLD (1), type
MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Duplicated MPI_COMM_WORLD (4),
type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Duplicated MPI_COMM_WORLD (4),
type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Duplicated MPI_COMM_WORLD (4),
type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Duplicated MPI_COMM_WORLD (4),
type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Duplicated MPI_COMM_WORLD (4),
type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Duplicated MPI_COMM_WORLD (4),
type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Reversed MPI_COMM_WORLD (5),
type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Reversed MPI_COMM_WORLD (5),
type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Reversed MPI_COMM_WORLD (5),
type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Reversed MPI_COMM_WORLD (5),
type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Reversed MPI_COMM_WORLD (5),
type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Reversed MPI_COMM_WORLD (5),
type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Halved Inter_communicator (11),
type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Halved Inter_communicator (11),
type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Halved Inter_communicator (11),
type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Halved Inter_communicator (11),
type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Halved Inter_communicator (11),
type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Halved Inter_communicator (11),
type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall Issend (27), comm Intracomm merged of the Halved
Inter_communicato (12), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:P2P test:Alltoall with topo comm (30), comm Full-connected Topology
(10), type MPI_CONTIGUOUS_INT (21) number of values:1000
ERROR class:P2P test:Alltoall with topo comm (30), comm Full-connected Topology
(10), type MPI_VECTOR_INT (22) number of values:1000
ERROR class:P2P test:Alltoall with topo comm (30), comm Full-connected Topology
(10), type MPI_HVECTOR_INT (23) number of values:1000
ERROR class:P2P test:Alltoall with topo comm (30), comm Full-connected Topology
(10), type MPI_INDEXED_INT (24) number of values:1000
ERROR class:P2P test:Alltoall with topo comm (30), comm Full-connected Topology
(10), type MPI_HINDEXED_INT (25) number of values:1000
ERROR class:P2P test:Alltoall with topo comm (30), comm Full-connected Topology
(10), type MPI_STRUCT_INT (26) number of values:1000
ERROR class:Collective test:Scatter (36), comm MPI_COMM_WORLD (1), type
MPI_SHORT (5) number of values:1000
ERROR class:Collective test:Scatter (36), comm MPI_COMM_WORLD (1), type
MPI_UNSIGNED_SHORT (6) number of values:1000
ERROR class:Collective test:Scatter (36), comm Duplicated MPI_COMM_WORLD (4),
type MPI_SHORT (5) number of values:1000
ERROR class:Collective test:Scatter (36), comm Duplicated MPI_COMM_WORLD (4),
type MPI_UNSIGNED_SHORT (6) number of values:1000
delta197
/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/ompi_info
running on 20*8 cores with 20 MPI-tasks and 8 threads
SHELL=/bin/bash
SLURM_JOB_USER=schloegl
SLURM_TASKS_PER_NODE=2(x10)
SLURM_JOB_UID=10103
SLURM_TASK_PID=50792
PKG_CONFIG_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib/pkgconfig:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib/pkgconfig:/mnt/nfs/clustersw/shared/cuda/11.2.2/pkgconfig
SLURM_LOCALID=0
SLURM_SUBMIT_DIR=/nfs/scistore16/jonasgrp/schloegl/slurm
HOSTNAME=delta197
LANGUAGE=en_US:en
SLURMD_NODENAME=delta197
_ModuleTable002_=ewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvaHdsb2MvMi43LjEubHVhIiwKZnVsbE5hbWUgPSAiaHdsb2MvMi43LjEiLApsb2FkT3JkZXIgPSAzLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImh3bG9jLzIuNy4xIiwKd1YgPSAiMDAwMDAwMDAyLjAwMDAwMDAwNy4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm9wZW5tcGkgPSB7CmZuID0gIi9tbnQvbmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvQ29yZS9vcGVubXBpLzQuMS4zZC5sdWEiLApmdWxsTmFtZSA9ICJvcGVubXBpLzQuMS4zZCIsCmxvYWRPcmRlciA9IDQsCnByb3BUID0g
MPICC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpicc
__LMOD_REF_COUNT_MODULEPATH=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/MPI/openmpi/4.1.3d:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Linux:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core:1;/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/modulefiles/Core:1
OMPI_MCA_btl=self,openib
_ModuleTable005_=c3cvRGViaWFuL2J1bGxzZXllL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIiwKfQo=
__LMOD_REF_COUNT_LD_RUN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1
SLURM_NODE_ALIASES=(null)
SLURM_CLUSTER_NAME=istscicomp
OMPI_MCA_btl_openib_allow_ib=1
SLURM_CPUS_ON_NODE=16
__LMOD_REF_COUNT__LMFILES_=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/ucx/1.12.1.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/cuda/11.2.2.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/hwloc/2.7.1.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/openmpi/4.1.3d.lua:1
SLURM_JOB_CPUS_PER_NODE=16(x10)
MPICXX=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpicxx
LMOD_DIR=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/libexec
MPIFC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpifort
PWD=/nfs/scistore16/jonasgrp/schloegl/slurm
SLURM_GTIDS=0
LOGNAME=schloegl
UCX_NET_DEVICES=ibp59s0
XDG_SESSION_TYPE=unspecified
SLURM_JOB_PARTITION=defaultp
MODULESHOME=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod
MANPATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/share/man:/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/share/man::
NUM_CORES=20*8
SLURM_JOB_NUM_NODES=10
OMPI_MCA_pml=ob1
OPENBLAS_NUM_THREADS=8
SLURM_JOBID=33083
SLURM_JOB_QOS=normal
MPI_HOME=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d
__LMOD_REF_COUNT_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/sbin:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/bin:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/bin:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/bin:1;/nfs/scistore16/jonasgrp/schloegl/bin:1;/usr/local/bin:1;/usr/bin:1;/bin:1;/usr/local/games:1;/usr/games:1
HOME=/nfs/scistore16/jonasgrp/schloegl
_ModuleTable_Sz_=5
__LMOD_REF_COUNT_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1
LANG=en_US.UTF-8
__LMOD_REF_COUNT_LOADEDMODULES=ucx/1.12.1:1;cuda/11.2.2:1;hwloc/2.7.1:1;openmpi/4.1.3d:1
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib/pkgconfig:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib/pkgconfig:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/pkgconfig:1
SLURM_PROCID=0
CUDA_DEVICE_ORDER=PCI_BUS_ID
MAN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/share/man:/mnt/nfs/clustersw/shared/cuda/11.2.2/share/man
__LMOD_REF_COUNT_CPATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/include:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/include:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/include:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/include:1
LMOD_SETTARG_FULL_SUPPORT=no
UCX_HOME=/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1
TMPDIR=/tmp
SLURM_CPUS_PER_TASK=8
SLURM_NTASKS=20
SLURM_TOPOLOGY_ADDR=sw11.delta197
LMOD_VERSION=8.6.12
_ModuleTable003_=e30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAib3Blbm1waS80LjEuM2QiLAp3ViA9ICIwMDAwMDAwMDQuMDAwMDAwMDAxLjAwMDAwMDAwMy4qZC4qemZpbmFsIiwKfSwKdWN4ID0gewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvdWN4LzEuMTIuMS5sdWEiLApmdWxsTmFtZSA9ICJ1Y3gvMS4xMi4xIiwKbG9hZE9yZGVyID0gMSwKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJ1Y3gvMS4xMi4xIiwKd1YgPSAiMDAwMDAwMDAxLjAwMDAwMDAxMi4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCn0sCm1wYXRoQSA9IHsKIi9tbnQv
MODULEPATH_ROOT=/mnt/nfs/clustersw/Debian/bullseye/modulefiles
SLURM_TOPOLOGY_ADDR_PATTERN=switch.node
XDG_SESSION_CLASS=background
LMOD_PKG=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod
SLURM_MEM_PER_NODE=2048
OMPI_MCA_opal_warn_on_missing_libcuda=0
SLURM_WORKING_CLUSTER=istscicomp:10.36.192.126:6817:9216:101
USER=schloegl
LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib
SLURM_NODELIST=delta[197-206]
OMP_MCA_mpi_param_check=YES
ENVIRONMENT=BATCH
LOADEDMODULES=ucx/1.12.1:cuda/11.2.2:hwloc/2.7.1:openmpi/4.1.3d
SLURM_JOB_ACCOUNT=itgrp
SLURM_PRIO_PROCESS=0
SLURM_NPROCS=20
LMOD_ROOT=/mnt/nfs/clustersw/Debian/bullseye/lmod
SHLVL=2
SLURM_NNODES=10
LD_RUN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib
BASH_ENV=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/init/bash
__LMOD_REF_COUNT_MAN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/share/man:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/share/man:1
LMOD_sys=Linux
DISTRIB_ID=Debian
__LMOD_REF_COUNT_MANPATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/share/man:1;/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/share/man:1
XDG_SESSION_ID=c37
SLURM_SUBMIT_HOST=gpu114
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpNUEkgPSAib3Blbm1waSIsCn0sCm1UID0gewpjdWRhID0gewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvY3VkYS8xMS4yLjIubHVhIiwKZnVsbE5hbWUgPSAiY3VkYS8xMS4yLjIiLApsb2FkT3JkZXIgPSAyLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImN1ZGEvMTEuMi4yIiwKd1YgPSAiXjAwMDAwMDExLjAwMDAwMDAwMi4wMDAwMDAwMDIuKnpmaW5hbCIsCn0sCmh3bG9jID0g
MPIEXEC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpiexec
MPIRUN=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpirun
DISTRIB_RELEASE=11
LD_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/extras/CUPTI/lib64:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/lib64
XDG_RUNTIME_DIR=/run/user/10103
SLURM_JOB_ID=33083
SLURM_NODEID=0
_ModuleTable004_=bmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvTVBJL29wZW5tcGkvNC4xLjNkIgosICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0xpbnV4IgosICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUiLCAiL21udC9uZnMvY2x1c3RlcnN3L0RlYmlhbi9idWxsc2V5ZS9sbW9kL2xtb2QvbW9kdWxlZmlsZXMvQ29yZSIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0xpbnV4Oi9tbnQvbmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvQ29yZTovbW50L25mcy9jbHVzdGVy
LMOD_FAMILY_MPI_VERSION=4.1.3d
OMP_NUM_THREADS=8
LMOD_FAMILY_MPI=openmpi
CUDA_HOME=/mnt/nfs/clustersw/shared/cuda/11.2.2
OMPI_MCA_mpi_cuda_support=1
SLURM_MPI_TYPE=pmix
__LMOD_REF_COUNT_C_INCLUDE_PATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:1
DISTRIB_CODENAME=bullseye
SLURM_CONF=/etc/slurm/slurm.conf
PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/sbin:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/bin:/mnt/nfs/clustersw/shared/cuda/11.2.2/bin:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/bin:/nfs/scistore16/jonasgrp/schloegl/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
SLURM_JOB_NAME=mpitest
MODULEPATH=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/MPI/openmpi/4.1.3d:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Linux:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core:/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/modulefiles/Core
SLURM_NTASKS_PER_NODE=2
_LMFILES_=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/ucx/1.12.1.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/cuda/11.2.2.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/hwloc/2.7.1.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/openmpi/4.1.3d.lua
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/10103/bus
LMOD_CMD=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/libexec/lmod
C_INCLUDE_PATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include
MKL_NUM_THREADS=8
MAIL=/var/mail/schloegl
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/extras/CUPTI/lib64:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/lib64:1
SLURM_JOB_GID=11114
SLURM_GET_USER_ENV=1
CPATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/include:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/include:/mnt/nfs/clustersw/shared/cuda/11.2.2/include:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/include
DISTRIB_DESCRIPTION=Debian GNU/Linux 11 (bullseye)
SLURM_JOB_NODELIST=delta[197-206]
BASH_FUNC_ml%%=() { eval $($LMOD_DIR/ml_cmd "$@")
}
BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
case "$-" in
*v*x*)
__lmod_sh_dbg='vx'
;;
*v*)
__lmod_sh_dbg='v'
;;
*x*)
__lmod_sh_dbg='x'
;;
esac;
fi;
if [ -n "${__lmod_sh_dbg:-}" ]; then
set +$__lmod_sh_dbg;
echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's
output" 1>&2;
fi;
_mlshopt="f";
case "$-" in
*f*)
unset _mlshopt
;;
esac;
if [ -n "${_mlshopt:-}" ]; then
set -$_mlshopt;
fi;
eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh);
__lmod_my_status=$?;
if [ -n "${_mlshopt:-}" ]; then
set +$_mlshopt;
fi;
unset _mlshopt;
if [ -n "${__lmod_sh_dbg:-}" ]; then
echo "Shell debugging restarted" 1>&2;
set -$__lmod_sh_dbg;
unset __lmod_sh_dbg;
fi;
return $__lmod_my_status
}
_=/usr/bin/env
==== MPIRUN
(Rank:0) tst_test_array[0]:Bcast
(Rank:0) tst_test_array[1]:Gather
(Rank:0) tst_test_array[2]:Allgather
(Rank:0) tst_test_array[3]:Allgather with MPI_IN_PLACE
(Rank:0) tst_test_array[4]:Scan sum
(Rank:0) tst_test_array[5]:Scatter
(Rank:0) tst_test_array[6]:Scatterv
(Rank:0) tst_test_array[7]:Scatterv with stride
(Rank:0) tst_test_array[8]:Reduce Min
(Rank:0) tst_test_array[9]:Reduce Max
(Rank:0) tst_test_array[10]:Reduce Min with MPI_IN_PLACE
(Rank:0) tst_test_array[11]:Reduce Max with MPI_IN_PLACE
(Rank:0) tst_test_array[12]:Allreduce Min
(Rank:0) tst_test_array[13]:Allreduce Max
(Rank:0) tst_test_array[14]:Allreduce Min/Max
(Rank:0) tst_test_array[15]:Allreduce Min/Max with MPI_IN_PLACE
(Rank:0) tst_test_array[16]:Allreduce Sum
(Rank:0) tst_test_array[17]:Alltoall
Number of failed tests: 0
delta197
/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/ompi_info
running on 20*8 cores with 20 MPI-tasks and 8 threads
SHELL=/bin/bash
SLURM_JOB_USER=schloegl
SLURM_TASKS_PER_NODE=2(x10)
SLURM_JOB_UID=10103
SLURM_TASK_PID=51040
PKG_CONFIG_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib/pkgconfig:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib/pkgconfig:/mnt/nfs/clustersw/shared/cuda/11.2.2/pkgconfig
SLURM_LOCALID=0
SLURM_SUBMIT_DIR=/nfs/scistore16/jonasgrp/schloegl/slurm
HOSTNAME=delta197
LANGUAGE=en_US:en
SLURMD_NODENAME=delta197
_ModuleTable002_=ewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvaHdsb2MvMi43LjEubHVhIiwKZnVsbE5hbWUgPSAiaHdsb2MvMi43LjEiLApsb2FkT3JkZXIgPSAzLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImh3bG9jLzIuNy4xIiwKd1YgPSAiMDAwMDAwMDAyLjAwMDAwMDAwNy4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCm9wZW5tcGkgPSB7CmZuID0gIi9tbnQvbmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvQ29yZS9vcGVubXBpLzQuMS4zZC5sdWEiLApmdWxsTmFtZSA9ICJvcGVubXBpLzQuMS4zZCIsCmxvYWRPcmRlciA9IDQsCnByb3BUID0g
MPICC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpicc
__LMOD_REF_COUNT_MODULEPATH=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/MPI/openmpi/4.1.3d:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Linux:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core:1;/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/modulefiles/Core:1
OMPI_MCA_btl=self,openib
_ModuleTable005_=c3cvRGViaWFuL2J1bGxzZXllL2xtb2QvbG1vZC9tb2R1bGVmaWxlcy9Db3JlIiwKfQo=
__LMOD_REF_COUNT_LD_RUN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1
SLURM_NODE_ALIASES=(null)
SLURM_CLUSTER_NAME=istscicomp
OMPI_MCA_btl_openib_allow_ib=1
SLURM_CPUS_ON_NODE=16
__LMOD_REF_COUNT__LMFILES_=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/ucx/1.12.1.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/cuda/11.2.2.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/hwloc/2.7.1.lua:1;/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/openmpi/4.1.3d.lua:1
SLURM_JOB_CPUS_PER_NODE=16(x10)
MPICXX=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpicxx
LMOD_DIR=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/libexec
MPIFC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpifort
PWD=/nfs/scistore16/jonasgrp/schloegl/slurm
SLURM_GTIDS=0
LOGNAME=schloegl
UCX_NET_DEVICES=ibp59s0
XDG_SESSION_TYPE=unspecified
SLURM_JOB_PARTITION=defaultp
MODULESHOME=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod
MANPATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/share/man:/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/share/man::
NUM_CORES=20*8
SLURM_JOB_NUM_NODES=10
OMPI_MCA_pml=ob1
OPENBLAS_NUM_THREADS=8
SLURM_JOBID=33084
SLURM_JOB_QOS=normal
MPI_HOME=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d
__LMOD_REF_COUNT_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/sbin:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/bin:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/bin:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/bin:1;/nfs/scistore16/jonasgrp/schloegl/bin:1;/usr/local/bin:1;/usr/bin:1;/bin:1;/usr/local/games:1;/usr/games:1
HOME=/nfs/scistore16/jonasgrp/schloegl
_ModuleTable_Sz_=5
__LMOD_REF_COUNT_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1
LANG=en_US.UTF-8
__LMOD_REF_COUNT_LOADEDMODULES=ucx/1.12.1:1;cuda/11.2.2:1;hwloc/2.7.1:1;openmpi/4.1.3d:1
__LMOD_REF_COUNT_PKG_CONFIG_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib/pkgconfig:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib/pkgconfig:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/pkgconfig:1
SLURM_PROCID=0
CUDA_DEVICE_ORDER=PCI_BUS_ID
MAN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/share/man:/mnt/nfs/clustersw/shared/cuda/11.2.2/share/man
__LMOD_REF_COUNT_CPATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/include:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/include:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/include:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/include:1
LMOD_SETTARG_FULL_SUPPORT=no
UCX_HOME=/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1
TMPDIR=/tmp
SLURM_CPUS_PER_TASK=8
SLURM_NTASKS=20
SLURM_TOPOLOGY_ADDR=sw11.delta197
LMOD_VERSION=8.6.12
_ModuleTable003_=e30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAib3Blbm1waS80LjEuM2QiLAp3ViA9ICIwMDAwMDAwMDQuMDAwMDAwMDAxLjAwMDAwMDAwMy4qZC4qemZpbmFsIiwKfSwKdWN4ID0gewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvdWN4LzEuMTIuMS5sdWEiLApmdWxsTmFtZSA9ICJ1Y3gvMS4xMi4xIiwKbG9hZE9yZGVyID0gMSwKcHJvcFQgPSB7fSwKc3RhY2tEZXB0aCA9IDEsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJ1Y3gvMS4xMi4xIiwKd1YgPSAiMDAwMDAwMDAxLjAwMDAwMDAxMi4wMDAwMDAwMDEuKnpmaW5hbCIsCn0sCn0sCm1wYXRoQSA9IHsKIi9tbnQv
MODULEPATH_ROOT=/mnt/nfs/clustersw/Debian/bullseye/modulefiles
SLURM_TOPOLOGY_ADDR_PATTERN=switch.node
XDG_SESSION_CLASS=background
LMOD_PKG=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod
SLURM_MEM_PER_NODE=2048
OMPI_MCA_opal_warn_on_missing_libcuda=0
SLURM_WORKING_CLUSTER=istscicomp:10.36.192.126:6817:9216:101
USER=schloegl
LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib
SLURM_NODELIST=delta[197-206]
OMP_MCA_mpi_param_check=YES
ENVIRONMENT=BATCH
LOADEDMODULES=ucx/1.12.1:cuda/11.2.2:hwloc/2.7.1:openmpi/4.1.3d
SLURM_JOB_ACCOUNT=itgrp
SLURM_PRIO_PROCESS=0
SLURM_NPROCS=20
LMOD_ROOT=/mnt/nfs/clustersw/Debian/bullseye/lmod
SHLVL=2
SLURM_NNODES=10
LD_RUN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib
BASH_ENV=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/init/bash
__LMOD_REF_COUNT_MAN_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/share/man:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/share/man:1
LMOD_sys=Linux
DISTRIB_ID=Debian
__LMOD_REF_COUNT_MANPATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/share/man:1;/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/share/man:1
XDG_SESSION_ID=c38
SLURM_SUBMIT_HOST=gpu114
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpNUEkgPSAib3Blbm1waSIsCn0sCm1UID0gewpjdWRhID0gewpmbiA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUvY3VkYS8xMS4yLjIubHVhIiwKZnVsbE5hbWUgPSAiY3VkYS8xMS4yLjIiLApsb2FkT3JkZXIgPSAyLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMSwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImN1ZGEvMTEuMi4yIiwKd1YgPSAiXjAwMDAwMDExLjAwMDAwMDAwMi4wMDAwMDAwMDIuKnpmaW5hbCIsCn0sCmh3bG9jID0g
MPIEXEC=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpiexec
MPIRUN=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin/mpirun
DISTRIB_RELEASE=11
LD_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/extras/CUPTI/lib64:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/lib:/mnt/nfs/clustersw/shared/cuda/11.2.2/lib64
XDG_RUNTIME_DIR=/run/user/10103
SLURM_JOB_ID=33084
SLURM_NODEID=0
_ModuleTable004_=bmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvTVBJL29wZW5tcGkvNC4xLjNkIgosICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0xpbnV4IgosICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0NvcmUiLCAiL21udC9uZnMvY2x1c3RlcnN3L0RlYmlhbi9idWxsc2V5ZS9sbW9kL2xtb2QvbW9kdWxlZmlsZXMvQ29yZSIsCn0sCnN5c3RlbUJhc2VNUEFUSCA9ICIvbW50L25mcy9jbHVzdGVyc3cvRGViaWFuL2J1bGxzZXllL21vZHVsZWZpbGVzL0xpbnV4Oi9tbnQvbmZzL2NsdXN0ZXJzdy9EZWJpYW4vYnVsbHNleWUvbW9kdWxlZmlsZXMvQ29yZTovbW50L25mcy9jbHVzdGVy
LMOD_FAMILY_MPI_VERSION=4.1.3d
OMP_NUM_THREADS=8
LMOD_FAMILY_MPI=openmpi
CUDA_HOME=/mnt/nfs/clustersw/shared/cuda/11.2.2
OMPI_MCA_mpi_cuda_support=1
SLURM_MPI_TYPE=pmix
__LMOD_REF_COUNT_C_INCLUDE_PATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:1
DISTRIB_CODENAME=bullseye
SLURM_CONF=/etc/slurm/slurm.conf
PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/bin:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/sbin:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/bin:/mnt/nfs/clustersw/shared/cuda/11.2.2/bin:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/bin:/nfs/scistore16/jonasgrp/schloegl/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
SLURM_JOB_NAME=mpitest
MODULEPATH=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/MPI/openmpi/4.1.3d:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Linux:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core:/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/modulefiles/Core
SLURM_NTASKS_PER_NODE=2
_LMFILES_=/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/ucx/1.12.1.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/cuda/11.2.2.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/hwloc/2.7.1.lua:/mnt/nfs/clustersw/Debian/bullseye/modulefiles/Core/openmpi/4.1.3d.lua
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/10103/bus
LMOD_CMD=/mnt/nfs/clustersw/Debian/bullseye/lmod/lmod/libexec/lmod
C_INCLUDE_PATH=/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include
MKL_NUM_THREADS=8
MAIL=/var/mail/schloegl
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/lib:1;/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/extras/CUPTI/lib64:1;/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/lib:1;/mnt/nfs/clustersw/shared/cuda/11.2.2/lib64:1
SLURM_JOB_GID=11114
SLURM_GET_USER_ENV=1
CPATH=/mnt/nfs/clustersw/Debian/bullseye/openmpi/4.1.3d/include:/mnt/nfs/clustersw/Debian/bullseye/hwloc/2.7.1/include:/mnt/nfs/clustersw/shared/cuda/11.2.2/targets/x86_64-linux/include:/mnt/nfs/clustersw/shared/cuda/11.2.2/include:/mnt/nfs/clustersw/Debian/bullseye/cuda/11.2/ucx/1.12.1/include
DISTRIB_DESCRIPTION=Debian GNU/Linux 11 (bullseye)
SLURM_JOB_NODELIST=delta[197-206]
BASH_FUNC_ml%%=() { eval $($LMOD_DIR/ml_cmd "$@")
}
BASH_FUNC_module%%=() { if [ -z "${LMOD_SH_DBG_ON+x}" ]; then
case "$-" in
*v*x*)
__lmod_sh_dbg='vx'
;;
*v*)
__lmod_sh_dbg='v'
;;
*x*)
__lmod_sh_dbg='x'
;;
esac;
fi;
if [ -n "${__lmod_sh_dbg:-}" ]; then
set +$__lmod_sh_dbg;
echo "Shell debugging temporarily silenced: export LMOD_SH_DBG_ON=1 for Lmod's
output" 1>&2;
fi;
_mlshopt="f";
case "$-" in
*f*)
unset _mlshopt
;;
esac;
if [ -n "${_mlshopt:-}" ]; then
set -$_mlshopt;
fi;
eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh);
__lmod_my_status=$?;
if [ -n "${_mlshopt:-}" ]; then
set +$_mlshopt;
fi;
unset _mlshopt;
if [ -n "${__lmod_sh_dbg:-}" ]; then
echo "Shell debugging restarted" 1>&2;
set -$__lmod_sh_dbg;
unset __lmod_sh_dbg;
fi;
return $__lmod_my_status
}
_=/usr/bin/env
==== MPIRUN --mca
(Rank:0) tst_test_array[0]:Bcast
(Rank:0) tst_test_array[1]:Gather
(Rank:0) tst_test_array[2]:Allgather
(Rank:0) tst_test_array[3]:Allgather with MPI_IN_PLACE
(Rank:0) tst_test_array[4]:Scan sum
(Rank:0) tst_test_array[5]:Scatter
(Rank:0) tst_test_array[6]:Scatterv
(Rank:0) tst_test_array[7]:Scatterv with stride
(Rank:0) tst_test_array[8]:Reduce Min
(Rank:0) tst_test_array[9]:Reduce Max
(Rank:0) tst_test_array[10]:Reduce Min with MPI_IN_PLACE
(Rank:0) tst_test_array[11]:Reduce Max with MPI_IN_PLACE
(Rank:0) tst_test_array[12]:Allreduce Min
(Rank:0) tst_test_array[13]:Allreduce Max
(Rank:0) tst_test_array[14]:Allreduce Min/Max
(Rank:0) tst_test_array[15]:Allreduce Min/Max with MPI_IN_PLACE
(Rank:0) tst_test_array[16]:Allreduce Sum
(Rank:0) tst_test_array[17]:Alltoall
Number of failed tests: 0
