Hi, Joshua Ladd <jladd.m...@gmail.com> writes:
> We cannot reproduce this. On four nodes 20 PPN with and w/o hcoll it > takes exactly the same 19 secs (80 ranks). > > What version of HCOLL are you using? Command line? Thanks for having a look at this. According to ompi_info, our OpenMPI (version 3.0.1) was configured with (and gcc version 7.2.0): ,---- | Configure command line: 'CFLAGS=-I/apps/OPENMPI/SRC/PMI/include' | '--prefix=/storage/apps/OPENMPI/3.0.1/gnu' | '--with-mxm=/opt/mellanox/mxm' | '--with-hcoll=/opt/mellanox/hcoll' | '--with-knem=/opt/knem-1.1.2.90mlnx2' | '--with-slurm' '--with-pmi=/usr' | '--with-pmi-libdir=/usr/lib64' | '--with-platform=../contrib/platform/mellanox/optimized' `---- Not sure if there is a better way to find out the HCOLL version, but the file hcoll_version.h in /opt/mellanox/hcoll/include/hcoll/api/ says we have version 3.8.1649 Code compiled as: ,---- | $ mpicc -o test_t thread_io.c test.c `---- To run the tests, I just submit the job to Slurm with the following script (changing the coll_hcoll_enable param accordingly): ,---- | #!/bin/bash | # | #SBATCH -J test | #SBATCH -N 5 | #SBATCH -n 51 | #SBATCH -t 00:07:00 | #SBATCH -o test-%j.out | #SBATCH -e test-%j.err | #SBATCH -D . | | module purge | module load openmpi/gnu/3.0.1 | | time mpirun --mca coll_hcoll_enable 1 -np 51 ./test_t `---- In the latest test I managed to squeeze in our queuing system, the hcoll-disabled run took ~3.5s, and the hcoll-enabled one ~43.5s (in this one I actually commented out all the fprintf statements just in case, so the code was pure communication). Thanks, -- Ángel de Vicente Tel.: +34 922 605 747 Web.: http://research.iac.es/proyecto/polmag/ --------------------------------------------------------------------------------------------- ADVERTENCIA: Sobre la privacidad y cumplimiento de la Ley de Protección de Datos, acceda a http://www.iac.es/disclaimer.php WARNING: For more information on privacy and fulfilment of the Law concerning the Protection of Data, consult http://www.iac.es/disclaimer.php?lang=en
