On Aug 08, 2016, at 05:17 AM, Paul Kapinos <kapi...@itc.rwth-aachen.de> wrote:
Dear Open MPI developers,
there is already a thread about 'sm BTL performace of the openmpi-2.0.0'
https://www.open-mpi.org/community/lists/devel/2016/07/19288.php
and we also see 30% bandwidth loss, on communication *via InfiniBand*.
And we also have a clue: the IB buffers seem not to be aligned in 2.0.0 - in
contrast to previous series (from at least 1.8.x).
That means,
- if we use a simple wrapper wrapping 'malloc' to the 32-bit-aligned-variant, we
get the full bandwidth using the same compiled binary; and
- there is nothing to grep in 'ompi_info -all | grep memalign' in 2.0.0 while in
1.10.3 there are 'btl_openib_memalign' and 'btl_openib_memalign_threshold'
parameters.
=> seem the whole 'IB buffer alignment' part vanished in /2.0.0 ?
Could we get the aligned IB buffers in 2.x series back, please? It's about 30%
of performance....
Open MPI developers will discuss this next week. This support was removed when we moved away from using ptmalloc2 to detect free/munmap/etc. Hooking malloc in is non-portable and can lead to a number of issues. In my opinion users should be handling the alignment of their own buffers and if that is not possible can use their own hooks to provide an aligned malloc. That said, I don't see how aligning the buffers could possibly lead to a 30% performance increase for large messages. What is the typical message size? It might suggest that something else in Open MPI is not performing as it should. Can you provide a reproducer that shows the bandwidth drop. If I can get a reproducer I may be able to improve the performance for 2.0.2 without re-adding the alignment code.
Best
Paul
P.S. btl_openib_get_alignment and btl_openib_put_alignment are by default '0' -
setting they high did not change the behaviour...
These are hardware constants used to indicate the alignment restrictions of the hardware. They can not be changed.
--
Dipl.-Inform. Paul Kapinos - High Performance Computing,
RWTH Aachen University, IT Center
Seffenter Weg 23, D 52074 Aachen (Germany)
Tel: +49 241/80-24915_______________________________________________
devel mailing list
devel@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ devel mailing list devel@lists.open-mpi.org https://rfd.newmexicoconsortium.org/mailman/listinfo/devel
