On Fri, 3 Mar 2017, Paul Kapinos wrote:
...
Note that on 1.10.x series (even on 1.10.6), enabling of
MPI_THREAD_MULTIPLE in lead to (silent) shutdown of the InfiniBand
fabric for that application => SLOW!
2.x versions (tested: 2.0.1) handle MPI_THREAD_MULTIPLE on InfiniBand
the right way up, however due to absence of memory hooks (= nut aligned
memory allocation) we get 20% less bandwidth on IB with 2.x versions
compared to 1.10.x versions of Open MPI (regardless with or without
support of MPI_THREAD_MULTIPLE).
On Intel OmniPath network both above issues seem to be not present, but
due to a performance bug in MPI_Free_mem your application can be
horribly slow (seen: CP2K) if the InfiniBand failback of OPA not
disabled manually, see
https://www.mail-archive.com/users@lists.open-mpi.org//msg30593.html
...
Hi Paul,
All very useful - thanks :)
Our (limited) testing seems to show no difference on 2.x with
MPI_THREAD_MULTIPLE enabled vs. disabled as well, which is good news. Glad
to hear another opinion.
Your 20% memory bandwidth performance hit on 2.x and the OPA problem are
concerning - will look at that. Are there tickets open for them?
Cheers,
Mark
_______________________________________________
users mailing list
users@lists.open-mpi.org
https://rfd.newmexicoconsortium.org/mailman/listinfo/users
