Hey,
I only may to add the XRC and RC have the same latency.
What is the command line that you use to run this benchmark ?
What is the system configuration (one hca, one active port ) ?
Any addition information about system configuration, mpi command line,
etc. will help to analyze your issue.
Regards,
Pasha (Mellanox guy :-) )
Jeff Squyres wrote:
I'll defer to the Mellanox guys to reply more in detail, but here's a few
thoughts:
- Is MVAPICH using XRC? (I never played with XRC much; it would surprise me if
it caused instability on the order of up to 100 micros -- I ask just to see if
it is an apples-to-apples comparison)
- The nRepeats value in this code is only 10, meaning that it only seems to be
doing 10 iterations on each size. For small sizes, this might well be not
enough to be accurate. Have you tried increasing it? Or using a different
benchmark app, such as NetPIPE, osu_latency, ...etc.?
On Feb 16, 2010, at 8:49 AM, Repsher, Stephen J wrote:
Well the "good" news is I can end your debate over binding here...setting
mpi_paffinity_alone 1 did nothing. (And personally as a user, I don't care what the
default is so long as info is readily apparent in the main docs...and I did see the FAQs
on it).
It did lead me to try another parameter though, -mca mpi_preconnect_all 1,
which seems to reduce the measured latency reliably of subounce, but it's still
sporadic and order ~10-100 microseconds. It leads me to think that OpenMPI has
issues with the method of measurement, which is simply to send progressively
larger blocked messages right after calling MPI_Init (starting at 0 bytes which
it times as the latency). OpenMPI's lazy connections clearly mess with this.
But still not consistently 1-2 microsecs...
Steve
-----Original Message-----
From: users-boun...@open-mpi.org [mailto:users-boun...@open-mpi.org] On Behalf
Of Ralph Castain
Sent: Monday, February 15, 2010 11:21 PM
To: Open MPI Users
Subject: Re: [OMPI users] Bad Infiniband latency with subounce
On Feb 15, 2010, at 8:44 PM, Terry Frankcombe wrote:
On Mon, 2010-02-15 at 20:18 -0700, Ralph Castain wrote:
Did you run it with -mca mpi_paffinity_alone 1? Given this is 1.4.1, you can
set the bindings to -bind-to-socket or -bind-to-core. Either will give you
improved performance.
IIRC, MVAPICH defaults to -bind-to-socket. OMPI defaults to no binding.
Is this sensible? Won't most users want processes bound? OMPI's
supposed to "to the right thing" out of the box, right?
Well, that depends on how you look at it. Been the subject of a lot of debate
within the devel community. If you bind by default and it is a shared node
cluster, then you can really mess people up. On the other hand, if you don't
bind by default, then people that run benchmarks without looking at the options
can get bad numbers. Unfortunately, there is no automated way to tell if the
cluster is configured for shared use or dedicated nodes.
I honestly don't know that "most users want processes bound". One installation
I was at set binding by default using the system mca param file, and got yelled at by a
group of users that had threaded apps - and most definitely did -not- want their
processes bound. After a while, it became clear that nothing we could do would make
everyone happy :-/
I doubt there is a right/wrong answer - at least, we sure can't find one. So we don't
bind by default so we "do no harm", and put out FAQs, man pages, mpirun option
help messages, etc. that explain the situation and tell you when/how to bind.
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
