FWIW, I *always* report MPI application time in wall-clock seconds
time. I know that some people (even among the OMPI developers)
disagree with me, but to me, there's nothing else that you can measure
that makes sense.
Case in point: when using the OpenFabrics network stack, very little
time is spent in the kernel because OpenFabrics networks are designed
to bypass the OS (e.g., we spin poll in userspace for OpenFabrics
message passing progress). Similar is true for shared memory (it's a
"network" because we use it to pass messages between MPI processes).
But what about TCP? When not using a TOE or other similar technology
(i.e., 99.99% of the time), you are making OS syscalls.
Hence, running the same program over these three different networks
can result in hugely different proportions of user vs. system time,
even though it's the same app and the same algorithm. Granted, some
of the networks are faster than the others, but the network should
always be the slowest part of your computation (assuming you have a
well-coded application). So which numbers should you report?
In short: the MPI implementation is doing things for you behind the
scenes. This raises some obvious questions:
1. Do you report the MPI execution times or not?
1a. If so, how do you account for the differences in network
progression (and other issues) based on the type of network?
1b. If not, how can you separate the MPI time from your application
time? (user/system does not make this differentiation; you need
additional tools to separate MPI vs. application time)
To me, only wall-clock execution time makes sense. The overall
performance of your application *includes* the time necessary for MPI/
message passing and everything else running on the machine. One of
the major points of parallel computing is to make things go faster.
To measure that, measure the wall-clock time of the application in
serial and then measure the wall-clock execution time in parallel
(perhaps for various different np values). Then you can (hopefully)
see clear, easy-to-understand speedup. To avoid OS-induced jitter and
negative timing effects, most people typically turn off as many OS
services as possible on the nodes that they're running, both for
production and benchmarking codes (I typically leave such services
enabled on my software development nodes, because they're helpful for
debugging, etc.).
Is wall-clock execution time the only / best metric? Certainly not.
But I strongly prefer it over user/system time -- I just don't think
that user/system time tell you what most people think they're telling
you in a parallel+MPI context.
On Nov 14, 2008, at 4:32 AM, Raymond Wan wrote:
Hi Fabian,
Thank you for clarifying things and confirming some of the things
that I thought. I guess I have a clearer understanding now.
Fabian Hänsel wrote:
Hmmmm, I guess user time does not matter since it is real time that
we are interested in reducing.
Right. Even if we *could* measure user time of every MPI worker
process
correctly this was not what you are interested in: Depending on the
algorithm a significant amount of time could get spend waiting for
MPI
messages to arrive - and that time would not count as user time, but
also was not 'wasted' as something important happens.
The reason why I was wondering is that some people in research
papers compare their algorithm (system) with another one by
measuring user time since it removes some of the effects of what the
system does on behalf of the user's process. And some people, I
guess, see this as a fairer comparison.
On the other hand, I guess I've realized the obvious -- that Open
MPI doesn't reduce the efficiency of the algorithm. Even worse,
increases in user time is an artifact of Open MPI, so it is somewhat
misleading if we are analyzing an algorithm. What MPI should do (if
properly used) is to reduce the real time and that's what we should
be reporting...even if it includes other things that we did not want
previously, like the time spent by the OS in swapping memory, etc.
[Papers I've read with graphs that have "time" on the y-axis and
"processors" on the x-axis rarely mention what time they are
measuring...but it seems obviously now that it must be real time
since user time should [???] increase with more processors.....I
think...of course, assuming we can total the user time across
machines accurately.]
Thank you for your message(s)! Think I got it now... :-)
Ray
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
--
Jeff Squyres
Cisco Systems
