On Feb 21, 2009, at 1:05 AM, Raymond Wan wrote:
Hi Ralph,
Thank you very much for your explanation!
Ralph Castain wrote:
It is a little bit of both:
* historical, because most MPI's default to mapping by slot, and
* performance, because procs that share a node can communicate via
shared memory, which is faster than sending messages over an
interconnect, and most apps are communication-bound
If your app is disk-intensive, then mapping it -bynode may be a
better
Ok -- by this, it seems that there is no "rule" that says one is
obviously better than the other. It depends on factors such as disk
access and shared memory access and which one is dominating. So, it
is worth to try both to see?
Can't hurt! You might be able to tell by knowing what your app is
doing, but otherwise, feel free to experiment.
option for you. That's why we provide it. Note, however, that you
can still wind up with multiple procs on a node. All "bynode" means
is that the ranks are numbered consecutively bynode - it doesn't
mean that there is only one proc/node.
I see. But if the number of processes (as specified using -np) is
less than the number of nodes, if "by node" is chosen, then is it
guaranteed that only one process will be on each node?
That is correct
Is there a way to write the hostfile to ensure this?
You don't need to do anything in the hostfile - if you use bynode and
np < #nodes, it is guaranteed that you will have only one proc/node
I was curious if a node has 4 slots, whether writing it 4 times in
the hostfile with 1 slot each has any meaning. Might be a bad idea
as we are trying to fool mpirun?
It won't have any meaning as we aggregate the results. In other words,
we read through the hostfile, and if a host appears more than once, we
simply add the #slots on subsequent entries to the earlier one. So we
wind up with just one instance of that host that has the total number
of slots allocated to it.
If you truly want one proc/node, then you should use the -pernode
option. This maps one proc on each node up to either the number of
procs you specified or the number of available nodes. If you don't
specify -np, we just put one proc on each node in your allocation/
hostfile.
I see ... I was not aware of that option; thank you!
Do a "man mpirun" and you will see that there are several mapping
options that might interest you, including:
1. npernode - let's you specify how many procs/node (as opposed to
"pernode", where you only get one proc/node - obviously, pernode is
the equivalent of "-npernode 1")
2. seq - a sequential mapper. This mapper will read a file (which can
be different from the hostfile used to specify your allocation) and
assign one proc to each entry in a sequential manner like this:
node1 ----> rank 0 goes on node1
node5 ----> rank 1 goes on node5
node1 ----> rank 2 goes on node1
...
3. rank_file - allows you to specify that rank x goes on node foo, and
what core/socket that rank should be bound to
The man page will describe all the various options. Which one is best
for your app really depends on what the app is doing, the capabilities
and topology of your cluster, etc. A little experimentation can help
you get a feel for when to use which one.
HTH
Ralph
Ray
_______________________________________________
users mailing list
us...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/users
