Okay, put back as r1846.
On 8/16/2011 1:23 PM, Jeff Squyres wrote:
We talked about this a lot today on the call (and then some more afterwards).
:-)
I think there's 2 important points here.
1. Ralph's original test was written with the intent of launching it with 1
process which would then do a series of local spawns. Even doing a huge
truckload of them, Ralph mentioned (on the phone to me today) that it only took
about 15 seconds.
15 seconds sounds to me like a stretch -- at least for the 2000
iterations that, not too long ago, were in the code. But, I quibble.
2. My test -- i.e., the current one in the ibm test suite directory -- is more of a
general "beat on the ORTE/spawn system" test. I.e., just spawn/reap a
bajillion times and ensure that it works. I think that it still breaks openib, for
example -- after you do a bunch of spawns, something runs out of resources (I don't
remember the exact failure scenario).
The comments to the putback "say it all." Main point is that the
number of spawning processes is cut back in some cases, and sleep-waits
have been introduced. Specifically in the case of loop_spawn, both
patterns (Ralph's and Jeff's) are in there, *and* there is a
user-settable time limit (ten minutes by default). So, /everyone/
should be happy! Happy happy joy joy.
Ralph's opinion is that we don't need to test for #1 any more. I don't think it would be bad
to test for #1 any more, but the C code for such a test could be a bit smarter (i.e., only
MCW rank 0 could COMM_SPAWN on COMM_SELF, and use a host info key of "localhost" to
ensure spawning locally, while any other MCW procs could idle looping on while (!done)
{sleep(1); MPI_Test(...,&done); } so that they don't spin the CPU).
This is a digression, but sleep(1) is excessive. If the cost of a
futile MPI_Test is a fraction of a millisecond, then checking out for a
full second should drop load by over 1000x. Run times will be
horrendous... not because of oversubscription, but because the machine
will be idle! Anyhow, again, just a digression. I hope/expect that the
spawn/load problems are now gone.
For #2, I don't disagree that Eugene's suggestions could make it a bit more
robust. After all, we only have so many hours for testing with so much
equipment; one test that runs for hours and hours probably isn't useful. You
can imagine a bunch of ways to make that test more useful: take an argv[1]
specifying the number of iterations, take an argv[1] that indicates a number of
seconds to run the test, ensure that you only spawn on half the MCW processes
and have the other half idle in a while(!done){...} loop, like mentioned above
so that you can spawn on CPUs that aren't spinning tightly on MPI progress,
...etc.
On Aug 15, 2011, at 11:47 AM, Eugene Loh wrote:
This is a question about ompi-tests/ibm/dynamic. Some of these tests (spawn, spawn_multiple,
loop_spawn/child, and no-disconnect) exercise MPI_Comm_spawn* functionality. Specifically,
they spawn additional processes (beyond the initial mpirun launch) and therefore exert a
different load on a test system than one might naively expect from the "mpirun
-np<np>" command line.
One approach to testing is to have the test harness know characteristics about
individual tests like this. E.g., if I have only 8 processors and I don't want
to oversubscribe, have the test harness know that particular tests should be
launched with fewer processes. On the other hand, building such generality
into a test harness when changes would have to be so pervasive (subjective
assessment) and so few tests require it may not make that much sense.
Another approach would be to manage oversubscription in the tests themselves.
E.g., for spawn.c, instead of spawning np new processes, do the following:
- idle np/2 of the processes
- have the remaining np/2 processes spawn np/2 new ones
(Okay, so that leaves open the possibility that the newly spawned processes might not
appear on the same nodes where idled processes have "made room" for them. Each
solution seems loaded with shortcomings.)
Anyhow, I was interested in some feedback on this topic. A very small number (1-4) of
spawning tests are causing us lots of problems (undue complexity in the test harness as
well as a bunch of our time for reasons I find difficult to explain succinctly). We're
inclined to modify the tests so that they're a little more social. E.g., make decisions
about how many of the launched processes should "really" be used, idling some
fraction of the processes, and continuing the test only with the remaining fraction.
