Jeff Squyres wrote:
It looks like Eugene's and George's fixes on coll sm resolve all the
known hangs. We still have flow control issues, but that is
temporarily being solved by the coll sync component. To be clear:
running with coll_sync_barrier_before 1000 seems to resolve all known
hangs, and we think that this is good enough for v1.3.3. We should
CMR whatever is necessary to the v1.3 branch.
==> We should also default coll_sync_barrier_before to 1000 for
v1.3.3 (i.e., ensure sync activates itself).
For the future, we have a two pronged plan:
I suspect the standard procedure is that we all look quickly at this
e-mail message, file appropriately, and then resume our normal lives.
Yes? Or, is such a plan put somehow into place?
1. Clean up the sm btl:
1a. Remove all dead code.
What do you mean here? (Possibly you mean getting rid of sm pending
sends if we implement 1b properly, but I'm not sure.)
1b. Resize free_list_max and fifo_size MCA params to effect good
enough flow control.
1c. Possibly: convert from FIFO's to linked lists (for future
maintenance purposes, not necessarily to fix problems).
Another idea is to have two kinds of FIFOs. One is just for returning
fragments. The other is for in-coming message fragments. It would even
seem as though one would no longer need "free lists", but just use the
ack FIFO to manage fragments. (ALL of this is complicated by the fact
that we have two kinds of fragments, eager and max, but fortunately
those details can be pushed onto the sorry fool who'll be implementing
all this. I wonder who that'll be.)
2. Test, enable, and continue to develop the coll sm module. Using
this module will avoid the p2p unexpected message queue explosion
that we're seeing (at least for collectives with short messages).
It nominally has broadcast, barrier, reduce, and allreduce
implemented. We really only need to a) test the heck outta them, and
b) add gather, scatter, scan, and exscan to the list. All the other
collective operations have implicit synchronization and won't run
into the unbounded unexpected queue issues. The bcast loop
reproducer seemed to work fine for me with the coll sm, but it
segv'ed immediately for Ralph. So clearly some work needs to be done.
We think that these two items should be the main features for 1.3.4.
