When we added the CM PML, we added a pml_max_contextid field to the PML
structure, which is the max size cid the PML can handle (because the
matching interfaces don't allow 32 bits to be used for the cid. At the
same time, the max cid for OB1 was shrunk significantly, so that the
header on a short message would be packed tightly with no alignment
padding.
At the time, we believed 32k simultaneous communicators was plenty, and
that CIDs were reused (we checked, I'm pretty sure). It sounds like
someone removed the CID reuse code, which seems rather bad to me. There
have to be unused CIDs in Ralph's example - is there a way to fallback out
of the block algorithm when it can't find a new CID and find one it can
reuse? Other than setting the multi-threaded case back on, that is?
Brian
On Thu, 30 Apr 2009, Edgar Gabriel wrote:
cid's are in fact not recycled in the block algorithm. The problem is that
comm_free is not collective, so you can not make any assumptions whether
other procs have also released that communicator.
But nevertheless, a cid in the communicator structure is a uint32_t, so it
should not hit the 16k limit there yet. this is not new, so if there is a
discrepancy between what the comm structure assumes that a cid is and what
the pml assumes, than this was in the code since the very first days of Open
MPI...
Thanks
Edgar
Brian W. Barrett wrote:
On Thu, 30 Apr 2009, Ralph Castain wrote:
We seem to have hit a problem here - it looks like we are seeing a
built-in limit on the number of communicators one can create in a
program. The program basically does a loop, calling MPI_Comm_split each
time through the loop to create a sub-communicator, does a reduce
operation on the members of the sub-communicator, and then calls
MPI_Comm_free to release it (this is a minimized reproducer for the real
code). After 64k times through the loop, the program fails.
This looks remarkably like a 16-bit index that hits a max value and then
blocks.
I have looked at the communicator code, but I don't immediately see such
a field. Is anyone aware of some other place where we would have a limit
that would cause this problem?
There's a maximum of 32768 communicator ids when using OB1 (each PML can
set the max contextid, although the communicator code is the part that
actually assigns a cid). Assuming that comm_free is actually properly
called, there should be plenty of cids available for that pattern. However,
I'm not sure I understand the block algorithm someone added to cid
allocation - I'd have to guess that there's something funny with that
routine and cids aren't being recycled properly.
Brian
_______________________________________________
devel mailing list
de...@open-mpi.org
http://www.open-mpi.org/mailman/listinfo.cgi/devel
