I am going to try to find a brute force way to serialize I/O from each pvfs2-client-core just to see if that solves the problem (maybe only allowing one buffer between pvfs2-client-core and kernel, rather than 5). If that does look like it fixed the problem, then we need a more elegant solution. Maybe waiting for acks before starting flows, or just somehow serializing flows that share sockets.
To follow up on this a little bit:
I modified pint-dev-shared.h to change PVFS2_BUFMAP_DESC_COUNT from 5 to 1 and recompiled. This controls the number of buffers that are shared between the kernel and pvfs2-client-core. With only one buffer available, that means that only one I/O operation can be posted from kernel to pvfs2-client-core at a time.
With this in place I was able to run over 100 iterations of our problem application last night without any errors or hangs.
We will probably use this as an intermediate solution to our problem until the ack/ordering issue is sorted out. We haven't yet measured to see what the performance impact is, but it should only affect applications that run multiple processes per node, since single processes only use one of these buffers at a time right now anyway.
-Phil _______________________________________________ Pvfs2-developers mailing list Pvfs2-developers@beowulf-underground.org http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
