On Jan 11, 2007, at 5:43 PM, Sam Lang wrote:
I could post the flow first before posting the ack, which will
remove the race condition guaranteed, but it might also be slower
(which takes longer, buffering the expected message and copying to
the flow buffers or allocating the flow buffers first)?
-sam
Sam,
I am not sure what you are asking here.
In TCP, the OS will receive and buffer the data. There is always a
copy regardless if you pre-post the receive or not. Are you asking
which is faster between memory copy and network transfer? If so, I
would think that the memory copy is always faster. Given that, the
current strategy (ack, then post the flow) makes the most sense.
In IB, I believe that A cannot write/put a large message to B until B
has allocated memory and sent the memory address to A. This is why
bmi_ib needs the RTS and CTS messages.
MX does this internally. When A posts a large send, MX sends a
"scout" message, which is equivalent to the RTS, to B that includes
the matching info and length. If B has posted a receive, than B
replies with an ack and A can start sending data. If B has not posted
a receive, then the scout message goes into the unexpected queue.
When B does post a matching receive, it then has to scan the
unexpected queue to see if it has already arrived. If so, it matches
and sends an ack to start the data transfer.
By pre-posting the receives, we eliminate the scanning a potentially
very long unexpected queue (I am thinking of the case of a storage
server handling 10s or 100s of clients).
If you pre-post the receives, then in the IB case you could send all
of that data in the ack to the initial sendunexpected and potentially
eliminate the RTS and CTS messages as well.
Pete, I could possible be smoking something and this is not possible
in IB at all. Any thoughts?
Sam, it may be that I am trying to optimize something that will not
provide much benefit at all. Can you send a patch that simply posts
the flow before the ack. I can test it on MX-10G and see if it
impacts performance at all. If not, leave things as they are.
Scott
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
