Re: [Pvfs2-developers] server flow post

Fri, 12 Jan 2007 12:24:25 -0800 


On Jan 12, 2007, at 12:32 PM, Scott Atchley wrote:


On Jan 12, 2007, at 12:52 PM, Sam Lang wrote:


On Jan 12, 2007, at 9:03 AM, Pete Wyckoff wrote:


I think that all makes sense.  Agree that the need for preposting
receives is to avoid big queues of waiting unexpected messages.


Hi Scott,
The attached patch posts the flow (and receives) before posting the send of the response ack. I posted the response ack before the flow, because the first call to BMI_memalloc (with a request for 1MB in your case) happens in the flow post call, so that delays posting of the response ack. I'm curious if this will actually give you better performance.
I also fixed that assert failure you were getting. Let me know if this works for you. 

Thanks,

-sam

<io-flow-post.patch>
I applied the patch, recompiled and ran pvfs2-cp again. I see about the same number of unexpected receives on the server as before:
It looks like there are more early expected messages than previously. You were only seeing about 9 per tag before IIRC.
We don't post the next BMI read until the previous has finished on the server, so there's still the possibility of a race-condition, because the client could send the next message before the server actually posted the receive for it, but the time in the unexpected queue should be pretty short.
Do you see any performance differences with the patch? Maybe increase the flow buffer sizes to something big (10MB or so) to see if allocation kills performance? 

-sam
[D 13:28:57.671374] PVFS2 Server version 2.6.1pre1-2007-01-11-200834 starting. Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 8 match 0xe000000100000008 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 9 match 0xe000000100000009 Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 10 match 0xe00000010000000a Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 11 match 0xe00000010000000b Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 12 match 0xe00000010000000c Early expected message length 1048576 tag 13 match 0xe00000010000000d Early expected message length 1048576 tag 13 match 0xe00000010000000d Early expected message length 1048576 tag 13 match 0xe00000010000000d Early expected message length 1048576 tag 13 match 0xe00000010000000d Early expected message length 538689 tag 13 match 0xe00000010000000d
Note that it starts at tag 8. Is the client sending the second set of buffers before the server has posted them? 

Scott



_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers

Reply via email to