On Nov 30, 2006, at 1:16 PM, Scott Atchley wrote:
On Nov 30, 2006, at 12:13 PM, Sam Lang wrote:
On Nov 30, 2006, at 9:27 AM, Scott Atchley wrote:
Hi all,
No one replied the the original post. The one I am most curious
about is (3). Is FlowBufferSizeBytes available to the BMI
implementations?
Hi Scott,
Sorry for not responding. I've included responses to your
questions inline.
No problem. I have been side-tracked on a couple things as well.
I am thinking about how to handle BMI_mx_memalloc(). I might be
able to assist the reg cache in MX if I do things a certain way.
I mention this below, on the server we always call BMI_memalloc to
allocate buffers and post them to BMI_post_send/BMI_post_recv. On
the client that's not always the case, the user buffer is usually
passed directly to BMI_post_send/BMI_post_recv. Would it help you
to know in advance of the post (or all posts) what the maximum
size of a buffer for BMI_post_send/BMI_post_recv is going to be?
We might be able to add a BMI info option that other methods could
ignore. We do limit it, but its all above the BMI interface that
this is done.
Knowing the max size in advance helps iff the buffers are allocated
using BMI_memalloc(). If not, it does not matter.
So an extra BMI info option would help on the server?
1. Is there an upper bound on how many transfer operations
between a pair of hosts at any one time (i.e. between a client
and host)?
There is a max of unexpected requests (UnexpectedRequests option
in the config file, defaults to 50). This limits the number of
unexpected messages that can be handled at once by the server, but
doesn't limit the number of IO operations (large expected messages).
I read this as the server will only service up to 50 unexpected
request, but that the client (or clients) can send more. Is that
correct?
Yes that's correct. There's no limit on unexpected messages a client
can send to one particular, or any number of servers.
2. If there is a limit in the above and it is greater than 1, is
that value the same for small (unexpected and some expected)
messages and large (expected) messages? Or are large messages
restricted to a lower value?
So I don't think there's a limit at present for expected
messages. The unexpected limit is the one I mentioned above.
Ok.
3. In the thread entitled "libpvfs2 usage", SamL mentioned the
tunable called FlowBufferSizeBytes and that it is set to 256KB
by default. Is this value the upper limit on large messages
(i.e. 8KB < large messages <= FlowBufferSizeBytes assuming 8 KB
is the maximum unexpected size)?
Yes, on the server BMI_post_recv and BMI_post_send won't get
called with values larger than that. On the client, there's no
strict limit on the size of the messages posted.
Below, you seem to imply that client messages are broken into
FlowBufferSizeBytes chunks. That would make sense if a server never
allocated more than FlowBufferSizeBytes.
Would the BMI method have any knowledge of what this value is
(i.e. can it use FlowBufferSizeBytes to allocate large buffers)?
On the server it does. BMI_memalloc is called from the flow with
exactly the 256K (FlowBufferSizeBytes) value. The buffer returned
is the one passed to BMI_post_recv or BMI_post_send (although, the
size of the request may be less than 256K). On the client, the
user buffer is used for calls to BMI_post_recv and BMI_post_send,
so you won't see BMI_memalloc calls first. The client does break
up the entire client request into chunks of FlowBufferSizeBytes
for the BMI posts, but you won't know what that value is in
advance of the post operations.
-sam
Ok. Given that the client does not use BMI_memalloc(), I will not
try to do anything for BMI_memalloc() for now other than call malloc
(). Do not worry about exposing the FlowBufferSizeBytes value for now.
Ok. Let us know if adding a BMI_set_info option would help.
-sam
Thanks,
Scott
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
