Hi All,
Sorry for not being around earlier to participate in this
discussion. I agree that the bits of code in
io_find_target_datafiles are nasty and I'll be sure to clean that up,
but the cause of this bug isn't with small IO. The check total_bytes
<= max_unexp_payload will still do the right thing (not enable small
IO) even if max_unexp_payload is negative. In fact, the problem
occurs because the noncontig request makes the normal IO request
larger than 4K, and when the sys-io state machine tries to post the
unexpected send, the BMI mx layer returns EINVAL because the
request is larger than its specified unexpected size (see mx.c:1391).
We do the same thing in other BMI methods (gm and ib), but the
unexpected limits there are bigger (8K for ib, 16K for gm, 16K for
tcp), and so we've never actually hit them with unexpected requests.
With a large indexed request, I think we would see the same errors
with ib and gm, unless we hit the limit of max request segments
first. Each individual segment of an indexed request take 80 bytes,
so we would need to have an indexed request with about 100 segments
before hitting the max for ib, and somewhere around 200 for tcp and
gm. The limit of of segments for a request is hardcoded to 100 right
now.
At this point, It seems like the best fix is the one Scott chose,
just increase the max unexpected size for MX. The alternative is to
split up unexpected requests if they're above a certain size into
multiple unexpecteds, and join them on the server. Messy and a lot
of work. If the unexpected message size isn't card specific, could
we make it something like 32K or 64K? Are there drawbacks to making
it that big?
Also, should we increase the limit of request segments allowed? It
might be inefficient for a user to create an MPI indexed dataype with
that many elements, but there are users that will probably do it
anyway. Alternatively, we could consider more efficiently encoding
each request segment in PVFS.
As an aside, the other methods return -EMSGSIZE, mx is returning
EINVAL, which may have made this harder to debug.
-sam
On Aug 7, 2007, at 3:21 PM, Pete Wyckoff wrote:
[EMAIL PROTECTED] wrote on Tue, 07 Aug 2007 15:21 -0400:
I assumed that small_io_size is fixed in PVFS2 and was greater than 4
KB, which why I volunteered to change bmi_mx. I chose 4 KB for bmi_mx
simply because that was the value I used in Lustre (kernel page
size). I am not wedded to 4 KB.
Okay, good reasoning. We'll let Sam tell us what he thinks. He did
the small io work. I can't think of a reason why any device could
not support a minimum of 8k, like you say, if that would make more
sense for the small io implementation.
-- Pete
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
