On Jan 10, 2007, at 1:05 PM, Scott Atchley wrote:
Hi all,
I used MPE to measure send, sendunexpected and recv on the client
during pvfs2-cp (from the client to the server) of a (just less
than) 256 MB file. This is using older 2.4 GHz Xeons with MX-2G (2
Gb/s line rate). I am using FlowBufferSizeBytes of 1 MB. I am using
the default FlowBuffersPerFlow. The server did not generate a log
and I am looking into why.
This first image shows the entire copy of the file from client to
server. Recv are blue, send are red and sendunexpected are orange.
There 25 transfer of 10 MB each and the 26th transfer is ~5.5 MB.
Note that there are distinct gaps between transfers in the range of
17-18 msecs.
<pvfs2-cp-3.png>
The second image shows one of the 25 bulk IO transfers. There is a
11 msec delay from the last send completing to the recv completing
(the large blue box). On the very far left is the initial
sendunexpected that starts this operation.
<pvfs2-cp-2.png>
The third image is of the start of the operation and includes the
first two 1 MB transfers. At the far left, you see the client post
a recv, then post the sendunexpected that contacts the server to
start the operation. Once the first recv completes (the server has
replied), the client sends the first 1 MB. Jumpshot did not draw
this in a way that is intuitive. The small red column on the left
actually completes just before the large red box on the right.
It is interesting that pvfs2-cp posts the first 1 MB send before
the recv. In actuality, it does not matter because the recv is
posted before the last IO transfer per operation.
Hi Scott, Nice images. Rob and I had discussed this behavior you're
seeing of receiving expected messages before the receive has been
posted on the server. We came up with a solution for this (or at
least, something for you to try), and I started implementing it, but
got pulled away by other things. I'll try to finish that up and get
you a patch to try.
The individual 1 MB transfers take about 4.5 msec, which works out
to about 1.8 Gb/s (out of the possible 2 Gb/s). The overall
throughput that pvfs2-cp reports is 1.0 Gb/s.
<pvfs2-cp-1.png>
I wonder if the gaps between operations is due to reading in the
file from disk or something else.
If I understand correctly, this test was a write. The gaps are
likely writing the data to disk. You could verify this by watching
the write acks that the server sends back to the client once all the
data has been written to disk for each operation. My guess would be
that the ack would come write before the next recv (blue block)
started, covering the gap between operations.
To give more detail, for write operations, the client posts a receive
for the write ack before posting the expected sends. Once the server
completes all the IO (pushes it to disk), it sends an expected 'write-
completed' message back to the client. You could probably track this
similar to how you're tracking the sendunexpected and response ack at
the beginning of the IO operation.
-sam
Are the delays at the end of the operation (after the last send
completes until the recv completes) due to the server processing
the received data before replying to the client?
I will try this next on perf.
Scott_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
