Also, during the copy, the client is using about 25% of cpu and the
server is using about 40%. When the server is idle, it is using about
25% cpu.
Scott
On Dec 29, 2006, at 10:43 AM, Scott Atchley wrote:
Hi all,
What performance do you typically see with a single client and
single server (not the same machine) with 10 Gb/s NICs?
I am using pvfs2-cp to copy a 1 GB file from the client to the
server. The client is reading from a tmpfs mount so it does not use
disk (I am not swapping). The server's backing store is also tmpfs.
I set FlowBufferSizeBytes to 1 MB. With tweaking, I am seeing about
400 MB/s.
On the same machine, if I use dd to copy from /dev/zero to /mnt/
tmpfs/zeros using 1 MB blocks, I get 300 MB/s for a 1 GB file.
Initially, I used the dumbest of BMI_meth_memalloc() and
BMI_meth_memfree(), where they are simply calls to malloc() and free
(), and I was getting about 300 MB/s. Thinking that this was the
problem, I tinkered with mallopt() to set higher thresholds for
trim and mmap. This added about 50 MB/s.
Next, I added pre-malloced memory on startup and I manage a list of
these buffers. This added another 50 MB/s to get me to 400 MB/s. I
tried playing with pvfs2-cp's -b option but performance never
improved over the default behavior. Interestingly, on the client,
pvfs2-cp only uses two 1 MB buffers (over and over) for the entire
1 GB transfer. Is this intentional? Does this mean, that only one
buffer is in flight while the other is being filled? Is there a way
to get pvfs2-cp to use more concurrent messages?
With Lustre, I see ~675 MB/s with a single client using one thread
to a single server. This is not going through the entire
filesystem, however. It is simply testing the network layer. By
default, though, Lustre will try to use 8 or 16 threads (depending
on a configurable parameter).
Scott
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
