Phil Carns wrote:
We recently ran some tests trying different sync settings in PVFS2.
We ran into one pleasant surprise, although probably it is already
obvious to others. Here is the setup:
12 clients
4 servers
read/write test application, 100 MB operations, large files
fibre channel SAN storage
The test application is essentially the same as was used in the
posting regarding kernel buffer sizes, although with different
parameters in this environment.
At any rate, to get to the point:
with TroveSyncData=no (default settings): 173 MB/s
with TroveSyncData=yes: 194 MB/s
I think the issue is that if syncdata is turned off, then the buffer
cache tends to get very full before it starts writing. This bursty
behavior isn't doing the SAN any favors- it has a big cache on the
back end and probably performs better with sustained writes that don't
put so much sudden peak traffic on the HBA card.
There are probably more sophisticated variations of this kind of
tuning around (/proc vm settings, using direct io, etc.) but this is
an easy config file change to get an extra 12% throughput.
This setting is a little more unpredictable for local scsi disks- some
combinations of application and node go faster but some go slower.
Overall it seems better for our environment to just leave data syncing
on for both SAN and local disk, but your mileage may vary.
This is different from results that we have seen in the past (maybe a
year ago or so) for local disk- it used to be a big penalty to sync
every data operation. I'm not sure what exactly happened to change
this (new dbpf design? alt-aio? better kernels?) but I'm not
complaining :)
One thing that we noticed while testing for storage challenge was that
(and everyone correct me if I'm wrong here) enabling the data-sync
causes a flush/sync to occur after every sizeof(FlowBuffer) bytes had
been written. I can imagine how this would help a SAN, but I'm
perplexed how it helps localdisk, what buffer size are you playing with?
We found that unless we were using HUGE (~size of cache on storage
controller) flowbuffers that this caused way too many syncs/seeks on the
disks and hurt performance quite a bit, maybe even as bad as 50%
performance because things were not being optimized for our disk
subsystems and we were issuing many small ops instead of fewer large ones.
Granted I havent been able to get 2.6.0 building properly yet to test
the latest out, but this was definitely the case for us on the 2.5 releases.
+=Kyle
-Phil
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
!DSPAM:456db371123088992556831!
--
Kyle Schochenmaier
[EMAIL PROTECTED]
Research Assistant, Dr. Brett Bode
AmesLab - US Dept.Energy
Scalable Computing Laboratory
_______________________________________________
Pvfs2-developers mailing list
Pvfs2-developers@beowulf-underground.org
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-developers
