Hi:
I performed some basic tests today with pvfs2 and 2.8.1. Running this
version, and all tests performed with the kernel connector (i.e.,
traditional filesystem mount), I performed a few tests.
My topology is as follows:
3 dedicated pvfs2 servers, serving I/O and metadata (although all
clients are given the address to the first server in their URI). Each
I/O server has 2 gig-e connections into the gigabit network switch for
the cluster, and are runing ALB ethernet load balancing. In theory,
each server has 2Gbps throughput potential now. For disk drives, all
my servers are running Dell PERC 6/e cards and MD1000's array of 15
SATA 750GB hard drives in hardware RAID-6. Each pvfs server is
responsible for just under 10TB of disk storage. Using the test
command below, it came out to 373MB/s to the local disk on the pvfs
server.
All of my clients are single Gig-E connected into the same gigabit
switch, same network. My network is comprised of ROCKS 5.1 (CentOS
5.2) and some just plain old CentOS 5 servers.
The test command was: dd if=/dev/zero of=<file>.out bs=4000K count=2800
First test: single machine to pvfs storage: 95.6MB/s
Second Test: two cluster machines to pvfs storage: 80.2 MB/s
Third test: 3 machines: 53.2MB/s
Fourth test: 4 machines: 44.7MB/s
This test surprised me greatly. My understanding was the big benifits
behind pvfs was the scalability; that with 3 I/O servers, I should
reasonably expect to get at least 3x the bandwidth. Given this, I
have a theoretical 6 Gbps to my storage, yet my actual throughput did
not scale much at all...My initial single-machine connection came out
at a bit under 1Gbps, and my 4 machine connection came up at 1.2Gbps.
Each time I added a machine, the throughput of them all went down.
What gives? My actual local disk throughput on my I/O servers is
373MB/s and the local pvfs2 server system load never broke 1.0, so
that wasn't the bottleneck...
Here's my pvfs2-fs.conf:
<Defaults>
UnexpectedRequests 50
EventLogging none
LogStamp datetime
BMIModules bmi_tcp
FlowModules flowproto_multiqueue
PerfUpdateInterval 1000
ServerJobBMITimeoutSecs 30
ServerJobFlowTimeoutSecs 30
ClientJobBMITimeoutSecs 300
ClientJobFlowTimeoutSecs 300
ClientRetryLimit 5
ClientRetryDelayMilliSecs 2000
StorageSpace /mnt/pvfs2
LogFile /var/log/pvfs2-server.log
</Defaults>
<Aliases>
Alias pvfs2-io-0-0 tcp://pvfs2-io-0-0:3334
Alias pvfs2-io-0-1 tcp://pvfs2-io-0-1:3334
Alias pvfs2-io-0-2 tcp://pvfs2-io-0-2:3334
</Aliases>
<Filesystem>
Name pvfs2-fs
ID 62659950
RootHandle 1048576
<MetaHandleRanges>
Range pvfs2-io-0-0 4-715827885
Range pvfs2-io-0-1 715827886-1431655767
Range pvfs2-io-0-2 1431655768-2147483649
</MetaHandleRanges>
<DataHandleRanges>
Range pvfs2-io-0-0 2147483650-2863311531
Range pvfs2-io-0-1 2863311532-3579139413
Range pvfs2-io-0-2 3579139414-4294967295
</DataHandleRanges>
<StorageHints>
TroveSyncMeta yes
TroveSyncData no
</StorageHints>
</Filesystem>
--Jim
_______________________________________________
Pvfs2-users mailing list
[email protected]
http://www.beowulf-underground.org/mailman/listinfo/pvfs2-users
