Hi Pavan,
Thanks for the reply - my comments inline below
Regards,
Thomas
-Original Message-
From: Pavan T C [mailto:t...@gluster.com]
Sent: Wednesday, 14 September 2011 9:19 PM
To: Thomas Jackson
Cc: gluster-users@gluster.org
Subject: Re: [Gluster-users] Slow performance - 4 hosts, 10 gigabit
ethernet, Gluster 3.2.3
On Friday 09 September 2011 10:30 AM, Thomas Jackson wrote:
Hi everyone,
Hello Thomas,
Try the following:
1. In the fuse volume file, try:
Under write-behind:
option cache-size 16MB
Under read-ahead:
option page-count 16
Under io-cache:
option cache-size=64MB
TJ: Results here are not pretty!
root@my-host:~# dd if=/dev/zero of=/mnt/cluster-volume/test.file
bs=1M count=1
1048576 bytes (10 GB) copied, 107.888 s, 97.2 MB/s
2. Did you get 9Gbits/Sec with iperf with a single thread or multiple
threads?
TJ: Single thread
3. Can you give me the output of:
sysctl -a | egrep 'rmem|wmem'
TJ: root@my-host:~# sysctl -a | egrep 'rmem|wmem'
error: permission denied on key 'vm.compact_memory'
vm.lowmem_reserve_ratio = 256 256 32
error: permission denied on key 'net.ipv4.route.flush'
net.core.wmem_max = 131071
net.core.rmem_max = 131071
net.core.wmem_default = 126976
net.core.rmem_default = 126976
net.ipv4.tcp_wmem = 409616384 4194304
net.ipv4.tcp_rmem = 409687380 4194304
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_wmem_min = 4096
error: permission denied on key 'net.ipv6.route.flush'
4. If it is not a problem for you, can you please create a pure distribute
setup (instead of distributed-replicate) and then report the numbers?
TJ: I've been able to do this with 2 hosts, while I was at it I also tested
a pure replica and pure stripe setup for comparison
Distribute = 313 MB/sec
Replica = 166 MB/sec
Stripe = 529 MB/sec
5. What is the inode size with which you formatted you XFS filesystem ?
This last point might not be related to your throughput problem, but if
you are planning to use this setup for a large number of files,
you might be better off using an inode size of 512 instead of the default
256 bytes. To do that, your mkfs command should be:
mkfs -t xfs -i size=512 /dev/disk device
TJ: This is destined for use with VM images, probably a maximum of 200 files
total. That said, I have tried with a bigger inode size and also with ext4
with very similar results each time
In a totally bizarre turn of events, turning on port bonding (each host has
2x 10gig storage ports) in ACTIVE/BACKUP mode has increased the speed a fair
bit
dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=4
4194304 bytes (42 GB) copied, 176.459 s, 238 MB/s
I have noticed that the inodes are getting very frequently locked/unlocked
by the afs from some brief debugging, not sure if that is related
Pavan
I am seeing slower-than-expected performance in Gluster 3.2.3 between
4 hosts with 10 gigabit eth between them all. Each host has 4x 300GB
SAS 15K drives in RAID10, 6-core Xeon E5645 @ 2.40GHz and 24GB RAM
running Ubuntu
10.04 64-bit (I have also tested with Scientific Linux 6.1 and Debian
Squeeze - same results on those as well). All of the hosts mount the
volume using the FUSE module. The base filesystem on all of the nodes
is XFS, however tests with ext4 have yielded similar results.
Command used to create the volume:
gluster volume create cluster-volume replica 2 transport tcp
node01:/mnt/local-store/ node02:/mnt/local-store/
node03:/mnt/local-store/ node04:/mnt/local-store/
Command used to mount the Gluster volume on each node:
mount -t glusterfs localhost:/cluster-volume /mnt/cluster-volume
Creating a 40GB file onto a node's local storage (ie no Gluster
involvement):
dd if=/dev/zero of=/mnt/local-store/test.file bs=1M count=4
4194304 bytes (42 GB) copied, 92.9264 s, 451 MB/s
Getting the same file off the node's local storage:
dd if=/mnt/local-store/test.file of=/dev/null
4194304 bytes (42 GB) copied, 81.858 s, 512 MB/s
40GB file onto the Gluster storage:
dd if=/dev/zero of=/mnt/cluster-volume/test.file bs=1M count=4
4194304 bytes (42 GB) copied, 226.934 s, 185 MB/s
Getting the same file off the Gluster storage
dd if=/mnt/cluster-volume/test.file of=/dev/null
4194304 bytes (42 GB) copied, 661.561 s, 63.4 MB/s
I have also tried using Gluster 3.1, with similar results.
According to the Gluster docs, I should be seeing roughly the lesser
of the drive speed and the network speed. The network is able to push
0.9GB/sec according to iperf so that definitely isn't a limiting
factor here, and each array is able to do 400-500MB/sec as per above
benchmarks. I've tried with/without jumbo frames as well, which doesn't
make any major difference.
The glusterfs process