Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, 2 Mar 2012, Bryan Whitehead wrote: I'd try putting all hostnames in /etc/hosts. Also, can you post ping times between each host ? They are in /etc/hosts. # ping6 -c3 alcippe PING alcippe(alcippe) 56 data bytes 64 bytes from alcippe: icmp_seq=1 ttl=64 time=0.160 ms 64 bytes from alcippe: icmp_seq=2 ttl=64 time=0.088 ms 64 bytes from alcippe: icmp_seq=3 ttl=64 time=0.150 ms --- alcippe ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.088/0.132/0.160/0.034 ms # ping6 -c3 aethra PING aethra(aethra.arcada.fi) 56 data bytes 64 bytes from aethra.arcada.fi: icmp_seq=1 ttl=64 time=0.154 ms 64 bytes from aethra.arcada.fi: icmp_seq=2 ttl=64 time=0.158 ms 64 bytes from aethra.arcada.fi: icmp_seq=3 ttl=64 time=0.164 ms --- aethra ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.154/0.158/0.164/0.015 ms # ping6 -c3 adraste PING adraste(adraste) 56 data bytes 64 bytes from adraste: icmp_seq=1 ttl=255 time=0.165 ms 64 bytes from adraste: icmp_seq=2 ttl=255 time=0.155 ms 64 bytes from adraste: icmp_seq=3 ttl=255 time=0.187 ms --- adraste ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 1998ms rtt min/avg/max/mdev = 0.155/0.169/0.187/0.013 ms As said before, I don't think there's a problem with the LAN. Trust me, I would know about it :) On Fri, Mar 2, 2012 at 8:55 AM, Harald Hannelius harald.hannel...@arcada.fi wrote: On Fri, 2 Mar 2012, Brian Candler wrote: On Fri, Mar 02, 2012 at 05:25:18PM +0200, Harald Hannelius wrote: I'll have to test with just a two-way replica, and see if I get better performance out of that. I'm gonna loose the capability to have one node at the other site then Ah... are these nodes separated by a WAN? Synchronous replication is pretty sensitive to latency. You might want to look at geo-replication instead (which I've not tested) No, it's a 1 Gbps LAN. The other site is within LAN-range. -- Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users -- Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, Mar 02, 2012 at 11:04:02AM +0200, Harald Hannelius wrote: This has probably been discussed before, but since I'm new on the list I hope You have patience with me. There are some tuning hints in this post: http://gluster.org/pipermail/gluster-users/2012-February/009584.html although mostly focussed on read performance. I don't know why your write performance is so slow. But to narrow it down, I suggest the following: - Create a single-node volume (non-replicated, non-distributed) a. Time writing to it via a glusterfs mount on the same node b. Time writing to it via a glusterfs mount from a different node If both are fast: then retest using a two-node replicated volume. If a is fast but b is slow: investigate your gigabit network If both are slow: try writing to your RAID1 volume locally # ls -lah /gluster/debtest.img -rw--- 1 root root 20G Mar 1 12:35 /gluster/debtest.img # du -ah /gluster/debtest.img 4.5G /gluster/debtest.img A sparse file, right? So another useful test would be a 4.5G regular file. Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, Mar 02, 2012 at 02:41:30PM +0200, Harald Hannelius wrote: So next is back to the four-node setup you had before. I would expect that to perform about the same. So would I expect too. But; # time dd if=/dev/zero bs=1M count=2 of=/gluster/testfile 2+0 records in 2+0 records out 2097152 bytes (21 GB) copied, 1058.22 s, 19.8 MB/s real 17m38.357s user 0m0.040s sys 0m12.501s Right, so we know: - replic of aethra and alcippe is fast - distrib/replic across all four nodes is slow So chopping further, what about: - replic of adraste and helen? This would show whether one of these nodes is at fault. At least I got double figure readings this time. Sometimes I get write speeds of 5-6 MB/s. Well, I'm a bit lost when you start talking about VMs. Is this a production environment, and you are doing these dd/cp tests *in addition* to the production load of VM traffic? Or are you doing tests on an unloaded system? Note: mail servers have a nasty habit of doing fsync() all the time, for every single received message. Tools which might be useful to observe the production load: iostat 1 # shows the count of I/O requests and KB read/written per second btrace /dev/sdb | grep ' [DC] ' # shows the actual I/O operations dispatched (D) and completed (C) # to the drive There are also gluster-layer tools but I've not tried them: http://download.gluster.com/pub/gluster/glusterfs/3.2/Documentation/AG/html/chap-Gluster_Administration_Guide-Monitor_Workload.html Regards, Brian. ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, 2 Mar 2012, Brian Candler wrote: On Fri, Mar 02, 2012 at 02:41:30PM +0200, Harald Hannelius wrote: So next is back to the four-node setup you had before. I would expect that to perform about the same. So would I expect too. But; # time dd if=/dev/zero bs=1M count=2 of=/gluster/testfile 2+0 records in 2+0 records out 2097152 bytes (21 GB) copied, 1058.22 s, 19.8 MB/s real17m38.357s user0m0.040s sys 0m12.501s Right, so we know: - replic of aethra and alcippe is fast - distrib/replic across all four nodes is slow So chopping further, what about: - replic of adraste and helen? The pattern for me starts to look like this; max-write-speed ~= link speed/nodes. Volume Name: test Type: Replicate Status: Started Number of Bricks: 2 Transport-type: tcp Bricks: Brick1: adraste:/data/single Brick2: helen:/data/single # time dd if=/dev/zero bs=1M count=1 of=/mnt/testfile 1+0 records in 1+0 records out 1048576 bytes (10 GB) copied, 195.816 s, 53.5 MB/s real3m15.821s user0m0.016s sys 0m8.169s This would show whether one of these nodes is at fault. At least I got double figure readings this time. Sometimes I get write speeds of 5-6 MB/s. Well, I'm a bit lost when you start talking about VMs. Is this a production environment, and you are doing these dd/cp tests *in addition* to the production load of VM traffic? Or are you doing tests on an unloaded system? I have some systems running in the background yes. They are not really production machines. Note: mail servers have a nasty habit of doing fsync() all the time, for every single received message. It looks like openldap's slapadd uses some kind of sync as well. The load-average on the KVM-host was up at 9.00 while slapadd was running. Tools which might be useful to observe the production load: iostat 1 # shows the count of I/O requests and KB read/written per second iotop is handy too. btrace /dev/sdb | grep ' [DC] ' # shows the actual I/O operations dispatched (D) and completed (C) # to the drive There are also gluster-layer tools but I've not tried them: http://download.gluster.com/pub/gluster/glusterfs/3.2/Documentation/AG/html/chap-Gluster_Administration_Guide-Monitor_Workload.html Regards, Brian. -- Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, 2 Mar 2012, Samuli Heinonen wrote: 2.3.2012 15:33, Harald Hannelius kirjoitti: The pattern for me starts to look like this; max-write-speed ~= link speed/nodes. Have you tried tuning performance.io-thread-count setting? More information about that can be found at http://docs.redhat.com/docs/en-US/Red_Hat_Storage_Software_Appliance/3.2/html/User_Guide/chap-User_Guide-Managing_Volumes.html Yes, as in a previous post; # gluster volume info Volume Name: virtuals Type: Distributed-Replicate Status: Started Number of Bricks: 2 x 2 = 4 Transport-type: tcp Bricks: Brick1: adraste:/data/brick Brick2: alcippe:/data/brick Brick3: aethra:/data/brick Brick4: helen:/data/brick Options Reconfigured: cluster.data-self-heal-algorithm: diff cluster.self-heal-window-size: 1 performance.io-thread-count: 64 performance.cache-size: 536870912 performance.write-behind-window-size: 16777216 performance.flush-behind: on -- Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, Mar 02, 2012 at 05:25:18PM +0200, Harald Hannelius wrote: I'll have to test with just a two-way replica, and see if I get better performance out of that. I'm gonna loose the capability to have one node at the other site then Ah... are these nodes separated by a WAN? Synchronous replication is pretty sensitive to latency. You might want to look at geo-replication instead (which I've not tested) ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
On Fri, 2 Mar 2012, Brian Candler wrote: On Fri, Mar 02, 2012 at 05:25:18PM +0200, Harald Hannelius wrote: I'll have to test with just a two-way replica, and see if I get better performance out of that. I'm gonna loose the capability to have one node at the other site then Ah... are these nodes separated by a WAN? Synchronous replication is pretty sensitive to latency. You might want to look at geo-replication instead (which I've not tested) No, it's a 1 Gbps LAN. The other site is within LAN-range. -- Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020 ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Re: [Gluster-users] Write performance in a replicated/distributed setup with KVM?
I'd try putting all hostnames in /etc/hosts. Also, can you post ping times between each host ? On Fri, Mar 2, 2012 at 8:55 AM, Harald Hannelius harald.hannel...@arcada.fi wrote: On Fri, 2 Mar 2012, Brian Candler wrote: On Fri, Mar 02, 2012 at 05:25:18PM +0200, Harald Hannelius wrote: I'll have to test with just a two-way replica, and see if I get better performance out of that. I'm gonna loose the capability to have one node at the other site then Ah... are these nodes separated by a WAN? Synchronous replication is pretty sensitive to latency. You might want to look at geo-replication instead (which I've not tested) No, it's a 1 Gbps LAN. The other site is within LAN-range. -- Harald Hannelius | harald.hannelius/a\arcada.fi | +358 50 594 1020 __**_ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/**mailman/listinfo/gluster-usershttp://gluster.org/cgi-bin/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
