I've done some more testing with tc and introduced latency on one of my testservers. With 9ms latency artificially introduced using tc ( sudo tc qdisc add dev bond0 root netem delay 9ms ) to a testserver in the same DC as the disperse volume servers I get more or less the same throughput as I do when testing DC1 <-> DC2 (which has ~9ms ping).
I know distribute volumes were more sensitive to latency in the past. At least I can max out a 1gig link with 9-10ms latency when using distribute. Disperse seems to max at 12-14MB/s with 8-10ms latency. ingard 2017-04-24 14:03 GMT+02:00 Ingard Mevåg <ing...@jotta.no>: > I can confirm mounting the disperse volume locally on one of the three > servers i got 211 MB/s with dd if=/dev/zero of=./local.dd.test bs=1M > count=10000. > > Its not very good concidering 10gig network, but at least 20x better than > 10-12MB/s > > 2017-04-24 13:53 GMT+02:00 Pranith Kumar Karampuri <pkara...@redhat.com>: > >> +Ashish >> >> Ashish, >> Could you help Ingard? Do let me know what you find. >> >> On Mon, Apr 24, 2017 at 4:50 PM, Ingard Mevåg <ing...@jotta.no> wrote: >> >>> Hi. I can't see a fuse thread at all. Please see attached screenshot of >>> top process with threads. Keep in mind this is from inside the container. >>> >>> 2017-04-24 12:17 GMT+02:00 Pranith Kumar Karampuri <pkara...@redhat.com> >>> : >>> >>>> We were able to saturate hardware with EC as well. Could you check >>>> 'top' in threaded mode to see if fuse thread is saturated when you run dd? >>>> >>>> On Mon, Apr 24, 2017 at 3:27 PM, Ingard Mevåg <ing...@jotta.no> wrote: >>>> >>>>> Hi >>>>> I've been playing with disperse volumes the past week, and so far i >>>>> can not get more than 12MB/s when i do a write test. I've tried a >>>>> distributed volume on the same bricks and gotten close to gigabit speeds. >>>>> iperf confirms gigabit speeds to all three servers in the storage pool. >>>>> >>>>> The three storage servers have 10gig nics (connected to the same >>>>> switch). The client is for a now a docker container in a 2nd DC (latency >>>>> roughly 8-9 ms). >>>>> >>>>> dpkg -l|grep -i gluster >>>>> ii glusterfs-client 3.10.1-ubuntu1~xenial1 >>>>> amd64 clustered file-system (client package) >>>>> ii glusterfs-common 3.10.1-ubuntu1~xenial1 >>>>> amd64 GlusterFS common libraries and translator modules >>>>> ii glusterfs-server 3.10.1-ubuntu1~xenial1 >>>>> amd64 clustered file-system (server package) >>>>> >>>>> $ gluster volume info >>>>> >>>>> Volume Name: DFS-ARCHIVE-001 >>>>> Type: Disperse >>>>> Volume ID: 1497bc85-cb47-4123-8f91-a07f55c11dcc >>>>> Status: Started >>>>> Snapshot Count: 0 >>>>> Number of Bricks: 1 x (4 + 2) = 6 >>>>> Transport-type: tcp >>>>> Bricks: >>>>> Brick1: dna-001:/mnt/data01/brick >>>>> Brick2: dna-001:/mnt/data02/brick >>>>> Brick3: dna-002:/mnt/data01/brick >>>>> Brick4: dna-002:/mnt/data02/brick >>>>> Brick5: dna-003:/mnt/data01/brick >>>>> Brick6: dna-003:/mnt/data02/brick >>>>> Options Reconfigured: >>>>> transport.address-family: inet >>>>> nfs.disable: on >>>>> >>>>> Anyone know the reason for the slow speeds on disperse vs distribute? >>>>> >>>>> kind regards >>>>> ingard >>>>> >>>>> _______________________________________________ >>>>> Gluster-users mailing list >>>>> Gluster-users@gluster.org >>>>> http://lists.gluster.org/mailman/listinfo/gluster-users >>>>> >>>> >>>> >>>> >>>> -- >>>> Pranith >>>> >>> >>> >>> >>> -- >>> Ingard Mevåg >>> Driftssjef >>> Jottacloud >>> >>> Mobil: +47 450 22 834 <+47%20450%2022%20834> >>> E-post: ing...@jottacloud.com >>> Webside: www.jottacloud.com >>> >> >> >> >> -- >> Pranith >> > > > > -- > Ingard Mevåg > Driftssjef > Jottacloud > > Mobil: +47 450 22 834 <+47%20450%2022%20834> > E-post: ing...@jottacloud.com > Webside: www.jottacloud.com > -- Ingard Mevåg Driftssjef Jottacloud Mobil: +47 450 22 834 E-post: ing...@jottacloud.com Webside: www.jottacloud.com
_______________________________________________ Gluster-users mailing list Gluster-users@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-users