date:20170711

Re: [Gluster-users] set owner:group on root of volume

2017-07-11 Thread mabi

Just found out I needed to set following two parameters:
gluster volume set myvol storage.owner-uid 1000
gluster volume set myvol storage.owner-gid 1000
In case that helps any one else :)

>  Original Message 
> Subject: set owner:group on root of volume
> Local Time: July 11, 2017 8:15 PM
> UTC Time: July 11, 2017 6:15 PM
> From: m...@protonmail.ch
> To: Gluster Users 
> Hi,
> By default the owner and group of a GlusterFS seems to be root:root now I 
> changed this by first mounting my volume using glusterfs/fuse on a client and 
> did the following
> chmod 1000:1000 /mnt/myglustervolume
> This changed correctly the owner and group to UID/GID 1000 of my volume but 
> like 1-2 hours later it was back to root:root. I tried again and this happens 
> again.
> Am I doing something wrong here? I am using GlusterFS 3.8.11 on Debian 8.
> Regards,
> M.___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] set owner:group on root of volume

2017-07-11 Thread mabi

Hi,
By default the owner and group of a GlusterFS seems to be root:root now I 
changed this by first mounting my volume using glusterfs/fuse on a client and 
did the following
chmod 1000:1000 /mnt/myglustervolume
This changed correctly the owner and group to UID/GID 1000 of my volume but 
like 1-2 hours later it was back to root:root. I tried again and this happens 
again.
Am I doing something wrong here? I am using GlusterFS 3.8.11 on Debian 8.
Regards,
M.___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

PS: I just tested between these 2:

 
mount -t glusterfs -o 
negative-timeout=1,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log
 192.168.140.41:/www /var/www
 mount -t glusterfs -o 
use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log 
192.168.140.41:/www /var/www
 So it means only 1 second negative timeout...
 In this particular test: ./smallfile_cli.py  --top /var/www/test --host-set 
192.168.140.41 --threads 8 --files 5 --file-size 64 --record-size 64

  The result is about 4 seconds with the negative timeout of 1 second defined 
and many many minutes without the negative timeout (I quit after 15 minutes of 
waiting)
 I will go over to some real world tests now to see how it performs there.
  Regards
Jo

 
 

 
-Original message-
From:Jo Goossens 
Sent:Tue 11-07-2017 18:23
Subject:Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To:Vijay Bellur ; 
CC:gluster-users@gluster.org; 
 

Hello Vijay,

 
 
What do you mean exactly? What info is missing?

 
PS: I already found out that for this particular test all the difference is 
made by : negative-timeout=600 , when removing it, it's much much slower again.

 
 
Regards

Jo
 
-Original message-
From:Vijay Bellur 
Sent:Tue 11-07-2017 18:16
Subject:Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To:Jo Goossens ; 
CC:gluster-users@gluster.org; Joe Julian ; 
 


On Tue, Jul 11, 2017 at 11:39 AM, Jo Goossens  > wrote:
 

Hello Joe,

 
 
I just did a mount like this (added the bold):

 
mount -t glusterfs -o 
attribute-timeout=600,entry-timeout=600,negative-timeout=600,fopen-keep-cache,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log
 192.168.140.41:/www /var/www
 Results:

 
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 5000 --file-size 64 --record-size 
64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5000
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 1.232004,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 1.148738,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 1.130913,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 1.183088,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 1.220752,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 1.228039,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 1.216787,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 1.229036,files = 5000,records = 
0,status = ok
total threads = 8
total files = 4
100.00% of requested files processed, minimum is  70.00
1.232004 sec elapsed time
32467.428972 files/sec
  
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 5 --file-size 64 
--record-size 64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hello Vijay,

 
 
What do you mean exactly? What info is missing?

 
PS: I already found out that for this particular test all the difference is 
made by : negative-timeout=600 , when removing it, it's much much slower again.

 
 
Regards

Jo
 
-Original message-
From:Vijay Bellur 
Sent:Tue 11-07-2017 18:16
Subject:Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To:Jo Goossens ; 
CC:gluster-users@gluster.org; Joe Julian ; 
 


On Tue, Jul 11, 2017 at 11:39 AM, Jo Goossens  > wrote:
 

Hello Joe,

 
 
I just did a mount like this (added the bold):

 
mount -t glusterfs -o 
attribute-timeout=600,entry-timeout=600,negative-timeout=600,fopen-keep-cache,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log
 192.168.140.41:/www /var/www
 Results:

 
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 5000 --file-size 64 --record-size 
64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5000
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 1.232004,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 1.148738,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 1.130913,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 1.183088,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 1.220752,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 1.228039,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 1.216787,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 1.229036,files = 5000,records = 
0,status = ok
total threads = 8
total files = 4
100.00% of requested files processed, minimum is  70.00
1.232004 sec elapsed time
32467.428972 files/sec
  
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 5 --file-size 64 
--record-size 64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host =

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Vijay Bellur

On Tue, Jul 11, 2017 at 11:39 AM, Jo Goossens 
wrote:

> Hello Joe,
>
>
>
>
>
> I just did a mount like this (added the bold):
>
>
> mount -t glusterfs -o
> *attribute-timeout=600,entry-timeout=600,negative-timeout=600,fopen-keep-cache*
> ,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log
> 192.168.140.41:/www /var/www
>
>
> Results:
>
>
> root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test
> --host-set 192.168.140.41 --threads 8 --files 5000 --file-size 64
> --record-size 64
> smallfile version 3.0
>hosts in test : ['192.168.140.41']
>top test directory(s) : ['/var/www/test']
>operation : cleanup
> files/thread : 5000
>  threads : 8
>record size (KB, 0 = maximum) : 64
>   file size (KB) : 64
>   file size distribution : fixed
>files per dir : 100
> dirs per dir : 10
>   threads share directories? : N
>  filename prefix :
>  filename suffix :
>  hash file number into dir.? : N
>  fsync after modify? : N
>   pause between files (microsec) : 0
> finish all requests? : Y
>   stonewall? : Y
>  measure response times? : N
> verify read? : Y
> verbose? : False
>   log to stderr? : False
>ext.attr.size : 0
>   ext.attr.count : 0
>permute host directories? : N
> remote program directory : /root/smallfile-master
>network thread sync. dir. : /var/www/test/network_shared
> starting all threads by creating starting gate file
> /var/www/test/network_shared/starting_gate.tmp
> host = 192.168.140.41,thr = 00,elapsed = 1.232004,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 01,elapsed = 1.148738,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 02,elapsed = 1.130913,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 03,elapsed = 1.183088,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 04,elapsed = 1.220752,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 05,elapsed = 1.228039,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 06,elapsed = 1.216787,files = 5000,records =
> 0,status = ok
> host = 192.168.140.41,thr = 07,elapsed = 1.229036,files = 5000,records =
> 0,status = ok
> total threads = 8
> total files = 4
> 100.00% of requested files processed, minimum is  70.00
> 1.232004 sec elapsed time
> 32467.428972 files/sec
>
>
>
> root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test
> --host-set 192.168.140.41 --threads 8 --files 5 --file-size 64
> --record-size 64
> smallfile version 3.0
>hosts in test : ['192.168.140.41']
>top test directory(s) : ['/var/www/test']
>operation : cleanup
> files/thread : 5
>  threads : 8
>record size (KB, 0 = maximum) : 64
>   file size (KB) : 64
>   file size distribution : fixed
>files per dir : 100
> dirs per dir : 10
>   threads share directories? : N
>  filename prefix :
>  filename suffix :
>  hash file number into dir.? : N
>  fsync after modify? : N
>   pause between files (microsec) : 0
> finish all requests? : Y
>   stonewall? : Y
>  measure response times? : N
> verify read? : Y
> verbose? : False
>   log to stderr? : False
>ext.attr.size : 0
>   ext.attr.count : 0
>permute host directories? : N
> remote program directory : /root/smallfile-master
>network thread sync. dir. : /var/www/test/network_shared
> starting all threads by creating starting gate file
> /var/www/test/network_shared/starting_gate.tmp
> host = 192.168.140.41,thr = 00,elapsed = 4.242312,files = 5,records =
> 0,status = ok
> host = 192.168.140.41,thr = 01,elapsed = 4.250831,files = 5,records =
> 0,status = ok
> host = 192.168.140.41,thr = 02,elapsed = 3.771269,files = 5,records =
> 0,status = ok
> host = 192.168.140.41,thr = 03,elapsed = 4.060653,files = 5,records =
> 0,status = ok
> host = 192.168.140.41,thr =

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hello Joe,

 
 
I just did a mount like this (added the bold):

 
mount -t glusterfs -o 
attribute-timeout=600,entry-timeout=600,negative-timeout=600,fopen-keep-cache,use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log
 192.168.140.41:/www /var/www
 Results:

 
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 5000 --file-size 64 --record-size 
64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5000
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 1.232004,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 1.148738,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 1.130913,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 1.183088,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 1.220752,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 1.228039,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 1.216787,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 1.229036,files = 5000,records = 
0,status = ok
total threads = 8
total files = 4
100.00% of requested files processed, minimum is  70.00
1.232004 sec elapsed time
32467.428972 files/sec
  
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 5 --file-size 64 
--record-size 64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 4.242312,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 4.250831,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 3.771269,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 4.060653,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 3.880653,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 3.847107,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 3.895537,files = 5,records = 
0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 3.966394,files

Re: [Gluster-users] Extremely slow du

2017-07-11 Thread Jo Goossens

Hi,

 
 
I also noticed disappearing of files with the combination of certain settings. 
If you use cluster.readdir-optimize but not some of the other settings, they 
don't disappear.

 
Unfortunately can't remember which setting was conflicting...

 
 
Performance wise I don't see difference between 3.10 and 3.11 over here. Didn't 
test recently with 3.8.



Regards

Jo

 

 
-Original message-
From:Vijay Bellur 
Sent:Tue 11-07-2017 17:22
Subject:Re: [Gluster-users] Extremely slow du
To:mohammad kashif ; Raghavendra Gowdappa 
; Poornima Gurusiddaiah ; 
CC:gluster-users Discussion List ; 
Hi Kashif,

Thank you for your feedback! Do you have some data on the nature of 
performance improvement observed with 3.11 in the new setup?

Adding Raghavendra and Poornima for validation of configuration and help 
with identifying why certain files disappeared from the mount point 
after enabling readdir-optimize.

Regards,
Vijay

On 07/11/2017 11:06 AM, mohammad kashif wrote:
> Hi Vijay and Experts
>
> I didn't want to experiment with my production setup so started  a
> parallel system with two server and around 80TB storage.  First
> configured with gluster 3.8 and had the same lookup performance issue.
> Then upgraded to 3.11 as you suggested and it made huge improvement in
> lookup time. I also did some more optimization as suggested in other
> threads.
> Now I am going to update my production server. I am planning to use
> following  optimization option, it would be very useful if you can point
> out any inconsistency or suggest some other options. My production setup
> has 5 servers consisting of  400TB storage and around 80 million files
> of varying lengths.
>
> Options Reconfigured:
> server.event-threads: 4
> client.event-threads: 4
> cluster.lookup-optimize: on
> cluster.readdir-optimize: off
> performance.client-io-threads: on
> performance.cache-size: 1GB
> performance.parallel-readdir: on
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> nfs.disable: on
> performance.readdir-ahead: on
> transport.address-family: inet
> auth.allow: 163.1.136.*
> diagnostics.latency-measurement: on
> diagnostics.count-fop-hits: on
>
> I found that setting cluster.readdir-optimize to 'on' made some files
> disappear from client !
>
> Thanks
>
> Kashif
>
>
>
> On Sun, Jun 18, 2017 at 4:57 PM, Vijay Bellur  > wrote:
>
>     Hi Mohammad,
>
>     A lot of time is being spent in addressing metadata calls as
>     expected. Can you consider testing out with 3.11 with md-cache [1]
>     and readdirp [2] improvements?
>
>     Adding Poornima and Raghavendra who worked on these enhancements to
>     help out further.
>
>     Thanks,
>     Vijay
>
>     [1] https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/
>     
>
>     [2] https://github.com/gluster/glusterfs/issues/166
>     
>
>     On Fri, Jun 16, 2017 at 2:49 PM, mohammad kashif
>     > wrote:
>
>         Hi Vijay
>
>         Did you manage to look into the gluster profile logs ?
>
>         Thanks
>
>         Kashif
>
>         On Mon, Jun 12, 2017 at 11:40 AM, mohammad kashif
>         > wrote:
>
>             Hi Vijay
>
>             I have enabled client profiling and used this script
>             
> https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh
>             
> 
>             to extract data. I am attaching output files. I don't have
>             any reference data to compare with my output. Hopefully you
>             can make some sense out of it.
>
>             On Sat, Jun 10, 2017 at 10:47 AM, Vijay Bellur
>             > wrote:
>
>                 Would it be possible for you to turn on client profiling
>                 and then run du? Instructions for turning on client
>                 profiling can be found at [1]. Providing the client
>                 profile information can help us figure out where the
>                 latency could be stemming from.
>
>                 Regards,
>                 Vijay
>
>                 [1] 
> https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling
>                 
> 
>
>                 On Fri, Jun 9, 2017 at 7:22 PM, mohammad kashif
>

Re: [Gluster-users] Extremely slow du

2017-07-11 Thread Vijay Bellur

Hi Kashif,

Thank you for your feedback! Do you have some data on the nature of
performance improvement observed with 3.11 in the new setup?

Adding Raghavendra and Poornima for validation of configuration and help
with identifying why certain files disappeared from the mount point
after enabling readdir-optimize.

Regards,
Vijay

On 07/11/2017 11:06 AM, mohammad kashif wrote:

Hi Vijay and Experts

I didn't want to experiment with my production setup so started a
parallel system with two server and around 80TB storage. First
configured with gluster 3.8 and had the same lookup performance issue.
Then upgraded to 3.11 as you suggested and it made huge improvement in
lookup time. I also did some more optimization as suggested in other
threads.
Now I am going to update my production server. I am planning to use
following optimization option, it would be very useful if you can point
out any inconsistency or suggest some other options. My production setup
has 5 servers consisting of 400TB storage and around 80 million files
of varying lengths.

Options Reconfigured:
server.event-threads: 4
client.event-threads: 4
cluster.lookup-optimize: on
cluster.readdir-optimize: off
performance.client-io-threads: on
performance.cache-size: 1GB
performance.parallel-readdir: on
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
nfs.disable: on
performance.readdir-ahead: on
transport.address-family: inet
auth.allow: 163.1.136.*
diagnostics.latency-measurement: on
diagnostics.count-fop-hits: on

I found that setting cluster.readdir-optimize to 'on' made some files
disappear from client !

Thanks

Kashif

On Sun, Jun 18, 2017 at 4:57 PM, Vijay Bellur > wrote:

Hi Mohammad,

A lot of time is being spent in addressing metadata calls as
expected. Can you consider testing out with 3.11 with md-cache [1]
and readdirp [2] improvements?

Adding Poornima and Raghavendra who worked on these enhancements to
help out further.

Thanks,
Vijay

[1] https://gluster.readthedocs.io/en/latest/release-notes/3.9.0/

[2] https://github.com/gluster/glusterfs/issues/166

On Fri, Jun 16, 2017 at 2:49 PM, mohammad kashif
> wrote:

Hi Vijay

Did you manage to look into the gluster profile logs ?

Thanks

Kashif

On Mon, Jun 12, 2017 at 11:40 AM, mohammad kashif
> wrote:

Hi Vijay

I have enabled client profiling and used this script

https://github.com/bengland2/gluster-profile-analysis/blob/master/gvp-client.sh

to extract data. I am attaching output files. I don't have
any reference data to compare with my output. Hopefully you
can make some sense out of it.

On Sat, Jun 10, 2017 at 10:47 AM, Vijay Bellur
> wrote:

Would it be possible for you to turn on client profiling
and then run du? Instructions for turning on client
profiling can be found at [1]. Providing the client
profile information can help us figure out where the
latency could be stemming from.

Regards,
Vijay

[1]
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Performance%20Testing/#client-side-profiling

On Fri, Jun 9, 2017 at 7:22 PM, mohammad kashif
>
wrote:

Hi Vijay

Thanks for your quick response. I am using gluster
3.8.11 on Centos 7 servers
glusterfs-3.8.11-1.el7.x86_64

clients are centos 6 but I tested with a centos 7
client as well and results didn't change

gluster volume info Volume Name: atlasglust
Type: Distribute
Volume ID: fbf0ebb8-deab-4388-9d8a-f722618a624b
Status: Started
Snapshot Count: 0
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: pplxgluster01.x.y.z:/glusteratlas/brick001/gv0
Brick2: pplxgluster02..x.y.z:/glusteratlas/brick002/gv0

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Joe Julian


On 07/11/2017 08:14 AM, Jo Goossens wrote:

RE: [Gluster-users] Gluster native mount is really slow compared to nfs

Hello Joe,

I really appreciate your feedback, but I already tried the opcache 
stuff (to not valildate at all). It improves of course then, but not 
completely somehow. Still quite slow.


I did not try the mount options yet, but I will now!

With nfs (doesnt matter much built-in version 3 or ganesha version 4) 
I can even host the site perfectly fast without these extreme opcache 
settings.


I still can't understand why the nfs mount is easily 80 times faster, 
actually no matter what options I set it seems. It's almost there is 
something really wrong somehow...




Because the linux nfs client doesn't touch the network for many 
operations and are, in stead, held in the kernel's FSCache.


I tried the ceph mount now and out of the box it's comparable with 
gluster with nfs mount.


Regards

Jo

BE: +32 53 599 000

NL: +31 85 888 4 555

https://www.hosted-power.com/


-Original message-
*From:* Joe Julian 
*Sent:* Tue 11-07-2017 17:04
*Subject:* Re: [Gluster-users] Gluster native mount is really slow
compared to nfs
*To:* gluster-users@gluster.org;

My standard response to someone needing filesystem performance for
www traffic is generally, "you're doing it wrong".
https://joejulian.name/blog/optimizing-web-performance-with-glusterfs/

That said, you might also look at these mount options:
attribute-timeout, entry-timeout, negative-timeout (set to some
large amount of time), and fopen-keep-cache.


On 07/11/2017 07:48 AM, Jo Goossens wrote:


Hello,

Here is the volume info as requested by soumya:

#gluster volume info www
Volume Name: www
Type: Replicate
Volume ID: 5d64ee36-828a-41fa-adbf-75718b954aff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.140.41:/gluster/www
Brick2: 192.168.140.42:/gluster/www
Brick3: 192.168.140.43:/gluster/www
Options Reconfigured:
cluster.read-hash-mode: 0
performance.quick-read: on
performance.write-behind-window-size: 4MB
server.allow-insecure: on
performance.read-ahead: disable
performance.readdir-ahead: on
performance.io-thread-count: 64
performance.io-cache: on
performance.client-io-threads: on
server.outstanding-rpc-limit: 128
server.event-threads: 3
client.event-threads: 3
performance.cache-size: 32MB
transport.address-family: inet
nfs.disable: on
nfs.addr-namelookup: off
nfs.export-volumes: on
nfs.rpc-auth-allow: 192.168.140.*
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-samba-metadata: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 10
performance.parallel-readdir: on
performance.cache-refresh-timeout: 60
performance.rda-cache-limit: 50MB
cluster.nufa: on
network.ping-timeout: 5
cluster.lookup-optimize: on
cluster.quorum-type: auto
I started with none of them set and I added/changed while
testing. But it was always slow, by tuning some kernel parameters
it improved slightly (just a few percent, nothing reasonable)
I also tried ceph just to compare, I got this with default
settings and no tweaks:
 ./smallfile_cli.py  --top /var/www/test --host-set
192.168.140.41 --threads 8 --files 5000 --file-size 64
--record-size 64
smallfile version 3.0
   hosts in test : ['192.168.140.41']
   top test directory(s) : ['/var/www/test']
   operation : cleanup
files/thread : 5000
 threads : 8
   record size (KB, 0 = maximum) : 64
  file size (KB) : 64
  file size distribution : fixed
   files per dir : 100
dirs per dir : 10
  threads share directories? : N
 filename prefix :
 filename suffix :
 hash file number into dir.? : N
 fsync after modify? : N
  pause between files (microsec) : 0
finish all requests? : Y
  stonewall? : Y
 measure response times? : N
verify read? : Y
verbose? : False
  log to stderr? : False
   ext.attr.size : 0
  ext.attr.count : 0
   permute host directories? : N
remote program directory : /root/smallfile-master

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hello Joe,

 
 
I really appreciate your feedback, but I already tried the opcache stuff (to 
not valildate at all). It improves of course then, but not completely somehow. 
Still quite slow.

 
I did not try the mount options yet, but I will now!

 
 
With nfs (doesnt matter much built-in version 3 or ganesha version 4) I can 
even host the site perfectly fast without these extreme opcache settings.

 
I still can't understand why the nfs mount is easily 80 times faster, actually 
no matter what options I set it seems. It's almost there is something really 
wrong somehow...

 
I tried the ceph mount now and out of the box it's comparable with gluster with 
nfs mount.

 
 
Regards

Jo

 
BE: +32 53 599 000

NL: +31 85 888 4 555

 
https://www.hosted-power.com/

 

 
-Original message-
From:Joe Julian 
Sent:Tue 11-07-2017 17:04
Subject:Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To:gluster-users@gluster.org; 
 

My standard response to someone needing filesystem performance for www traffic 
is generally, "you're doing it wrong". 
https://joejulian.name/blog/optimizing-web-performance-with-glusterfs/
 
 That said, you might also look at these mount options: attribute-timeout, 
entry-timeout, negative-timeout (set to some large amount of time), and 
fopen-keep-cache.

 
 
On 07/11/2017 07:48 AM, Jo Goossens wrote:
 

Hello,

 

 
 

 
 

Here is the volume info as requested by soumya:

 

 
 
#gluster volume info www
 
  
Volume Name: www
 
Type: Replicate
 
Volume ID: 5d64ee36-828a-41fa-adbf-75718b954aff
 
Status: Started
 
Snapshot Count: 0
 
Number of Bricks: 1 x 3 = 3
 
Transport-type: tcp
 
Bricks:
 
Brick1: 192.168.140.41:/gluster/www
 
Brick2: 192.168.140.42:/gluster/www
 
Brick3: 192.168.140.43:/gluster/www
 
Options Reconfigured:
 
cluster.read-hash-mode: 0
 
performance.quick-read: on
 
performance.write-behind-window-size: 4MB
 
server.allow-insecure: on
 
performance.read-ahead: disable
 
performance.readdir-ahead: on
 
performance.io-thread-count: 64
 
performance.io-cache: on
 
performance.client-io-threads: on
 
server.outstanding-rpc-limit: 128
 
server.event-threads: 3
 
client.event-threads: 3
 
performance.cache-size: 32MB
 
transport.address-family: inet
 
nfs.disable: on
 
nfs.addr-namelookup: off
 
nfs.export-volumes: on
 
nfs.rpc-auth-allow: 192.168.140.*
 
features.cache-invalidation: on
 
features.cache-invalidation-timeout: 600
 
performance.stat-prefetch: on
 
performance.cache-samba-metadata: on
 
performance.cache-invalidation: on
 
performance.md-cache-timeout: 600
 
network.inode-lru-limit: 10
 
performance.parallel-readdir: on
 
performance.cache-refresh-timeout: 60
 
performance.rda-cache-limit: 50MB
 
cluster.nufa: on
 
network.ping-timeout: 5
 
cluster.lookup-optimize: on
 
cluster.quorum-type: auto
 
  
I started with none of them set and I added/changed while testing. But it was 
always slow, by tuning some kernel parameters it improved slightly (just a few 
percent, nothing reasonable)
 
  
I also tried ceph just to compare, I got this with default settings and no 
tweaks:
 
  
 ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 
--files 5000 --file-size 64 --record-size 64
 
smallfile version 3.0
 
                           hosts in test : ['192.168.140.41']
 
                   top test directory(s) : ['/var/www/test']
 
                               operation : cleanup
 
                            files/thread : 5000
 
                                 threads : 8
 
           record size (KB, 0 = maximum) : 64
 
                          file size (KB) : 64
 
                  file size distribution : fixed
 
                           files per dir : 100
 
                            dirs per dir : 10
 
              threads share directories? : N
 
                         filename prefix :
 
                         filename suffix :
 
             hash file number into dir.? : N
 
                     fsync after modify? : N
 
          pause between files (microsec) : 0
 
                    finish all requests? : Y
 
                              stonewall? : Y
 
                 measure response times? : N
 
                            verify read? : Y
 
                                verbose? : False
 
                          log to stderr? : False
 
                           ext.attr.size : 0
 
                          ext.attr.count : 0
 
               permute host directories? : N
 
                remote program directory : /root/smallfile-master
 
               network thread sync. dir. : /var/www/test/network_shared
 
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
 
host = 192.168.140.41,thr = 00,elapsed = 1.339621,files = 5000,records = 
0,status = ok
 
host = 192.168.140.41,thr = 01,elapsed = 1.436776,files = 5000,records = 
0,status = ok
 
host = 192.168.140.41,thr = 02,elapsed = 1.498681,files =

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Joe Julian

My standard response to someone needing filesystem performance for www 
traffic is generally, "you're doing it wrong". 
https://joejulian.name/blog/optimizing-web-performance-with-glusterfs/


That said, you might also look at these mount options: 
attribute-timeout, entry-timeout, negative-timeout (set to some large 
amount of time), and fopen-keep-cache.



On 07/11/2017 07:48 AM, Jo Goossens wrote:

RE: [Gluster-users] Gluster native mount is really slow compared to nfs

Hello,

Here is the volume info as requested by soumya:

#gluster volume info www
Volume Name: www
Type: Replicate
Volume ID: 5d64ee36-828a-41fa-adbf-75718b954aff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.140.41:/gluster/www
Brick2: 192.168.140.42:/gluster/www
Brick3: 192.168.140.43:/gluster/www
Options Reconfigured:
cluster.read-hash-mode: 0
performance.quick-read: on
performance.write-behind-window-size: 4MB
server.allow-insecure: on
performance.read-ahead: disable
performance.readdir-ahead: on
performance.io-thread-count: 64
performance.io-cache: on
performance.client-io-threads: on
server.outstanding-rpc-limit: 128
server.event-threads: 3
client.event-threads: 3
performance.cache-size: 32MB
transport.address-family: inet
nfs.disable: on
nfs.addr-namelookup: off
nfs.export-volumes: on
nfs.rpc-auth-allow: 192.168.140.*
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-samba-metadata: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 10
performance.parallel-readdir: on
performance.cache-refresh-timeout: 60
performance.rda-cache-limit: 50MB
cluster.nufa: on
network.ping-timeout: 5
cluster.lookup-optimize: on
cluster.quorum-type: auto
I started with none of them set and I added/changed while testing. But 
it was always slow, by tuning some kernel parameters it improved 
slightly (just a few percent, nothing reasonable)
I also tried ceph just to compare, I got this with default settings 
and no tweaks:
 ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 
--threads 8 --files 5000 --file-size 64 --record-size 64

smallfile version 3.0
   hosts in test : ['192.168.140.41']
   top test directory(s) : ['/var/www/test']
   operation : cleanup
files/thread : 5000
 threads : 8
   record size (KB, 0 = maximum) : 64
  file size (KB) : 64
  file size distribution : fixed
   files per dir : 100
dirs per dir : 10
  threads share directories? : N
 filename prefix :
 filename suffix :
 hash file number into dir.? : N
 fsync after modify? : N
  pause between files (microsec) : 0
finish all requests? : Y
  stonewall? : Y
 measure response times? : N
verify read? : Y
verbose? : False
  log to stderr? : False
   ext.attr.size : 0
  ext.attr.count : 0
   permute host directories? : N
remote program directory : /root/smallfile-master
   network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 1.339621,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 1.436776,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 1.498681,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 1.483886,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 1.454833,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 1.469340,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 1.439060,files = 5000,records 
= 0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 1.375074,files = 5000,records 
= 0,status = ok

total threads = 8
total files = 4
100.00% of requested files processed, minimum is  70.00
1.498681 sec elapsed time
26690.134975 files/sec


Regards

Jo

-Original message-
*From:* Jo Goossens 
*Sent:* Tue 11-07-2017 12:15
*Subject:* Re: [Gluster-users] Gluster native mount is really slow
compared to nfs
*To:* Soumya Koduri ; gluster-users@gluster.org;
*CC:* Ambarish Soman ;

Hello,

Here is some speedtest with a new setup we just made with gluster
3.10, there are no other differences, except glusterfs versus

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread lemonnierk

Hi,

We've been doing that for some clients, basically it works fine if you
configure your OPCache very very agressivly. Increase the available ram
for it, disable any form of opcache validating from disk and it'll work
great, 'cause your app won't touch gluster.
Then whenever you make a change in the PHP, just restart PHP to force
it to reload the source from gluster.
For example :

zend_extension = opcache.so

[opcache]
opcache.enable = 1
opcache.enable_cli = 1
opcache.memory_consumption = 1024
opcache.max_accelerated_files = 8
opcache.revalidate_freq = 300
opcache.validate_timestamps = 1
opcache.interned_strings_buffer = 32
opcache.fast_shutdown = 1

With that config, it works well. Needs some getting used to though, since
you'll need to restart php to see any change in the sources applied.

If you use something with an on-disk cache (Prestashop, magento, typo3 ..)
do think of storing that in a redis or something, never on gluster, that'd
kill performances. I've seen a gain of ~10 seconds by just moving the cache
from gluster to redis for Magento for example.


On Tue, Jul 11, 2017 at 11:01:52AM +0200, Jo Goossens wrote:
> Hello,
> 
>  
>  
> We tried tons of settings to get a php app running on a native gluster mount:
> 
>  
> e.g.: 192.168.140.41:/www /var/www glusterfs 
> defaults,_netdev,backup-volfile-servers=192.168.140.42:192.168.140.43,direct-io-mode=disable
>  0 0
> 
>  
> I tried some mount variants in order to speed up things without luck.
> 
>  
>  
> After that I tried nfs (native gluster nfs 3 and ganesha nfs 4), it was a 
> crazy performance difference.
> 
>  
> e.g.: 192.168.140.41:/www /var/www nfs4 defaults,_netdev 0 0
> 
>  
> I tried a test like this to confirm the slowness:
> 
>  
> ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 
> --files 5000 --file-size 64 --record-size 64
>  This test finished in around 1.5 seconds with NFS and in more than 250 
> seconds without nfs (can't remember exact numbers, but I reproduced it 
> several times for both).
>  With the native gluster mount the php app had loading times of over 10 
> seconds, with the nfs mount the php app loaded around 1 second maximum and 
> even less. (reproduced several times)
>   I tried all kind of performance settings and variants of this but not 
> helped , the difference stayed huge, here are some of the settings played 
> with in random order:
> 
>  
> gluster volume set www features.cache-invalidation on
> gluster volume set www features.cache-invalidation-timeout 600
> gluster volume set www performance.stat-prefetch on
> gluster volume set www performance.cache-samba-metadata on
> gluster volume set www performance.cache-invalidation on
> gluster volume set www performance.md-cache-timeout 600
> gluster volume set www network.inode-lru-limit 25
>  gluster volume set www performance.cache-refresh-timeout 60
> gluster volume set www performance.read-ahead disable
> gluster volume set www performance.readdir-ahead on
> gluster volume set www performance.parallel-readdir on
> gluster volume set www performance.write-behind-window-size 4MB
> gluster volume set www performance.io-thread-count 64
>  gluster volume set www performance.client-io-threads on
>  gluster volume set www performance.cache-size 1GB
> gluster volume set www performance.quick-read on
> gluster volume set www performance.flush-behind on
> gluster volume set www performance.write-behind on
> gluster volume set www nfs.disable on
>  gluster volume set www client.event-threads 3
> gluster volume set www server.event-threads 3
>   
>  
> The NFS ha adds a lot of complexity which we wouldn't need at all in our 
> setup, could you please explain what is going on here? Is NFS the only 
> solution to get acceptable performance? Did I miss one crucial settting 
> perhaps?
> 
>  
> We're really desperate, thanks a lot for your help!
> 
>  
>  
> PS: We tried with gluster 3.11 and 3.8 on Debian, both had terrible 
> performance when not used with nfs.
> 
>  
>  
> 
> 
> Kind regards
> 
> Jo Goossens
> 
>  
>  
>  

> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users



signature.asc
Description: Digital signature
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hello,

 
 
Here is the volume info as requested by soumya:

 
#gluster volume info www
 Volume Name: www
Type: Replicate
Volume ID: 5d64ee36-828a-41fa-adbf-75718b954aff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.140.41:/gluster/www
Brick2: 192.168.140.42:/gluster/www
Brick3: 192.168.140.43:/gluster/www
Options Reconfigured:
cluster.read-hash-mode: 0
performance.quick-read: on
performance.write-behind-window-size: 4MB
server.allow-insecure: on
performance.read-ahead: disable
performance.readdir-ahead: on
performance.io-thread-count: 64
performance.io-cache: on
performance.client-io-threads: on
server.outstanding-rpc-limit: 128
server.event-threads: 3
client.event-threads: 3
performance.cache-size: 32MB
transport.address-family: inet
nfs.disable: on
nfs.addr-namelookup: off
nfs.export-volumes: on
nfs.rpc-auth-allow: 192.168.140.*
features.cache-invalidation: on
features.cache-invalidation-timeout: 600
performance.stat-prefetch: on
performance.cache-samba-metadata: on
performance.cache-invalidation: on
performance.md-cache-timeout: 600
network.inode-lru-limit: 10
performance.parallel-readdir: on
performance.cache-refresh-timeout: 60
performance.rda-cache-limit: 50MB
cluster.nufa: on
network.ping-timeout: 5
cluster.lookup-optimize: on
cluster.quorum-type: auto
 I started with none of them set and I added/changed while testing. But it was 
always slow, by tuning some kernel parameters it improved slightly (just a few 
percent, nothing reasonable)
 I also tried ceph just to compare, I got this with default settings and no 
tweaks:
  ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 
--files 5000 --file-size 64 --record-size 64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 5000
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 1.339621,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 1.436776,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 1.498681,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 1.483886,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 1.454833,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 1.469340,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 1.439060,files = 5000,records = 
0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 1.375074,files = 5000,records = 
0,status = ok
total threads = 8
total files = 4
100.00% of requested files processed, minimum is  70.00
1.498681 sec elapsed time
26690.134975 files/sec
  
Regards

Jo

 
-Original message-
From:Jo Goossens 
Sent:Tue 11-07-2017 12:15
Subject:Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To:Soumya Koduri ; gluster-users@gluster.org; 
CC:Ambarish Soman ; 
 

Hello,

 
 
Here is some speedtest with a new setup we just made with gluster 3.10, there 
are no other differences, except glusterfs versus nfs. The nfs is about 80 
times faster:

 
 
root@app1:~/smallfile-master# mount -t glusterfs -o 
use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log 
192.168.140.41:/www /var/www
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 500 --file-size 64 --record-size 
64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']

Re: [Gluster-users] Replica 3 with arbiter - heal error?

2017-07-11 Thread Pavel Szalbot

I tested the same procedure on volume with following config and cannot
reproduce the issue. Should I file a bug?

transport.address-family: inet
performance.readdir-ahead: on
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 1
features.shard: on
user.cifs: off

Btw nevermind the 40 seconds timeout, got the network.ping-timeout ;-)
-ps


On Tue, Jul 11, 2017 at 3:03 PM, Pavel Szalbot  wrote:
> Hello,
>
> I have a Gluster 3.8.13 with replica 3 arbiter volume mounted and run
> there a following script:
>
> while true; do echo "$(date)" >> a.txt; sleep 2; done
>
> After few seconds I add a rule to the firewall on the client, that
> blocks access to node specified during mount e.g. if volume is mounted
> with:
>
> mount -t glusterfs -o backupvolfile-server=10.0.0.2 10.0.0.1:/vol /mnt/vol
>
> I add:
>
> iptables -A OUTPUT -d 10.0.0.1 -j REJECT
>
> This causes the script above to block for approximately 40 seconds
> until gluster client tries backupvolfile-server (can this timeout be
> changed?) and everything continues as expected.
>
> Heal info shows that this file (a.txt) undergoes healing. About a
> minute later, last line of the a.txt contains $(date) same as just
> before the firewall modification. Each consecutive write e.g. echo
> "STRING" >> a.txt actually appends not the "STRING", but number of
> bytes previously written.
>
> If the file content just before firewall rule addition is:
> Tue Jul 11 14:19:37 CEST 2017
> Tue Jul 11 14:19:39 CEST 2017
>
> It will later become (which is OK):
> Tue Jul 11 14:19:37 CEST 2017
> Tue Jul 11 14:19:39 CEST 2017
> Tue Jul 11 14:20:18 CEST 2017
> Tue Jul 11 14:20:20 CEST 2017
> Tue Jul 11 14:20:22 CEST 2017
>
> But after some time, file content is only:
> Tue Jul 11 14:19:37 CEST 2017
> Tue Jul 11 14:19:39 CEST 2017
>
> And echo "STRING" >> a.txt makes it (6 bytes appended, not STRING):
> Tue Jul 11 14:19:37 CEST 2017
> Tue Jul 11 14:19:39 CEST 2017
> Tue Ju
>
> Another echo "STRING" >> a.txt causes the content to be:
> Tue Jul 11 14:19:37 CEST 2017
> Tue Jul 11 14:19:39 CEST 2017
> Tue Jul 11 1
>
> Removing the firewall rule does not change the content and different
> client with access to all nodes sees exactly the same content as this
> one.
>
> Is this normal behavior or bug or is there any configuration that I
> should have changed in order to have replica 3 with arbiter highly
> available?
>
> I stumbled upon this while testing how to upgrade Gluster so the
> clients resp. VMs on the clients are not affected by the "transport
> endpoint error" caused by primary mountpoint undergoing upgrade and
> therefore glusterd being not available for several seconds.
>
> Volume config:
> server.allow-insecure: on
> server.outstanding-rpc-limit: 1024
> performance.read-ahead: off
> performance.io-thread-count: 64
> performance.client-io-threads: on
> performance.cache-size: 1GB
> cluster.self-heal-daemon: enable
> nfs.disable: on
> performance.readdir-ahead: on
> features.shard: on
> performance.quick-read: off
> performance.io-cache: off
> performance.stat-prefetch: off
> performance.low-prio-threads: 32
> network.remote-dio: enable
> cluster.eager-lock: enable
> cluster.quorum-type: auto
> cluster.server-quorum-type: server
> cluster.data-self-heal-algorithm: full
> cluster.locking-scheme: granular
> cluster.shd-max-threads: 8
> cluster.shd-wait-qlength: 1
> user.cifs: off
>
> -ps
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Performance Translators Documentation

2017-07-11 Thread Christopher Schmidt

Hi,

I had some issues (org.apache.lucene.index.CorruptIndexException) with
Lucene (resp. ElasticSearch) working on a GlusterFS Volume and Kubernetes.

For testing I switched off all performance translators...

And I wonder if there is somewhere documentation, who they are and what
they are doing?

best Chris
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Replica 3 with arbiter - heal error?

2017-07-11 Thread Pavel Szalbot

Hello,

I have a Gluster 3.8.13 with replica 3 arbiter volume mounted and run
there a following script:

while true; do echo "$(date)" >> a.txt; sleep 2; done

After few seconds I add a rule to the firewall on the client, that
blocks access to node specified during mount e.g. if volume is mounted
with:

mount -t glusterfs -o backupvolfile-server=10.0.0.2 10.0.0.1:/vol /mnt/vol

I add:

iptables -A OUTPUT -d 10.0.0.1 -j REJECT

This causes the script above to block for approximately 40 seconds
until gluster client tries backupvolfile-server (can this timeout be
changed?) and everything continues as expected.

Heal info shows that this file (a.txt) undergoes healing. About a
minute later, last line of the a.txt contains $(date) same as just
before the firewall modification. Each consecutive write e.g. echo
"STRING" >> a.txt actually appends not the "STRING", but number of
bytes previously written.

If the file content just before firewall rule addition is:
Tue Jul 11 14:19:37 CEST 2017
Tue Jul 11 14:19:39 CEST 2017

It will later become (which is OK):
Tue Jul 11 14:19:37 CEST 2017
Tue Jul 11 14:19:39 CEST 2017
Tue Jul 11 14:20:18 CEST 2017
Tue Jul 11 14:20:20 CEST 2017
Tue Jul 11 14:20:22 CEST 2017

But after some time, file content is only:
Tue Jul 11 14:19:37 CEST 2017
Tue Jul 11 14:19:39 CEST 2017

And echo "STRING" >> a.txt makes it (6 bytes appended, not STRING):
Tue Jul 11 14:19:37 CEST 2017
Tue Jul 11 14:19:39 CEST 2017
Tue Ju

Another echo "STRING" >> a.txt causes the content to be:
Tue Jul 11 14:19:37 CEST 2017
Tue Jul 11 14:19:39 CEST 2017
Tue Jul 11 1

Removing the firewall rule does not change the content and different
client with access to all nodes sees exactly the same content as this
one.

Is this normal behavior or bug or is there any configuration that I
should have changed in order to have replica 3 with arbiter highly
available?

I stumbled upon this while testing how to upgrade Gluster so the
clients resp. VMs on the clients are not affected by the "transport
endpoint error" caused by primary mountpoint undergoing upgrade and
therefore glusterd being not available for several seconds.

Volume config:
server.allow-insecure: on
server.outstanding-rpc-limit: 1024
performance.read-ahead: off
performance.io-thread-count: 64
performance.client-io-threads: on
performance.cache-size: 1GB
cluster.self-heal-daemon: enable
nfs.disable: on
performance.readdir-ahead: on
features.shard: on
performance.quick-read: off
performance.io-cache: off
performance.stat-prefetch: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 1
user.cifs: off

-ps
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

2017-07-11 Thread Pavel Szalbot

Well it was probably caused by running replica 2 and doing online
upgrade. However I added brick, turned volume to replica 3 with
arbiter and got very strange issue I will mail to this list in a
moment...

Thanks.
-ps


On Tue, Jul 11, 2017 at 1:55 PM, Pranith Kumar Karampuri
 wrote:
>
>
> On Tue, Jul 11, 2017 at 5:12 PM, Diego Remolina  wrote:
>>
>> >
>> > You should first upgrade servers and then clients. New servers can
>> > understand old clients, but it is not easy for old servers to understand
>> > new
>> > clients in case it started doing something new.
>>
>> But isn't that the reason op-version exists? So that regardless of
>> client/server mix, nobody tries to do "new" things above the current
>> op-version?
>>
>>
>> He is not changing mayor versions, just a small step from 3.8.12 to
>> 3.8.13. Corruption should not be happening.
>
>
> For some reason 3.8 upgrade guide is not where it is supposed to be.
> We highly recommend upgrading servers ahead of clients.
> https://github.com/nixpanic/glusterdocs/commit/f6d48dc17f2cb6ee4680e372520ec3358641b2bc
>
> I think at some point it is better to make this mandatory. Without a
> predefined way of upgrading, it is very difficult to fix bugs in backward
> compatible manner.
>
> I am not sure why the corruption happened either :-(. Pavel, could you give
> log files may be?
>
>>
>> Diego
>> ___
>> Gluster-users mailing list
>> Gluster-users@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

2017-07-11 Thread Pranith Kumar Karampuri

On Tue, Jul 11, 2017 at 5:12 PM, Diego Remolina  wrote:

> >
> > You should first upgrade servers and then clients. New servers can
> > understand old clients, but it is not easy for old servers to understand
> new
> > clients in case it started doing something new.
>
> But isn't that the reason op-version exists? So that regardless of
> client/server mix, nobody tries to do "new" things above the current
> op-version?
>

> He is not changing mayor versions, just a small step from 3.8.12 to
> 3.8.13. Corruption should not be happening.
>

For some reason 3.8 upgrade guide is not where it is supposed to be.
We highly recommend upgrading servers ahead of clients.
https://github.com/nixpanic/glusterdocs/commit/f6d48dc17f2cb6ee4680e372520ec3358641b2bc

I think at some point it is better to make this mandatory. Without a
predefined way of upgrading, it is very difficult to fix bugs in backward
compatible manner.

I am not sure why the corruption happened either :-(. Pavel, could you give
log files may be?

> Diego
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>

-- 
Pranith
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

2017-07-11 Thread Diego Remolina

>
> You should first upgrade servers and then clients. New servers can
> understand old clients, but it is not easy for old servers to understand new
> clients in case it started doing something new.

But isn't that the reason op-version exists? So that regardless of
client/server mix, nobody tries to do "new" things above the current
op-version?

He is not changing mayor versions, just a small step from 3.8.12 to
3.8.13. Corruption should not be happening.

Diego
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hello,

 
 
Here is some speedtest with a new setup we just made with gluster 3.10, there 
are no other differences, except glusterfs versus nfs. The nfs is about 80 
times faster:

 
 
root@app1:~/smallfile-master# mount -t glusterfs -o 
use-readdirp=no,log-level=WARNING,log-file=/var/log/glusterxxx.log 
192.168.140.41:/www /var/www
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 500 --file-size 64 --record-size 
64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 500
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 68.845450,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 67.601088,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 58.677994,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 65.901922,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 66.971720,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 71.245102,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 06,elapsed = 67.574845,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 07,elapsed = 54.263242,files = 500,records = 
0,status = ok
total threads = 8
total files = 4000
100.00% of requested files processed, minimum is  70.00
71.245102 sec elapsed time
56.144211 files/sec
 umount /var/www
 root@app1:~/smallfile-master# mount -t nfs -o tcp 192.168.140.41:/www /var/www
root@app1:~/smallfile-master# ./smallfile_cli.py  --top /var/www/test 
--host-set 192.168.140.41 --threads 8 --files 500 --file-size 64 --record-size 
64
smallfile version 3.0
                           hosts in test : ['192.168.140.41']
                   top test directory(s) : ['/var/www/test']
                               operation : cleanup
                            files/thread : 500
                                 threads : 8
           record size (KB, 0 = maximum) : 64
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix :
                         filename suffix :
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /root/smallfile-master
               network thread sync. dir. : /var/www/test/network_shared
starting all threads by creating starting gate file 
/var/www/test/network_shared/starting_gate.tmp
host = 192.168.140.41,thr = 00,elapsed = 0.962424,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 01,elapsed = 0.942673,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 02,elapsed = 0.940622,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 03,elapsed = 0.915218,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 04,elapsed = 0.934349,files = 500,records = 
0,status = ok
host = 192.168.140.41,thr = 05,elapsed = 0.922466,files = 500,records = 
0,status = ok
host =

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hi all,

 
 
One more thing, we have 3 apps servers with the gluster on it, replicated on 3 
different gluster nodes. (So the gluster nodes are app servers at the same 
time). We could actually almost work locally if we wouldn't need to have the 
same files on the 3 nodes and redundancy :)

 
Initial cluster was created like this:

 
gluster volume create www replica 3 transport tcp 192.168.140.41:/gluster/www 
192.168.140.42:/gluster/www 192.168.140.43:/gluster/www force
gluster volume set www network.ping-timeout 5
gluster volume set www performance.cache-size 1024MB
gluster volume set www nfs.disable on # No need for NFS currently
gluster volume start www
 To my understanding it still wouldn't explain why nfs has such great 
performance compared to native ...
  
Regards

Jo

 

 
-Original message-
From:Soumya Koduri 
Sent:Tue 11-07-2017 11:16
Subject:Re: [Gluster-users] Gluster native mount is really slow compared to nfs
To:Jo Goossens ; gluster-users@gluster.org; 
CC:Ambarish Soman ; Karan Sandha ; 
+ Ambarish

On 07/11/2017 02:31 PM, Jo Goossens wrote:
> Hello,
>
>
>
>
>
> We tried tons of settings to get a php app running on a native gluster
> mount:
>
>
>
> e.g.: 192.168.140.41:/www /var/www glusterfs
> defaults,_netdev,backup-volfile-servers=192.168.140.42:192.168.140.43,direct-io-mode=disable
> 0 0
>
>
>
> I tried some mount variants in order to speed up things without luck.
>
>
>
>
>
> After that I tried nfs (native gluster nfs 3 and ganesha nfs 4), it was
> a crazy performance difference.
>
>
>
> e.g.: 192.168.140.41:/www /var/www nfs4 defaults,_netdev 0 0
>
>
>
> I tried a test like this to confirm the slowness:
>
>
>
> ./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41
> --threads 8 --files 5000 --file-size 64 --record-size 64
>
> This test finished in around 1.5 seconds with NFS and in more than 250
> seconds without nfs (can't remember exact numbers, but I reproduced it
> several times for both).
>
> With the native gluster mount the php app had loading times of over 10
> seconds, with the nfs mount the php app loaded around 1 second maximum
> and even less. (reproduced several times)
>
>
>
> I tried all kind of performance settings and variants of this but not
> helped , the difference stayed huge, here are some of the settings
> played with in random order:
>

Request Ambarish & Karan (cc'ed who have been working on evaluating 
performance of various access protocols gluster supports) to look at the 
below settings and provide inputs.

Thanks,
Soumya

>
>
> gluster volume set www features.cache-invalidation on
> gluster volume set www features.cache-invalidation-timeout 600
> gluster volume set www performance.stat-prefetch on
> gluster volume set www performance.cache-samba-metadata on
> gluster volume set www performance.cache-invalidation on
> gluster volume set www performance.md-cache-timeout 600
> gluster volume set www network.inode-lru-limit 25
>
> gluster volume set www performance.cache-refresh-timeout 60
> gluster volume set www performance.read-ahead disable
> gluster volume set www performance.readdir-ahead on
> gluster volume set www performance.parallel-readdir on
> gluster volume set www performance.write-behind-window-size 4MB
> gluster volume set www performance.io-thread-count 64
>
> gluster volume set www performance.client-io-threads on
>
> gluster volume set www performance.cache-size 1GB
> gluster volume set www performance.quick-read on
> gluster volume set www performance.flush-behind on
> gluster volume set www performance.write-behind on
> gluster volume set www nfs.disable on
>
> gluster volume set www client.event-threads 3
> gluster volume set www server.event-threads 3
>
>
>
>
>
>
> The NFS ha adds a lot of complexity which we wouldn't need at all in our
> setup, could you please explain what is going on here? Is NFS the only
> solution to get acceptable performance? Did I miss one crucial settting
> perhaps?
>
>
>
> We're really desperate, thanks a lot for your help!
>
>
>
>
>
> PS: We tried with gluster 3.11 and 3.8 on Debian, both had terrible
> performance when not used with nfs.
>
>
>
>
>
>
>
> Kind regards
>
> Jo Goossens
>
>
>
>
>
>
>
>
>
> ___
> Gluster-users mailing list
> Gluster-users@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Soumya Koduri


+ Ambarish

On 07/11/2017 02:31 PM, Jo Goossens wrote:

Hello,





We tried tons of settings to get a php app running on a native gluster
mount:



e.g.: 192.168.140.41:/www /var/www glusterfs
defaults,_netdev,backup-volfile-servers=192.168.140.42:192.168.140.43,direct-io-mode=disable
0 0



I tried some mount variants in order to speed up things without luck.





After that I tried nfs (native gluster nfs 3 and ganesha nfs 4), it was
a crazy performance difference.



e.g.: 192.168.140.41:/www /var/www nfs4 defaults,_netdev 0 0



I tried a test like this to confirm the slowness:



./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41
--threads 8 --files 5000 --file-size 64 --record-size 64

This test finished in around 1.5 seconds with NFS and in more than 250
seconds without nfs (can't remember exact numbers, but I reproduced it
several times for both).

With the native gluster mount the php app had loading times of over 10
seconds, with the nfs mount the php app loaded around 1 second maximum
and even less. (reproduced several times)



I tried all kind of performance settings and variants of this but not
helped , the difference stayed huge, here are some of the settings
played with in random order:



Request Ambarish & Karan (cc'ed who have been working on evaluating 
performance of various access protocols gluster supports) to look at the 
below settings and provide inputs.


Thanks,
Soumya




gluster volume set www features.cache-invalidation on
gluster volume set www features.cache-invalidation-timeout 600
gluster volume set www performance.stat-prefetch on
gluster volume set www performance.cache-samba-metadata on
gluster volume set www performance.cache-invalidation on
gluster volume set www performance.md-cache-timeout 600
gluster volume set www network.inode-lru-limit 25

gluster volume set www performance.cache-refresh-timeout 60
gluster volume set www performance.read-ahead disable
gluster volume set www performance.readdir-ahead on
gluster volume set www performance.parallel-readdir on
gluster volume set www performance.write-behind-window-size 4MB
gluster volume set www performance.io-thread-count 64

gluster volume set www performance.client-io-threads on

gluster volume set www performance.cache-size 1GB
gluster volume set www performance.quick-read on
gluster volume set www performance.flush-behind on
gluster volume set www performance.write-behind on
gluster volume set www nfs.disable on

gluster volume set www client.event-threads 3
gluster volume set www server.event-threads 3






The NFS ha adds a lot of complexity which we wouldn't need at all in our
setup, could you please explain what is going on here? Is NFS the only
solution to get acceptable performance? Did I miss one crucial settting
perhaps?



We're really desperate, thanks a lot for your help!





PS: We tried with gluster 3.11 and 3.8 on Debian, both had terrible
performance when not used with nfs.







Kind regards

Jo Goossens









___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Gluster native mount is really slow compared to nfs

2017-07-11 Thread Jo Goossens

Hello,

 
 
We tried tons of settings to get a php app running on a native gluster mount:

 
e.g.: 192.168.140.41:/www /var/www glusterfs 
defaults,_netdev,backup-volfile-servers=192.168.140.42:192.168.140.43,direct-io-mode=disable
 0 0

 
I tried some mount variants in order to speed up things without luck.

 
 
After that I tried nfs (native gluster nfs 3 and ganesha nfs 4), it was a crazy 
performance difference.

 
e.g.: 192.168.140.41:/www /var/www nfs4 defaults,_netdev 0 0

 
I tried a test like this to confirm the slowness:

 
./smallfile_cli.py  --top /var/www/test --host-set 192.168.140.41 --threads 8 
--files 5000 --file-size 64 --record-size 64
 This test finished in around 1.5 seconds with NFS and in more than 250 seconds 
without nfs (can't remember exact numbers, but I reproduced it several times 
for both).
 With the native gluster mount the php app had loading times of over 10 
seconds, with the nfs mount the php app loaded around 1 second maximum and even 
less. (reproduced several times)
  I tried all kind of performance settings and variants of this but not helped 
, the difference stayed huge, here are some of the settings played with in 
random order:

 
gluster volume set www features.cache-invalidation on
gluster volume set www features.cache-invalidation-timeout 600
gluster volume set www performance.stat-prefetch on
gluster volume set www performance.cache-samba-metadata on
gluster volume set www performance.cache-invalidation on
gluster volume set www performance.md-cache-timeout 600
gluster volume set www network.inode-lru-limit 25
 gluster volume set www performance.cache-refresh-timeout 60
gluster volume set www performance.read-ahead disable
gluster volume set www performance.readdir-ahead on
gluster volume set www performance.parallel-readdir on
gluster volume set www performance.write-behind-window-size 4MB
gluster volume set www performance.io-thread-count 64
 gluster volume set www performance.client-io-threads on
 gluster volume set www performance.cache-size 1GB
gluster volume set www performance.quick-read on
gluster volume set www performance.flush-behind on
gluster volume set www performance.write-behind on
gluster volume set www nfs.disable on
 gluster volume set www client.event-threads 3
gluster volume set www server.event-threads 3
  
 
The NFS ha adds a lot of complexity which we wouldn't need at all in our setup, 
could you please explain what is going on here? Is NFS the only solution to get 
acceptable performance? Did I miss one crucial settting perhaps?

 
We're really desperate, thanks a lot for your help!

 
 
PS: We tried with gluster 3.11 and 3.8 on Debian, both had terrible performance 
when not used with nfs.

 
 


Kind regards

Jo Goossens

 
 
 
___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

Re: [Gluster-users] set owner:group on root of volume

[Gluster-users] set owner:group on root of volume

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Extremely slow du

Re: [Gluster-users] Extremely slow du

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Replica 3 with arbiter - heal error?

[Gluster-users] Performance Translators Documentation

[Gluster-users] Replica 3 with arbiter - heal error?

Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

Re: [Gluster-users] Gluster native mount is really slow compared to nfs

[Gluster-users] Gluster native mount is really slow compared to nfs

23 matches

Site Navigation

Mail list logo

Footer information