Re: [Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?
On Thu, 18 Feb 2016 10:14:59 +1000 Dan Mons wrote: > Without knowing the details, I'm putting my money on cache. > > Choosing how to mount Gluster is workload dependent. If you're doing > a lot of small files with single threaded writes, I suggest NFS. Your > client's nfscache will dramatically improve performance from the > end-user's point of view. > > If you're doing heavy multi-threaded reads and writes, and you have > very good bandwidth from your client (e.g.: 10GbE) FUSE+GlusterFS is > better, as it allows your client to talk to all Gluster nodes. > [...] Dan, forgive my jump in this matter which is obvious to everyone using glusterfs for years: fuse+glusterfs is simply sh*t talking of performance. There is absolutely nobody whose setup wouldn't be at least two (to several hundred) times faster using simple NFS. So Stefans numbers are no surprise. I really cannot believe you are trying to argue for fuse. It is completely clear that fuse is only used because of the incompetence to write a kernel-space driver (and this was said years ago by the people who originally wrote the whole lot). You probably can find this answer to my question in the archives of this (or the devel) list years back. And because of this I pretty much stopped writing here, I mean you cannot blame someone for not being skilled enough to produce the right code in a GPL situation. The basic concept is good, the implementation is just a mess. And that's it. Regards, Stephan > If you are using FUSE+GlusterFS, on the gluster nodes themselves, > experiment with the "performance.write-behind-window-size" and > "performance.cache-size" options. Note that these will affect the > cache used by the clients, so don't set them so high as to exhaust the > RAM of any client connecting (or, for low-memory clients, use NFS > instead). > > Gluster ships with conservative defaults for cache, which is a good > thing. It's up to the user to tweak for their optimal needs. > > There's no right or wrong answer here. Experiment with NFS and > various cache allocations with FUSE+GlusterFS, and see how you go. > And again, consider your workloads, and whether or not they're taking > full advantage of the FUSE client's ability to deal with highly > parallel workloads. > > -Dan > > Dan Mons - VFX Sysadmin > Cutting Edge > http://cuttingedge.com.au > > > On 18 February 2016 at 08:56, Stefan Jakobs wrote: > > Van Renterghem Stijn: > >> Interval2 > >> Block Size: 1b+ 16b+ > >> 32b+ > >> No. of Reads:0 0 > >> 0 No. of Writes: 34225 > >>575 > >> > >>Block Size: 64b+ 128b+ > >> 256b+ No. of Reads:0 0 > >>0 No. of Writes: 143 898 > >> 118 > >> > >>Block Size:512b+1024b+ > >> 2048b+ No. of Reads:1 4 > >>11 No. of Writes: 82 0 > >> 0 > >> > >>Block Size: 4096b+8192b+ > >> 16384b+ No. of Reads: 1131 > >> 39 No. of Writes:0 0 > >> 0 > >> > >>Block Size: 32768b+ 65536b+ > >> 131072b+ No. of Reads: 59 148 > >> 555 No. of Writes:0 0 > >> 0 > >> > >> %-latency Avg-latency Min-Latency Max-Latency No. of calls > >> Fop - --- --- --- > >> 0.00 0.00 us 0.00 us 0.00 us 1 > >> FORGET 0.00 0.00 us 0.00 us 0.00 us201 > >> RELEASE 0.00 0.00 us 0.00 us 0.00 us 54549 > >> RELEASEDIR 0.00 47.00 us 47.00 us 47.00 us 1 > >> REMOVEXATTR 0.00 94.00 us 74.00 us 114.00 us 2 > >> XATTROP 0.00 191.00 us 191.00 us 191.00 us 1 > >> TRUNCATE 0.00 53.50 us 35.00 us 74.00 us 4 > >> STATFS 0.00 79.67 us 70.00 us 91.00 us 3 > >> RENAME 0.00 37.33 us 27.00 us 68.00 us 15 > >> INODELK 0.00 190.67 us 116.00 us 252.00 us 3 > >> UNLINK 0.00 28.83 us 8.00 us 99.00 us 30 > >> ENTRYLK 0.00 146.33 us 117.00 us 188.00 us 6 > >> CREATE 0.00 37.63 us 12.00 us 73.00 us 84 > >> READDIR 0.00 23.75 us 8.00 us 75.00 us198 > >> FLUSH 0.00 65.33 us 42.00 us 141.00 us204 > >> OPEN 0.01 45.78 us 11.00 us 191.00 us944 > >> FINODELK 0.01 8
Re: [Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?
Without knowing the details, I'm putting my money on cache. Choosing how to mount Gluster is workload dependent. If you're doing a lot of small files with single threaded writes, I suggest NFS. Your client's nfscache will dramatically improve performance from the end-user's point of view. If you're doing heavy multi-threaded reads and writes, and you have very good bandwidth from your client (e.g.: 10GbE) FUSE+GlusterFS is better, as it allows your client to talk to all Gluster nodes. If you are using FUSE+GlusterFS, on the gluster nodes themselves, experiment with the "performance.write-behind-window-size" and "performance.cache-size" options. Note that these will affect the cache used by the clients, so don't set them so high as to exhaust the RAM of any client connecting (or, for low-memory clients, use NFS instead). Gluster ships with conservative defaults for cache, which is a good thing. It's up to the user to tweak for their optimal needs. There's no right or wrong answer here. Experiment with NFS and various cache allocations with FUSE+GlusterFS, and see how you go. And again, consider your workloads, and whether or not they're taking full advantage of the FUSE client's ability to deal with highly parallel workloads. -Dan Dan Mons - VFX Sysadmin Cutting Edge http://cuttingedge.com.au On 18 February 2016 at 08:56, Stefan Jakobs wrote: > Van Renterghem Stijn: >> Interval2 >> Block Size: 1b+ 16b+ 32b+ >> No. of Reads:0 0 >> 0 No. of Writes: 34225 >>575 >> >>Block Size: 64b+ 128b+ >> 256b+ No. of Reads:0 0 >>0 No. of Writes: 143 898 >> 118 >> >>Block Size:512b+1024b+ >> 2048b+ No. of Reads:1 4 >>11 No. of Writes: 82 0 >> 0 >> >>Block Size: 4096b+8192b+ >> 16384b+ No. of Reads: 1131 >> 39 No. of Writes:0 0 >> 0 >> >>Block Size: 32768b+ 65536b+ >> 131072b+ No. of Reads: 59 148 >> 555 No. of Writes:0 0 >> 0 >> >> %-latency Avg-latency Min-Latency Max-Latency No. of calls >> Fop - --- --- --- >> 0.00 0.00 us 0.00 us 0.00 us 1 >> FORGET 0.00 0.00 us 0.00 us 0.00 us201 >> RELEASE 0.00 0.00 us 0.00 us 0.00 us 54549 >> RELEASEDIR 0.00 47.00 us 47.00 us 47.00 us 1 >> REMOVEXATTR 0.00 94.00 us 74.00 us 114.00 us 2 >> XATTROP 0.00 191.00 us 191.00 us 191.00 us 1 >> TRUNCATE 0.00 53.50 us 35.00 us 74.00 us 4 >> STATFS 0.00 79.67 us 70.00 us 91.00 us 3 >> RENAME 0.00 37.33 us 27.00 us 68.00 us 15 >> INODELK 0.00 190.67 us 116.00 us 252.00 us 3 >> UNLINK 0.00 28.83 us 8.00 us 99.00 us 30 >> ENTRYLK 0.00 146.33 us 117.00 us 188.00 us 6 >> CREATE 0.00 37.63 us 12.00 us 73.00 us 84 >> READDIR 0.00 23.75 us 8.00 us 75.00 us198 >> FLUSH 0.00 65.33 us 42.00 us 141.00 us204 >> OPEN 0.01 45.78 us 11.00 us 191.00 us944 >> FINODELK 0.01 80.34 us 31.00 us 211.00 us859 >> READ 0.02 96.74 us 50.00 us 188.00 us944 >> FXATTROP 0.02 55.84 us 24.00 us 140.00 us 1707 >> FSTAT 0.02 52.89 us 21.00 us 175.00 us 2183 >> WRITE 0.02 59.69 us 11.00 us 235.00 us 2312 >> GETXATTR 0.03 51.18 us 8.00 us 142.00 us 3091 >> STAT 0.46 48.66 us 1.00 us 179.00 us 54549 >> OPENDIR 1.13 135.93 us 18.00 us 16362.00 us 48124 >> READDIRP 98.29 70.46 us 16.00 us2903.00 us8104385 >> LOOKUP >> >> Duration: 7560 seconds >>Data Read: 91208567 bytes = 91MB >> Data Written: 292007 bytes = 0,292MB > > How did you collect these statistics? > > Thanks > Stefan > ___ > Gluster-users mailing list > Gluster-users@gluster.org > http://www.gluster.org/mailman/listinfo/gluster-users ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-use
Re: [Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?
Van Renterghem Stijn: > Interval2 > Block Size: 1b+ 16b+ 32b+ > No. of Reads:0 0 > 0 No. of Writes: 34225 >575 > >Block Size: 64b+ 128b+ > 256b+ No. of Reads:0 0 >0 No. of Writes: 143 898 > 118 > >Block Size:512b+1024b+ > 2048b+ No. of Reads:1 4 >11 No. of Writes: 82 0 > 0 > >Block Size: 4096b+8192b+ > 16384b+ No. of Reads: 1131 > 39 No. of Writes:0 0 > 0 > >Block Size: 32768b+ 65536b+ > 131072b+ No. of Reads: 59 148 > 555 No. of Writes:0 0 > 0 > > %-latency Avg-latency Min-Latency Max-Latency No. of calls > Fop - --- --- --- > 0.00 0.00 us 0.00 us 0.00 us 1 > FORGET 0.00 0.00 us 0.00 us 0.00 us201 > RELEASE 0.00 0.00 us 0.00 us 0.00 us 54549 > RELEASEDIR 0.00 47.00 us 47.00 us 47.00 us 1 > REMOVEXATTR 0.00 94.00 us 74.00 us 114.00 us 2 > XATTROP 0.00 191.00 us 191.00 us 191.00 us 1 > TRUNCATE 0.00 53.50 us 35.00 us 74.00 us 4 > STATFS 0.00 79.67 us 70.00 us 91.00 us 3 > RENAME 0.00 37.33 us 27.00 us 68.00 us 15 > INODELK 0.00 190.67 us 116.00 us 252.00 us 3 > UNLINK 0.00 28.83 us 8.00 us 99.00 us 30 > ENTRYLK 0.00 146.33 us 117.00 us 188.00 us 6 > CREATE 0.00 37.63 us 12.00 us 73.00 us 84 > READDIR 0.00 23.75 us 8.00 us 75.00 us198 > FLUSH 0.00 65.33 us 42.00 us 141.00 us204 > OPEN 0.01 45.78 us 11.00 us 191.00 us944 > FINODELK 0.01 80.34 us 31.00 us 211.00 us859 > READ 0.02 96.74 us 50.00 us 188.00 us944 > FXATTROP 0.02 55.84 us 24.00 us 140.00 us 1707 > FSTAT 0.02 52.89 us 21.00 us 175.00 us 2183 > WRITE 0.02 59.69 us 11.00 us 235.00 us 2312 > GETXATTR 0.03 51.18 us 8.00 us 142.00 us 3091 > STAT 0.46 48.66 us 1.00 us 179.00 us 54549 > OPENDIR 1.13 135.93 us 18.00 us 16362.00 us 48124 > READDIRP 98.29 70.46 us 16.00 us2903.00 us8104385 > LOOKUP > > Duration: 7560 seconds >Data Read: 91208567 bytes = 91MB > Data Written: 292007 bytes = 0,292MB How did you collect these statistics? Thanks Stefan ___ Gluster-users mailing list Gluster-users@gluster.org http://www.gluster.org/mailman/listinfo/gluster-users
[Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?
Hi, I have setup a server with a new installation of Gluster. The volume type is 'Replicate'. 1) I mounted the volume with Fuse IP1:/app /srv/data glusterfs defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2 0 0 When I start my application, it takes 2h until the application is started Below you can see the stats after the application is started. I can see a very high LOOKUP value. Can you explain this high value ? The volume type is replicate, so I should think, I shouldn't have LOOKUPs ? Interval2 Block Size: 1b+ 16b+ 32b+ No. of Reads:0 0 0 No. of Writes: 34225 575 Block Size: 64b+ 128b+ 256b+ No. of Reads:0 0 0 No. of Writes: 143 898 118 Block Size:512b+1024b+2048b+ No. of Reads:1 411 No. of Writes: 82 0 0 Block Size: 4096b+8192b+ 16384b+ No. of Reads: 113139 No. of Writes:0 0 0 Block Size: 32768b+ 65536b+ 131072b+ No. of Reads: 59 148 555 No. of Writes:0 0 0 %-latency Avg-latency Min-Latency Max-Latency No. of calls Fop - --- --- --- 0.00 0.00 us 0.00 us 0.00 us 1 FORGET 0.00 0.00 us 0.00 us 0.00 us201 RELEASE 0.00 0.00 us 0.00 us 0.00 us 54549 RELEASEDIR 0.00 47.00 us 47.00 us 47.00 us 1 REMOVEXATTR 0.00 94.00 us 74.00 us 114.00 us 2 XATTROP 0.00 191.00 us 191.00 us 191.00 us 1TRUNCATE 0.00 53.50 us 35.00 us 74.00 us 4 STATFS 0.00 79.67 us 70.00 us 91.00 us 3 RENAME 0.00 37.33 us 27.00 us 68.00 us 15 INODELK 0.00 190.67 us 116.00 us 252.00 us 3 UNLINK 0.00 28.83 us 8.00 us 99.00 us 30 ENTRYLK 0.00 146.33 us 117.00 us 188.00 us 6 CREATE 0.00 37.63 us 12.00 us 73.00 us 84 READDIR 0.00 23.75 us 8.00 us 75.00 us198 FLUSH 0.00 65.33 us 42.00 us 141.00 us204OPEN 0.01 45.78 us 11.00 us 191.00 us944FINODELK 0.01 80.34 us 31.00 us 211.00 us859READ 0.02 96.74 us 50.00 us 188.00 us944FXATTROP 0.02 55.84 us 24.00 us 140.00 us 1707 FSTAT 0.02 52.89 us 21.00 us 175.00 us 2183 WRITE 0.02 59.69 us 11.00 us 235.00 us 2312GETXATTR 0.03 51.18 us 8.00 us 142.00 us 3091STAT 0.46 48.66 us 1.00 us 179.00 us 54549 OPENDIR 1.13 135.93 us 18.00 us 16362.00 us 48124READDIRP 98.29 70.46 us 16.00 us2903.00 us8104385 LOOKUP Duration: 7560 seconds Data Read: 91208567 bytes = 91MB Data Written: 292007 bytes = 0,292MB 2) I have tried some tuning options, but that didn't changed anything: #gluster volume info app Volume Name: app Type: Replicate Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041 Status: Started Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: IP1:/exports/app/app Brick2: IP2:/exports/app/app Options Reconfigured: cluster.readdir-optimize: on server.event-threads: 8 client.event-threads: 8 cluster.lookup-optimize: on diagnostics.count-fop-hits: on diagnostics.latency-measurement: on auth.allow: client1,client2 nfs.rpc-auth-allow: client1,client2 nfs.export-volumes: on nfs.addr-namelookup: off nfs.disable: off performance.readdir-ahead: on performance.io-thread-count: 64 3) I then have enabled NFS support. I stopped the application and unmounted the volume. I then mounted it again with nfs: IP1:/app/srv/data nfs rsize=4096,wsize=4096,hard,intr 0 0 I started the application again and it was running within 3minutes. The st
