[Gluster-devel] xlator.mount.fuse.itable.lru_limit=0 at client fuse process
Dear, Developers, Many tuning and benchmark on different gluster release, 3.12.15, 4.1, 3.11, the client fuse process will eat hundreds of GB RAM memory with 256G memory system, then OOM and was killed at some time. To consult many google search, fuse related paper, benchmark, testing, We can not eventually determine what's the reason why memory grows larger and larger. We make sure, xlator.mount.fuse.itable.lru_limit=0 at client fuse process could give us some clues. I guess, gluster fuse process will cache files inode at client end, and never kick old inode eventually. However, I do not know if this is design issue, or some tradoff, or bugs. my configuration: Options Reconfigured: performance.write-behind-window-size: 256MB performance.write-behind: on cluster.lookup-optimize: on transport.listen-backlog: 1024 performance.io-thread-count: 6 performance.cache-size: 10GB performance.quick-read: on performance.parallel-readdir: on network.inode-lru-limit: 5 cluster.quorum-reads: on cluster.quorum-count: 2 cluster.quorum-type: fixed cluster.server-quorum-type: server client.event-threads: 4 performance.stat-prefetch: on performance.md-cache-timeout: 600 cluster.min-free-disk: 5% performance.flush-behind: on transport.address-family: inet nfs.disable: on performance.client-io-threads: on cluster.server-quorum-ratio: 51% We extremely hope some replies from community, extremely. Even telling us the trouble can not be resolved for design reason is GREAT for us. Thanks a lot. - Fei ___ Gluster-devel mailing list Gluster-devel@gluster.org https://lists.gluster.org/mailman/listinfo/gluster-devel
Re: [Gluster-devel] Any mature(better) solution(way) to handle slow performance on 'ls -l, '.
Thanks for you kind reply, which is very helpful for me, I use 3.11, Thanks a lot. :-) - Fei On Thu, Jun 7, 2018 at 6:59 PM Poornima Gurusiddaiah wrote: > > If you are not using applications that rely on 100% metadata consistency, > like Databases, Kafka, AMQ etc. you can use the below mentioned volume > options: > > # gluster volume set group metadata-cache > > # gluster volume set network.inode-lru-limit 20 > > # gluster volume set performance.readdir-ahead on > > # gluster volume set performance.parallel-readdir on > > For more information refer to [1] > > Also, which version og Gluster are you using? Its preferred to use 3.11 or > above for these perf enhancements. > Note that parallel-readdir is going to help in increasing the ls -l > performance drastically in your case, but there are few corner case known > issues. > > Regards, > Poornima > > [1] > https://github.com/gluster/glusterdocs/pull/342/files#diff-62f536ad33b2c2210d023b0cffec2c64 > > On Wed, May 30, 2018, 8:29 PM Yanfei Wang wrote: >> >> Hi experts on glusterFS, >> >> In our testbed, we found that the ' ls -l' performance is pretty slow. >> Indeed from the prospect of glusterFS design space, we need to avoid >> 'ls ' directory which will traverse all bricks sequentially in our >> current knowledge. >> >> We use generic setting for our testbed: >> >> ``` >> Volume Name: gv0 >> Type: Distributed-Replicate >> Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5dffad4d0 >> Status: Started >> Snapshot Count: 0 >> Number of Bricks: 19 x 3 = 57 >> Transport-type: tcp >> Bricks: >> ... >> Options Reconfigured: >> features.inode-quota: off >> features.quota: off >> cluster.quorum-reads: on >> cluster.quorum-count: 2 >> cluster.quorum-type: fixed >> transport.address-family: inet >> nfs.disable: on >> performance.client-io-threads: off >> cluster.server-quorum-ratio: 51% >> >> ``` > > >> >> Carefully consulting docs, the NFS client is preferred client solution >> for better 'ls' performance. However, this better performance comes >> from caching meta info locally, I think, and the caching mechanism >> will cause the penalty of data coherence, right? >> >> I want to know what's the best or mature way to trade-off the 'ls ' >> performance with data coherence in in reality? Any comments are >> welcome. >> >> Thanks. >> >> -Fei >> ___ >> Gluster-devel mailing list >> Gluster-devel@gluster.org >> http://lists.gluster.org/mailman/listinfo/gluster-devel ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] Any mature(better) solution(way) to handle slow performance on 'ls -l, '.
Hi experts on glusterFS, In our testbed, we found that the ' ls -l' performance is pretty slow. Indeed from the prospect of glusterFS design space, we need to avoid 'ls ' directory which will traverse all bricks sequentially in our current knowledge. We use generic setting for our testbed: ``` Volume Name: gv0 Type: Distributed-Replicate Volume ID: 4a6f96f8-b3fb-4550-bd19-e1a5dffad4d0 Status: Started Snapshot Count: 0 Number of Bricks: 19 x 3 = 57 Transport-type: tcp Bricks: ... Options Reconfigured: features.inode-quota: off features.quota: off cluster.quorum-reads: on cluster.quorum-count: 2 cluster.quorum-type: fixed transport.address-family: inet nfs.disable: on performance.client-io-threads: off cluster.server-quorum-ratio: 51% ``` Carefully consulting docs, the NFS client is preferred client solution for better 'ls' performance. However, this better performance comes from caching meta info locally, I think, and the caching mechanism will cause the penalty of data coherence, right? I want to know what's the best or mature way to trade-off the 'ls ' performance with data coherence in in reality? Any comments are welcome. Thanks. -Fei ___ Gluster-devel mailing list Gluster-devel@gluster.org http://lists.gluster.org/mailman/listinfo/gluster-devel
[Gluster-devel] How gluster handle split-brain in the corner case from non-overlapping range lock in same file?
Hi,
https://docs.gluster.org/en/v3/Administrator%20Guide/arbiter-volumes-and-quorum/,
said,
```
There is a corner case even with replica 3 volumes where the file can
end up in a split-brain. AFR usually takes range locks for the
{offset, length} of the write. If 3 writes happen on the same file at
non-overlapping {offset, length} and each write fails on (only) one
different brick, then we have AFR xattrs of the file blaming each
other.
```
Could some body give more details on it? Any clues are welcome.
Thanks a lot
- Fei
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-devel
