Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

OK, given GlusterFS v3.7.6 with the following patches: === Kaleb S KEITHLEY (1): fuse: use-after-free fix in fuse-bridge, revisited Pranith Kumar K (1): mount/fuse: Fix use-after-free crash Soumya Koduri (3): gfapi: Fix inode nlookup counts inode: Retire the inodes from

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Here are the results of "rsync" test. I've got 2 volumes — source and target — performing multiple files rsyncing from one volume to another. Source volume: === root 22259 3.5 1.5 1204200 771004 ? Ssl Jan23 109:42 /usr/sbin/ glusterfs --volfile-server=glusterfs.example.com

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Also, I've repeated the same "find" test again, but with glusterfs process launched under valgrind. And here is valgrind output: https://gist.github.com/097afb01ebb2c5e9e78d On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote: > Thanks for all your tests and times, it looks promising

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

BTW, am I the only one who sees in max_size=4294965480 almost 2^32? Could that be integer overflow? On неділя, 24 січня 2016 р. 13:23:55 EET Oleksandr Natalenko wrote: > The leak definitely remains. I did "find /mnt/volume -type d" over GlusterFS > volume, with mentioned patches applied and

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

The leak definitely remains. I did "find /mnt/volume -type d" over GlusterFS volume, with mentioned patches applied and without "kernel notifier loop terminated" message, but "glusterfs" process consumed ~4GiB of RAM after "find" finished. Here is statedump:

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the following patches: === Kaleb S KEITHLEY (1): fuse: use-after-free fix in fuse-bridge, revisited Pranith Kumar K (1): mount/fuse: Fix use-after-free crash Soumya Koduri (3): gfapi: Fix inode nlookup counts

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/22/2016 01:20 PM, Joe Julian wrote: > > > On 01/22/16 09:53, Kaleb S. KEITHLEY wrote: >> On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote: >>> On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote: I presume by this you mean you're not seeing the "kernel notifier loop

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/22/2016 12:15 PM, Oleksandr Natalenko wrote: > OK, compiles and runs well now, I presume by this you mean you're not seeing the "kernel notifier loop terminated" error in your logs. > but still leaks. Hmmm. My system is not leaking. Last 24 hours the RSZ and VSZ are stable:

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote: > On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote: >> I presume by this you mean you're not seeing the "kernel notifier loop >> terminated" error in your logs. > > Correct, but only with simple traversing. Have to test under

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I perform the tests using 1) rsync (massive copy of millions of files); 2) find (simple tree traversing). To check if memory leak happens, I use find tool. I've performed two traversing (w/ and w/o fopen-keep-cache=off) with remount between them, but I didn't encounter "kernel notifier loop

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

If this message appears way before the volume is unmounted, can you try to start the volume manually using this command and repeat the tests ? glusterfs --fopen-keep-cache=off --volfile-server= --volfile-id=/ This will prevent invalidation requests to be sent to the kernel, so there

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I see extra GF_FREE (node); added with two patches: === $ git diff HEAD~2 | gist https://gist.github.com/9524fa2054cc48278ea8 === Is that intentionally? I guess I face double-free issue. On четвер, 21 січня 2016 р. 17:29:53 EET Kaleb KEITHLEY wrote: > On 01/20/2016 04:08 AM, Oleksandr Natalenko

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

With the proposed patches I get the following assertion while copying files to GlusterFS volume: === glusterfs: mem-pool.c:305: __gf_free: Assertion `0xCAFEBABE == header->magic' failed. Program received signal SIGABRT, Aborted. [Switching to Thread 0x7fffe9ffb700 (LWP 12635)]

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote: > Yes, there are couple of messages like this in my logs too (I guess one > message per each remount): > > === > [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0- > glusterfs-fuse: kernel notifier loop terminated > === >

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/21/2016 06:59 PM, Oleksandr Natalenko wrote: > I see extra GF_FREE (node); added with two patches: > > === > $ git diff HEAD~2 | gist > https://gist.github.com/9524fa2054cc48278ea8 > === > > Is that intentionally? I guess I face double-free issue. > I presume you're referring to the

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I'm seeing a similar problem with 3.7.6. This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t objects in fuse. Looking at the code I see they are used to send invalidations to kernel fuse, however this is done in a separate thread that writes a log message when it exits. On

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Yes, there are couple of messages like this in my logs too (I guess one message per each remount): === [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0- glusterfs-fuse: kernel notifier loop terminated === On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote: >

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Here is another RAM usage stats and statedump of GlusterFS mount approaching to just another OOM: === root 32495 1.4 88.3 4943868 1697316 ? Ssl Jan13 129:18 /usr/sbin/ glusterfs --volfile-server=server.example.com --volfile-id=volume /mnt/volume ===

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

And another statedump of FUSE mount client consuming more than 7 GiB of RAM: https://gist.github.com/136d7c49193c798b3ade DHT-related leak? On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote: > On 01/13/2016 04:08 PM, Soumya Koduri wrote: > > On 01/12/2016 12:46 PM, Oleksandr

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/13/2016 04:08 PM, Soumya Koduri wrote: On 01/12/2016 12:46 PM, Oleksandr Natalenko wrote: Just in case, here is Valgrind output on FUSE client with 3.7.6 + API-related patches we discussed before: https://gist.github.com/cd6605ca19734c1496a4 Thanks for sharing the results. I made

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/12/2016 12:17 PM, Mathieu Chateau wrote: I tried like suggested: echo 3 > /proc/sys/vm/drop_caches sync It lower a bit usage: before: Images intégrées 2 after: Images intégrées 1 Thanks Mathieu. There is a drop in memory usage after dropping vfs cache but doesn't seem

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Hello, I also experience high memory usage on my gluster clients. Sample : [image: Images intégrées 1] Can I help in testing/debugging ? Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-12 7:24 GMT+01:00 Soumya Koduri : > > > On 01/11/2016 05:11 PM, Oleksandr

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I tried like suggested: echo 3 > /proc/sys/vm/drop_caches sync It lower a bit usage: before: [image: Images intégrées 2] after: [image: Images intégrées 1] Cordialement, Mathieu CHATEAU http://www.lotp.fr 2016-01-12 7:34 GMT+01:00 Mathieu Chateau : > Hello, >

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I have made changes to fix the lookup leak in a different way (as discussed with Pranith) and uploaded them in the latest patch set #4 - http://review.gluster.org/#/c/13096/ Please check if it resolves the mem leak and hopefully doesn't result in any assertion :) Thanks, Soumya On

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote: Brief test shows that Ganesha stopped leaking and crashing, so it seems to be good for me. Thanks for checking. Nevertheless, back to my original question: what about FUSE client? It is still leaking despite all the fixes applied. Should

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Just in case, here is Valgrind output on FUSE client with 3.7.6 + API-related patches we discussed before: https://gist.github.com/cd6605ca19734c1496a4 12.01.2016 08:24, Soumya Koduri написав: For fuse client, I tried vfs drop_caches as suggested by Vijay in an earlier mail. Though all the

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Brief test shows that Ganesha stopped leaking and crashing, so it seems to be good for me. Nevertheless, back to my original question: what about FUSE client? It is still leaking despite all the fixes applied. Should it be considered another issue? 11.01.2016 12:26, Soumya Koduri написав:

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I could reproduce while testing deep directories with in the mount point. I root caus'ed the issue & had discussion with Pranith to understand the purpose and recommended way of taking nlookup on inodes. I shall make changes to my existing fix and post the patch soon. Thanks for your patience!

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

OK, here is valgrind log of patched Ganesha (I took recent version of your patchset, 8685abfc6d) with Entries_HWMARK set to 500. https://gist.github.com/5397c152a259b9600af0 See no huge runtime leaks now. However, I've repeated this test with another volume in replica and got the following

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/06/2016 01:58 PM, Oleksandr Natalenko wrote: OK, here is valgrind log of patched Ganesha (I took recent version of your patchset, 8685abfc6d) with Entries_HWMARK set to 500. https://gist.github.com/5397c152a259b9600af0 See no huge runtime leaks now. Glad to hear this :) However,

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I tried to debug the inode* related leaks and seen some improvements after applying the below patches when ran the same test (but will smaller load). Could you please apply those patches & confirm the same? a) http://review.gluster.org/13125 This will fix the inodes & their ctx related leaks

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Correct, I used FUSE mount. Shouldn't gfapi be used by FUSE mount helper (/ usr/bin/glusterfs)? On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote: > On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote: > > Unfortunately, both patches didn't make any difference for me. > > > > I've

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote: Unfortunately, both patches didn't make any difference for me. I've patched 3.7.6 with both patches, recompiled and installed patched GlusterFS package on client side and mounted volume with ~2M of files. The I performed usual tree traverse

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/06/2016 03:53 AM, Oleksandr Natalenko wrote: OK, I've repeated the same traversing test with patched GlusterFS API, and here is new Valgrind log: https://gist.github.com/17ecb16a11c9aed957f5 Fuse mount doesn't use gfapi helper. Does your above GlusterFS API application call

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

OK, I've repeated the same traversing test with patched GlusterFS API, and here is new Valgrind log: https://gist.github.com/17ecb16a11c9aed957f5 Still leaks. On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote: > On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote: > > Unfortunately,

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Unfortunately, both patches didn't make any difference for me. I've patched 3.7.6 with both patches, recompiled and installed patched GlusterFS package on client side and mounted volume with ~2M of files. The I performed usual tree traverse with simple "find". Memory RES value went from

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

-devel@gluster.org > >> Sent: Monday, December 28, 2015 9:32:07 AM > >> Subject: Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS > >> FUSE client>> > >> On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote: > >>> Also, here

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 01/03/2016 09:23 AM, Oleksandr Natalenko wrote: Another Valgrind run. I did the following: === valgrind --leak-check=full --show-leak-kinds=all --log- file="valgrind_fuse.log" /usr/bin/glusterfs -N --volfile- server=some.server.com --volfile-id=somevolume /mnt/volume === then cd to

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Here is another Valgrind log of similar scenario but with drop_caches before umount: https://gist.github.com/06997ecc8c7bce83aec1 Also, I've tried to drop caches on production VM with GluserFS volume mounted and memleaking for several weeks with absolutely no effect: === root 945 0.1

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

r.org > Sent: Monday, December 28, 2015 9:32:07 AM > Subject: Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE > client > > > > On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote: > > Also, here is valgrind output with our custom tool, that does Gl

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote: Also, here is valgrind output with our custom tool, that does GlusterFS volume traversing (with simple stats) just like find tool. In this case NFS-Ganesha is not used. https://gist.github.com/e4602a50d3c98f7a2766 hi Oleksandr, I went

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Thanks for sharing the results. Shall look at the leaks and update. -Soumya On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote: Also, here is valgrind output with our custom tool, that does GlusterFS volume traversing (with simple stats) just like find tool. In this case NFS-Ganesha is not

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

1. test with Cache_Size = 256 and Entries_HWMark = 4096 Before find . -type f: root 3120 0.6 11.0 879120 208408 ? Ssl 17:39 0:00 /usr/bin/ ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT After: root 3120 11.4 24.3 1170076 458168 ? Ssl

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 12/25/2015 08:56 PM, Oleksandr Natalenko wrote: What units Cache_Size is measured in? Bytes? Its actually (Cache_Size * sizeof_ptr) bytes. If possible, could you please run ganesha process under valgrind? Will help in detecting leaks. Thanks, Soumya 25.12.2015 16:58, Soumya Koduri

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

What units Cache_Size is measured in? Bytes? 25.12.2015 16:58, Soumya Koduri написав: On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote: Another addition: it seems to be GlusterFS API library memory leak because NFS-Ganesha also consumes huge amount of memory while doing ordinary "find . -type

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote: Another addition: it seems to be GlusterFS API library memory leak because NFS-Ganesha also consumes huge amount of memory while doing ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory usage: === root 5416 34.2 78.5

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Also, here is valgrind output with our custom tool, that does GlusterFS volume traversing (with simple stats) just like find tool. In this case NFS-Ganesha is not used. https://gist.github.com/e4602a50d3c98f7a2766 One may see GlusterFS-related leaks here as well. On пʼятниця, 25 грудня 2015

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

OK, I've rebuild GlusterFS v3.7.6 with debug enabled as well as NFS-Ganesha with debug enabled as well (and libc allocator). Here is my test steps: 1. launch nfs-ganesha: valgrind --leak-check=full --show-leak-kinds=all --log-file="valgrind.log" / opt/nfs-ganesha/bin/ganesha.nfsd -F -L

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Here are two consecutive statedumps of brick in question memory usage [1] [2]. glusterfs client process went from ~630 MB to ~1350 MB of memory usage in less than one hour. Volume options: === cluster.lookup-optimize: on cluster.readdir-optimize: on client.event-threads: 4

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

Hi Oleksandr, You are right. The description should have said it as the limit on the number of inodes in the lru list of the inode cache. I have sent a patch for that. http://review.gluster.org/#/c/12242/ Regards, Raghavendra Bhat On Thu, Sep 24, 2015 at 1:44 PM, Oleksandr Natalenko <

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

google vdsm memory leak..it's been discussed on list last year and earlier this one... On Thu, Sep 24, 2015 at 10:14 AM, Oleksandr Natalenko < oleksa...@natalenko.name> wrote: > In our GlusterFS deployment we've encountered something like memory leak > in GlusterFS FUSE client. > > We use

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

I've checked statedump of volume in question and haven't found lots of iobuf as mentioned in that bugreport. However, I've noticed that there are lots of LRU records like this: === [conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1] gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595 nlookup=1

Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE client

We use bare GlusterFS installation with no oVirt involved. 24.09.2015 10:29, Gabi C wrote: google vdsm memory leak..it's been discussed on list last year and earlier this one... ___ Gluster-devel mailing list Gluster-devel@gluster.org