OK, given GlusterFS v3.7.6 with the following patches:
===
Kaleb S KEITHLEY (1):
fuse: use-after-free fix in fuse-bridge, revisited
Pranith Kumar K (1):
mount/fuse: Fix use-after-free crash
Soumya Koduri (3):
gfapi: Fix inode nlookup counts
inode: Retire the inodes from t
Here are the results of "rsync" test. I've got 2 volumes — source and target —
performing multiple files rsyncing from one volume to another.
Source volume:
===
root 22259 3.5 1.5 1204200 771004 ? Ssl Jan23 109:42 /usr/sbin/
glusterfs --volfile-server=glusterfs.example.com --volfile-
Also, I've repeated the same "find" test again, but with glusterfs process
launched under valgrind. And here is valgrind output:
https://gist.github.com/097afb01ebb2c5e9e78d
On неділя, 24 січня 2016 р. 09:33:00 EET Mathieu Chateau wrote:
> Thanks for all your tests and times, it looks promising
BTW, am I the only one who sees in
max_size=4294965480
almost 2^32? Could that be integer overflow?
On неділя, 24 січня 2016 р. 13:23:55 EET Oleksandr Natalenko wrote:
> The leak definitely remains. I did "find /mnt/volume -type d" over GlusterFS
> volume, with mentioned patches applied and with
The leak definitely remains. I did "find /mnt/volume -type d" over GlusterFS
volume, with mentioned patches applied and without "kernel notifier loop
terminated" message, but "glusterfs" process consumed ~4GiB of RAM after
"find" finished.
Here is statedump:
https://gist.github.com/10cde83c63f
Thanks for all your tests and times, it looks promising :)
Cordialement,
Mathieu CHATEAU
http://www.lotp.fr
2016-01-23 22:30 GMT+01:00 Oleksandr Natalenko :
> OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the
> following
> patches:
>
> ===
> Kaleb S KEITHLEY (1):
> fuse:
OK, now I'm re-performing tests with rsync + GlusterFS v3.7.6 + the following
patches:
===
Kaleb S KEITHLEY (1):
fuse: use-after-free fix in fuse-bridge, revisited
Pranith Kumar K (1):
mount/fuse: Fix use-after-free crash
Soumya Koduri (3):
gfapi: Fix inode nlookup counts
On 01/22/2016 01:20 PM, Joe Julian wrote:
>
>
> On 01/22/16 09:53, Kaleb S. KEITHLEY wrote:
>> On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
>>> On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
I presume by this you mean you're not seeing the "kernel notifier loop
>>>
On 01/22/16 09:53, Kaleb S. KEITHLEY wrote:
On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
I presume by this you mean you're not seeing the "kernel notifier loop
terminated" error in your logs.
Correct, but only with sim
On 01/22/2016 12:43 PM, Oleksandr Natalenko wrote:
> On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
>> I presume by this you mean you're not seeing the "kernel notifier loop
>> terminated" error in your logs.
>
> Correct, but only with simple traversing. Have to test under rsy
On пʼятниця, 22 січня 2016 р. 12:32:01 EET Kaleb S. KEITHLEY wrote:
> I presume by this you mean you're not seeing the "kernel notifier loop
> terminated" error in your logs.
Correct, but only with simple traversing. Have to test under rsync.
> Hmmm. My system is not leaking. Last 24 hours the R
On 01/22/2016 12:15 PM, Oleksandr Natalenko wrote:
> OK, compiles and runs well now,
I presume by this you mean you're not seeing the "kernel notifier loop
terminated" error in your logs.
> but still leaks.
Hmmm. My system is not leaking. Last 24 hours the RSZ and VSZ are
stable:
http://downlo
OK, compiles and runs well now, but still leaks. Will try to load the volume
with rsync.
On четвер, 21 січня 2016 р. 20:40:45 EET Kaleb KEITHLEY wrote:
> On 01/21/2016 06:59 PM, Oleksandr Natalenko wrote:
> > I see extra GF_FREE (node); added with two patches:
> >
> > ===
> > $ git diff HEAD~2 |
On 01/21/2016 06:59 PM, Oleksandr Natalenko wrote:
> I see extra GF_FREE (node); added with two patches:
>
> ===
> $ git diff HEAD~2 | gist
> https://gist.github.com/9524fa2054cc48278ea8
> ===
>
> Is that intentionally? I guess I face double-free issue.
>
I presume you're referring to the relea
I see extra GF_FREE (node); added with two patches:
===
$ git diff HEAD~2 | gist
https://gist.github.com/9524fa2054cc48278ea8
===
Is that intentionally? I guess I face double-free issue.
On четвер, 21 січня 2016 р. 17:29:53 EET Kaleb KEITHLEY wrote:
> On 01/20/2016 04:08 AM, Oleksandr Natalenko
With the proposed patches I get the following assertion while copying files to
GlusterFS volume:
===
glusterfs: mem-pool.c:305: __gf_free: Assertion `0xCAFEBABE == header->magic'
failed.
Program received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffe9ffb700 (LWP 12635)]
0x76f215f
On 01/20/2016 04:08 AM, Oleksandr Natalenko wrote:
> Yes, there are couple of messages like this in my logs too (I guess one
> message per each remount):
>
> ===
> [2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
> glusterfs-fuse: kernel notifier loop terminated
> ===
>
I perform the tests using 1) rsync (massive copy of millions of files); 2)
find (simple tree traversing).
To check if memory leak happens, I use find tool. I've performed two
traversing (w/ and w/o fopen-keep-cache=off) with remount between them, but I
didn't encounter "kernel notifier loop ter
If this message appears way before the volume is unmounted, can you try
to start the volume manually using this command and repeat the tests ?
glusterfs --fopen-keep-cache=off --volfile-server=
--volfile-id=/
This will prevent invalidation requests to be sent to the kernel, so
there shouldn
Yes, there are couple of messages like this in my logs too (I guess one
message per each remount):
===
[2016-01-18 23:42:08.742447] I [fuse-bridge.c:3875:notify_kernel_loop] 0-
glusterfs-fuse: kernel notifier loop terminated
===
On середа, 20 січня 2016 р. 09:51:23 EET Xavier Hernandez wrote:
>
I'm seeing a similar problem with 3.7.6.
This latest statedump contains a lot of gf_fuse_mt_invalidate_node_t
objects in fuse. Looking at the code I see they are used to send
invalidations to kernel fuse, however this is done in a separate thread
that writes a log message when it exits. On the
And another statedump of FUSE mount client consuming more than 7 GiB of RAM:
https://gist.github.com/136d7c49193c798b3ade
DHT-related leak?
On середа, 13 січня 2016 р. 16:26:59 EET Soumya Koduri wrote:
> On 01/13/2016 04:08 PM, Soumya Koduri wrote:
> > On 01/12/2016 12:46 PM, Oleksandr Natalenko
Here is another RAM usage stats and statedump of GlusterFS mount approaching
to just another OOM:
===
root 32495 1.4 88.3 4943868 1697316 ? Ssl Jan13 129:18 /usr/sbin/
glusterfs --volfile-server=server.example.com --volfile-id=volume /mnt/volume
===
https://gist.github.com/86198201c79e
I've applied client_cbk_cache_invalidation leak patch, and here are the
results.
Launch:
===
valgrind --leak-check=full --show-leak-kinds=all
--log-file="valgrind_fuse.log" /usr/bin/glusterfs -N
--volfile-server=server.example.com --volfile-id=somevolume
/mnt/somevolume
find /mnt/somevolume
On 01/13/2016 04:08 PM, Soumya Koduri wrote:
On 01/12/2016 12:46 PM, Oleksandr Natalenko wrote:
Just in case, here is Valgrind output on FUSE client with 3.7.6 +
API-related patches we discussed before:
https://gist.github.com/cd6605ca19734c1496a4
Thanks for sharing the results. I made c
On 01/12/2016 12:17 PM, Mathieu Chateau wrote:
I tried like suggested:
echo 3 > /proc/sys/vm/drop_caches
sync
It lower a bit usage:
before:
Images intégrées 2
after:
Images intégrées 1
Thanks Mathieu. There is a drop in memory usage after dropping vfs cache
but doesn't seem significa
On 01/12/2016 12:46 PM, Oleksandr Natalenko wrote:
Just in case, here is Valgrind output on FUSE client with 3.7.6 +
API-related patches we discussed before:
https://gist.github.com/cd6605ca19734c1496a4
Thanks for sharing the results. I made changes to fix one leak reported
there wrt ' cli
I tried like suggested:
echo 3 > /proc/sys/vm/drop_caches
sync
It lower a bit usage:
before:
[image: Images intégrées 2]
after:
[image: Images intégrées 1]
Cordialement,
Mathieu CHATEAU
http://www.lotp.fr
2016-01-12 7:34 GMT+01:00 Mathieu Chateau :
> Hello,
>
> I also experience high m
Hello,
I also experience high memory usage on my gluster clients. Sample :
[image: Images intégrées 1]
Can I help in testing/debugging ?
Cordialement,
Mathieu CHATEAU
http://www.lotp.fr
2016-01-12 7:24 GMT+01:00 Soumya Koduri :
>
>
> On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:
>
>> Br
Just in case, here is Valgrind output on FUSE client with 3.7.6 +
API-related patches we discussed before:
https://gist.github.com/cd6605ca19734c1496a4
12.01.2016 08:24, Soumya Koduri написав:
For fuse client, I tried vfs drop_caches as suggested by Vijay in an
earlier mail. Though all the ino
On 01/11/2016 05:11 PM, Oleksandr Natalenko wrote:
Brief test shows that Ganesha stopped leaking and crashing, so it seems
to be good for me.
Thanks for checking.
Nevertheless, back to my original question: what about FUSE client? It
is still leaking despite all the fixes applied. Should it
Brief test shows that Ganesha stopped leaking and crashing, so it seems
to be good for me.
Nevertheless, back to my original question: what about FUSE client? It
is still leaking despite all the fixes applied. Should it be considered
another issue?
11.01.2016 12:26, Soumya Koduri написав:
I
I have made changes to fix the lookup leak in a different way (as
discussed with Pranith) and uploaded them in the latest patch set #4
- http://review.gluster.org/#/c/13096/
Please check if it resolves the mem leak and hopefully doesn't result in
any assertion :)
Thanks,
Soumya
On 01
I could reproduce while testing deep directories with in the mount
point. I root caus'ed the issue & had discussion with Pranith to
understand the purpose and recommended way of taking nlookup on inodes.
I shall make changes to my existing fix and post the patch soon.
Thanks for your patience!
OK, I've patched GlusterFS v3.7.6 with 43570a01 and 5cffb56b (the most recent
revisions) and NFS-Ganesha v2.3.0 with 8685abfc (most recent revision too).
On traversing GlusterFS volume with many files in one folder via NFS mount I
get an assertion:
===
ganesha.nfsd: inode.c:716: __inode_forget:
On 01/06/2016 01:58 PM, Oleksandr Natalenko wrote:
OK, here is valgrind log of patched Ganesha (I took recent version of
your patchset, 8685abfc6d) with Entries_HWMARK set to 500.
https://gist.github.com/5397c152a259b9600af0
See no huge runtime leaks now.
Glad to hear this :)
However, I've
OK, here is valgrind log of patched Ganesha (I took recent version of
your patchset, 8685abfc6d) with Entries_HWMARK set to 500.
https://gist.github.com/5397c152a259b9600af0
See no huge runtime leaks now. However, I've repeated this test with
another volume in replica and got the following Gan
On 01/06/2016 03:53 AM, Oleksandr Natalenko wrote:
OK, I've repeated the same traversing test with patched GlusterFS API, and
here is new Valgrind log:
https://gist.github.com/17ecb16a11c9aed957f5
Fuse mount doesn't use gfapi helper. Does your above GlusterFS API
application call glfs_fini()
OK, I've repeated the same traversing test with patched GlusterFS API, and
here is new Valgrind log:
https://gist.github.com/17ecb16a11c9aed957f5
Still leaks.
On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:
> On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:
> > Unfortunately, b
Correct, I used FUSE mount. Shouldn't gfapi be used by FUSE mount helper (/
usr/bin/glusterfs)?
On вівторок, 5 січня 2016 р. 22:52:25 EET Soumya Koduri wrote:
> On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:
> > Unfortunately, both patches didn't make any difference for me.
> >
> > I've patch
On 01/05/2016 05:56 PM, Oleksandr Natalenko wrote:
Unfortunately, both patches didn't make any difference for me.
I've patched 3.7.6 with both patches, recompiled and installed patched
GlusterFS package on client side and mounted volume with ~2M of files.
The I performed usual tree traverse wi
Unfortunately, both patches didn't make any difference for me.
I've patched 3.7.6 with both patches, recompiled and installed patched
GlusterFS package on client side and mounted volume with ~2M of files.
The I performed usual tree traverse with simple "find".
Memory RES value went from ~130M
I tried to debug the inode* related leaks and seen some improvements
after applying the below patches when ran the same test (but will
smaller load). Could you please apply those patches & confirm the same?
a) http://review.gluster.org/13125
This will fix the inodes & their ctx related leaks d
Here is another Valgrind log of similar scenario but with drop_caches before
umount:
https://gist.github.com/06997ecc8c7bce83aec1
Also, I've tried to drop caches on production VM with GluserFS volume mounted
and memleaking for several weeks with absolutely no effect:
===
root 945 0.1 48
On 01/03/2016 09:23 AM, Oleksandr Natalenko wrote:
Another Valgrind run.
I did the following:
===
valgrind --leak-check=full --show-leak-kinds=all --log-
file="valgrind_fuse.log" /usr/bin/glusterfs -N --volfile-
server=some.server.com --volfile-id=somevolume /mnt/volume
===
then cd to /mnt/vol
age -
> >
> >> From: "Pranith Kumar Karampuri"
> >> To: "Oleksandr Natalenko" , "Soumya Koduri"
> >> Cc: gluster-us...@gluster.org,
> >> gluster-devel@gluster.org
> >> Sent: Monday, December 28, 2015 9:32:07 AM
>
: Re: [Gluster-devel] [Gluster-users] Memory leak in GlusterFS FUSE
client
On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:
Also, here is valgrind output with our custom tool, that does GlusterFS
volume
traversing (with simple stats) just like find tool. In this case
NFS-Ganesha
is not
- Original Message -
> From: "Pranith Kumar Karampuri"
> To: "Oleksandr Natalenko" , "Soumya Koduri"
>
> Cc: gluster-us...@gluster.org, gluster-devel@gluster.org
> Sent: Monday, December 28, 2015 9:32:07 AM
> Subject: Re: [Gluster-deve
On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:
Also, here is valgrind output with our custom tool, that does GlusterFS volume
traversing (with simple stats) just like find tool. In this case NFS-Ganesha
is not used.
https://gist.github.com/e4602a50d3c98f7a2766
hi Oleksandr,
I went t
Thanks for sharing the results. Shall look at the leaks and update.
-Soumya
On 12/26/2015 04:45 AM, Oleksandr Natalenko wrote:
Also, here is valgrind output with our custom tool, that does GlusterFS volume
traversing (with simple stats) just like find tool. In this case NFS-Ganesha
is not used.
Also, here is valgrind output with our custom tool, that does GlusterFS volume
traversing (with simple stats) just like find tool. In this case NFS-Ganesha
is not used.
https://gist.github.com/e4602a50d3c98f7a2766
One may see GlusterFS-related leaks here as well.
On пʼятниця, 25 грудня 2015 р.
OK, I've rebuild GlusterFS v3.7.6 with debug enabled as well as NFS-Ganesha
with debug enabled as well (and libc allocator).
Here is my test steps:
1. launch nfs-ganesha:
valgrind --leak-check=full --show-leak-kinds=all --log-file="valgrind.log" /
opt/nfs-ganesha/bin/ganesha.nfsd -F -L ./ganesh
1. test with Cache_Size = 256 and Entries_HWMark = 4096
Before find . -type f:
root 3120 0.6 11.0 879120 208408 ? Ssl 17:39 0:00 /usr/bin/
ganesha.nfsd -L /var/log/ganesha.log -f /etc/ganesha/ganesha.conf -N NIV_EVENT
After:
root 3120 11.4 24.3 1170076 458168 ? Ssl 17:
On 12/25/2015 08:56 PM, Oleksandr Natalenko wrote:
What units Cache_Size is measured in? Bytes?
Its actually (Cache_Size * sizeof_ptr) bytes. If possible, could you
please run ganesha process under valgrind? Will help in detecting leaks.
Thanks,
Soumya
25.12.2015 16:58, Soumya Koduri напи
What units Cache_Size is measured in? Bytes?
25.12.2015 16:58, Soumya Koduri написав:
On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type
On 12/24/2015 09:17 PM, Oleksandr Natalenko wrote:
Another addition: it seems to be GlusterFS API library memory leak
because NFS-Ganesha also consumes huge amount of memory while doing
ordinary "find . -type f" via NFSv4.2 on remote client. Here is memory
usage:
===
root 5416 34.2 78.5 2
Here are two consecutive statedumps of brick in question memory usage
[1] [2]. glusterfs client process went from ~630 MB to ~1350 MB of
memory usage in less than one hour.
Volume options:
===
cluster.lookup-optimize: on
cluster.readdir-optimize: on
client.event-threads: 4
network.inode-lru-li
Hi Oleksandr,
You are right. The description should have said it as the limit on the
number of inodes in the lru list of the inode cache. I have sent a patch
for that.
http://review.gluster.org/#/c/12242/
Regards,
Raghavendra Bhat
On Thu, Sep 24, 2015 at 1:44 PM, Oleksandr Natalenko <
oleksa...
oh, my bad...
coulb be this one?
https://bugzilla.redhat.com/show_bug.cgi?id=1126831
Anyway, on ovirt+gluster w I experienced similar behavior...
On Thu, Sep 24, 2015 at 10:32 AM, Oleksandr Natalenko <
oleksa...@natalenko.name> wrote:
> We use bare GlusterFS installation with no oVirt involved.
google vdsm memory leak..it's been discussed on list last year and earlier
this one...
On Thu, Sep 24, 2015 at 10:14 AM, Oleksandr Natalenko <
oleksa...@natalenko.name> wrote:
> In our GlusterFS deployment we've encountered something like memory leak
> in GlusterFS FUSE client.
>
> We use replica
I've checked statedump of volume in question and haven't found lots of
iobuf as mentioned in that bugreport.
However, I've noticed that there are lots of LRU records like this:
===
[conn.1.bound_xl./bricks/r6sdLV07_vd0_mail/mail.lru.1]
gfid=c4b29310-a19d-451b-8dd1-b3ac2d86b595
nlookup=1
fd-coun
We use bare GlusterFS installation with no oVirt involved.
24.09.2015 10:29, Gabi C wrote:
google vdsm memory leak..it's been discussed on list last year and
earlier this one...
___
Gluster-devel mailing list
Gluster-devel@gluster.org
http://www.glust
62 matches
Mail list logo
