Re: [Gluster-users] Run away memory with gluster mount

[Nithya Balachandran]

On 21 February 2018 at 21:11, Dan Ragle <dan...@biblestuph.com> wrote:

>
>
> On 2/3/2018 8:58 AM, Dan Ragle wrote:
>
>>
>>
>> On 2/2/2018 2:13 AM, Nithya Balachandran wrote:
>>
>>> Hi Dan,
>>>
>>> It sounds like you might be running into [1]. The patch has been posted
>>> upstream and the fix should be in the next release.
>>> In the meantime, I'm afraid there is no way to get around this without
>>> restarting the process.
>>>
>>> Regards,
>>> Nithya
>>>
>>> [1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264
>>>
>>>
>> Much appreciated. Will watch for the next release and retest then.
>>
>> Cheers!
>>
>> Dan
>>
>>
> FYI, this looks like it's fixed in 3.12.6. Ran the test setup with
> repeated ls listings for just shy of 48 hours with no increase in RAM
> usage. Next will try my production application load for awhile to see if it
> holds steady.
>
> The gf_dht_mt_dht_layout_t memusage num_allocs went quickly up to 105415
> and then stayed there for the entire 48 hours.
>
>
Excellent. Thanks for letting us know.

Nithya


> Thanks for the quick response,
>
> Dan
>
>
>>> On 2 February 2018 at 02:57, Dan Ragle <dan...@biblestuph.com >> dan...@biblestuph.com>> wrote:
>>>
>>>
>>>
>>> On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:
>>>
>>>
>>>
>>> - Original Message -
>>>
>>> From: "Dan Ragle" <dan...@biblestuph.com>
>>> To: "Raghavendra Gowdappa" <rgowd...@redhat.com
>>> <mailto:rgowd...@redhat.com>>, "Ravishankar N"
>>> <ravishan...@redhat.com <mailto:ravishan...@redhat.com>>
>>>     Cc: gluster-users@gluster.org
>>> <mailto:gluster-users@gluster.org>, "Csaba Henk"
>>> <ch...@redhat.com <mailto:ch...@redhat.com>>, "Niels de Vos"
>>> <nde...@redhat.com <mailto:nde...@redhat.com>>, "Nithya
>>> Balachandran" <nbala...@redhat.com >> nbala...@redhat.com>>
>>> Sent: Monday, January 29, 2018 9:02:21 PM
>>> Subject: Re: [Gluster-users] Run away memory with gluster
>>> mount
>>>
>>>
>>>
>>> On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:
>>>
>>>
>>>
>>> - Original Message -
>>>
>>> From: "Ravishankar N" <ravishan...@redhat.com
>>> <mailto:ravishan...@redhat.com>>
>>>             To: "Dan Ragle" <dan...@biblestuph.com>,
>>> gluster-users@gluster.org
>>> <mailto:gluster-users@gluster.org>
>>> Cc: "Csaba Henk" <ch...@redhat.com
>>> <mailto:ch...@redhat.com>>, "Niels de Vos"
>>> <nde...@redhat.com <mailto:nde...@redhat.com>>,
>>> "Nithya Balachandran" <nbala...@redhat.com
>>> <mailto:nbala...@redhat.com>>,
>>> "Raghavendra Gowdappa" <rgowd...@redhat.com
>>> <mailto:rgowd...@redhat.com>>
>>> Sent: Saturday, January 27, 2018 10:23:38 AM
>>> Subject: Re: [Gluster-users] Run away memory with
>>> gluster mount
>>>
>>>
>>>
>>> On 01/27/2018 02:29 AM, Dan Ragle wrote:
>>>
>>>
>>> On 1/25/2018 8:21 PM, Ravishankar N wrote:
>>>
>>>
>>>
>>> On 01/25/2018 11:04 PM, Dan Ragle wrote:
>>>
>>> *sigh* trying again to correct
>>> formatting ... apologize for the
>>> earlier mess.
>>>
>>> Having a memory issue with Gluster
>>> 3.12.4 and not sure how to
>>> troubleshoot. I don't *think* this is
>>> expected behavior.
>>>
>>>   

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 2/3/2018 8:58 AM, Dan Ragle wrote:



On 2/2/2018 2:13 AM, Nithya Balachandran wrote:

Hi Dan,

It sounds like you might be running into [1]. The patch has been 
posted upstream and the fix should be in the next release.
In the meantime, I'm afraid there is no way to get around this without 
restarting the process.


Regards,
Nithya

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264



Much appreciated. Will watch for the next release and retest then.

Cheers!

Dan



FYI, this looks like it's fixed in 3.12.6. Ran the test setup with 
repeated ls listings for just shy of 48 hours with no increase in RAM 
usage. Next will try my production application load for awhile to see if 
it holds steady.


The gf_dht_mt_dht_layout_t memusage num_allocs went quickly up to 105415 
and then stayed there for the entire 48 hours.


Thanks for the quick response,

Dan



On 2 February 2018 at 02:57, Dan Ragle <dan...@biblestuph.com 
<mailto:dan...@biblestuph.com>> wrote:




    On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:



    - Original Message -

    From: "Dan Ragle" <dan...@biblestuph.com>
    To: "Raghavendra Gowdappa" <rgowd...@redhat.com
    <mailto:rgowd...@redhat.com>>, "Ravishankar N"
    <ravishan...@redhat.com <mailto:ravishan...@redhat.com>>
    Cc: gluster-users@gluster.org
    <mailto:gluster-users@gluster.org>, "Csaba Henk"
    <ch...@redhat.com <mailto:ch...@redhat.com>>, "Niels de Vos"
    <nde...@redhat.com <mailto:nde...@redhat.com>>, "Nithya
    Balachandran" <nbala...@redhat.com 
<mailto:nbala...@redhat.com>>

    Sent: Monday, January 29, 2018 9:02:21 PM
    Subject: Re: [Gluster-users] Run away memory with gluster 
mount




    On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:



    - Original Message -

    From: "Ravishankar N" <ravishan...@redhat.com
    <mailto:ravishan...@redhat.com>>
    To: "Dan Ragle" <dan...@biblestuph.com>,
    gluster-users@gluster.org
    <mailto:gluster-users@gluster.org>
    Cc: "Csaba Henk" <ch...@redhat.com
    <mailto:ch...@redhat.com>>, "Niels de Vos"
    <nde...@redhat.com <mailto:nde...@redhat.com>>,
    "Nithya Balachandran" <nbala...@redhat.com
    <mailto:nbala...@redhat.com>>,
    "Raghavendra Gowdappa" <rgowd...@redhat.com
    <mailto:rgowd...@redhat.com>>
    Sent: Saturday, January 27, 2018 10:23:38 AM
    Subject: Re: [Gluster-users] Run away memory with
    gluster mount



    On 01/27/2018 02:29 AM, Dan Ragle wrote:


    On 1/25/2018 8:21 PM, Ravishankar N wrote:



    On 01/25/2018 11:04 PM, Dan Ragle wrote:

    *sigh* trying again to correct
    formatting ... apologize for the
    earlier mess.

    Having a memory issue with Gluster
    3.12.4 and not sure how to
    troubleshoot. I don't *think* this is
    expected behavior.

    This is on an updated CentOS 7 box. The
    setup is a simple two node
    replicated layout where the two nodes
    act as both server and
    client.

    The volume in question:

    Volume Name: GlusterWWW
    Type: Replicate
    Volume ID:
    8e9b0e79-f309-4d9b-a5bb-45d065f3
    Status: Started
    Snapshot Count: 0
    Number of Bricks: 1 x 2 = 2
    Transport-type: tcp
    Bricks:
    Brick1:

vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www

    Brick2:

vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www

    Options Reconfigured:
    nfs.disable: on
    cluster.favorite-child-policy: mtime
    transport.address-family: inet

   

Re: [Gluster-users] Run away memory with gluster mount

[Raghavendra Gowdappa]

I missed your reply :). Sorry about that.

- Original Message -
> From: "Raghavendra Gowdappa" <rgowd...@redhat.com>
> To: "Dan Ragle" <dan...@biblestuph.com>
> Cc: "Csaba Henk" <ch...@redhat.com>, "gluster-users" 
> <gluster-users@gluster.org>
> Sent: Tuesday, February 6, 2018 1:14:10 AM
> Subject: Re: [Gluster-users] Run away memory with gluster mount
> 
> Hi Dan,
> 
> I had a suggestion and a question in my previous response. Let us know
> whether the suggestion helps and please let us know about your data-set
> (like how many directories/files and how these directories/files are
> organised) to understand the problem better.
> 
> 
> 
> > In the
> > meantime can you remount glusterfs with options
> > --entry-timeout=0 and --attribute-timeout=0? This will make sure
> > that kernel won't cache inodes/attributes of the file and should
> > bring down the memory usage.
> >
> > I am curious to know what is your data-set like? Is it the case
> > of too many directories and files present in deep directories? I
> > am wondering whether a significant number of inodes cached by
> > kernel are there to hold dentry structure in kernel.
> 
> 
> 
> regards,
> Raghavendra
> 
> - Original Message -
> > From: "Dan Ragle" <dan...@biblestuph.com>
> > To: "Nithya Balachandran" <nbala...@redhat.com>
> > Cc: "gluster-users" <gluster-users@gluster.org>, "Csaba Henk"
> > <ch...@redhat.com>
> > Sent: Saturday, February 3, 2018 7:28:15 PM
> > Subject: Re: [Gluster-users] Run away memory with gluster mount
> > 
> > 
> > 
> > On 2/2/2018 2:13 AM, Nithya Balachandran wrote:
> > > Hi Dan,
> > > 
> > > It sounds like you might be running into [1]. The patch has been posted
> > > upstream and the fix should be in the next release.
> > > In the meantime, I'm afraid there is no way to get around this without
> > > restarting the process.
> > > 
> > > Regards,
> > > Nithya
> > > 
> > > [1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264
> > > 
> > 
> > Much appreciated. Will watch for the next release and retest then.
> > 
> > Cheers!
> > 
> > Dan
> > 
> > > 
> > > On 2 February 2018 at 02:57, Dan Ragle <dan...@biblestuph.com
> > > <mailto:dan...@biblestuph.com>> wrote:
> > > 
> > > 
> > > 
> > > On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:
> > > 
> > > 
> > > 
> > > - Original Message -
> > > 
> > > From: "Dan Ragle" <dan...@biblestuph.com>
> > > To: "Raghavendra Gowdappa" <rgowd...@redhat.com
> > > <mailto:rgowd...@redhat.com>>, "Ravishankar N"
> > > <ravishan...@redhat.com <mailto:ravishan...@redhat.com>>
> > > Cc: gluster-users@gluster.org
> > > <mailto:gluster-users@gluster.org>, "Csaba Henk"
> > > <ch...@redhat.com <mailto:ch...@redhat.com>>, "Niels de Vos"
> > > <nde...@redhat.com <mailto:nde...@redhat.com>>, "Nithya
> > > Balachandran" <nbala...@redhat.com
> > > <mailto:nbala...@redhat.com>>
> > > Sent: Monday, January 29, 2018 9:02:21 PM
> > > Subject: Re: [Gluster-users] Run away memory with gluster
> > > mount
> > > 
> > > 
> > > 
> > > On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:
> > > 
> > > 
> > > 
> > > - Original Message -----
> > > 
> > >     From: "Ravishankar N" <ravishan...@redhat.com
> > > <mailto:ravishan...@redhat.com>>
> > > To: "Dan Ragle" <dan...@biblestuph.com>,
> > > gluster-users@gluster.org
> > > <mailto:gluster-users@gluster.org>
> > > Cc: "Csaba Henk" <ch...@redhat.com
> > > <mailto:ch...@redhat.com>>, "Niels de Vos"
> > > <nde...@redhat.com <mailto:

Re: [Gluster-users] Run away memory with gluster mount

[Raghavendra Gowdappa]

Hi Dan,

I had a suggestion and a question in my previous response. Let us know whether 
the suggestion helps and please let us know about your data-set (like how many 
directories/files and how these directories/files are organised) to understand 
the problem better.



> In the
> meantime can you remount glusterfs with options
> --entry-timeout=0 and --attribute-timeout=0? This will make sure
> that kernel won't cache inodes/attributes of the file and should
> bring down the memory usage.
>
> I am curious to know what is your data-set like? Is it the case
> of too many directories and files present in deep directories? I
> am wondering whether a significant number of inodes cached by
> kernel are there to hold dentry structure in kernel.



regards,
Raghavendra

- Original Message -
> From: "Dan Ragle" <dan...@biblestuph.com>
> To: "Nithya Balachandran" <nbala...@redhat.com>
> Cc: "gluster-users" <gluster-users@gluster.org>, "Csaba Henk" 
> <ch...@redhat.com>
> Sent: Saturday, February 3, 2018 7:28:15 PM
> Subject: Re: [Gluster-users] Run away memory with gluster mount
> 
> 
> 
> On 2/2/2018 2:13 AM, Nithya Balachandran wrote:
> > Hi Dan,
> > 
> > It sounds like you might be running into [1]. The patch has been posted
> > upstream and the fix should be in the next release.
> > In the meantime, I'm afraid there is no way to get around this without
> > restarting the process.
> > 
> > Regards,
> > Nithya
> > 
> > [1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264
> > 
> 
> Much appreciated. Will watch for the next release and retest then.
> 
> Cheers!
> 
> Dan
> 
> > 
> > On 2 February 2018 at 02:57, Dan Ragle <dan...@biblestuph.com
> > <mailto:dan...@biblestuph.com>> wrote:
> > 
> > 
> > 
> > On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:
> > 
> > 
> > 
> > - Original Message -
> > 
> > From: "Dan Ragle" <dan...@biblestuph.com>
> > To: "Raghavendra Gowdappa" <rgowd...@redhat.com
> > <mailto:rgowd...@redhat.com>>, "Ravishankar N"
> > <ravishan...@redhat.com <mailto:ravishan...@redhat.com>>
> > Cc: gluster-users@gluster.org
> >     <mailto:gluster-users@gluster.org>, "Csaba Henk"
> > <ch...@redhat.com <mailto:ch...@redhat.com>>, "Niels de Vos"
> > <nde...@redhat.com <mailto:nde...@redhat.com>>, "Nithya
> > Balachandran" <nbala...@redhat.com
> > <mailto:nbala...@redhat.com>>
> > Sent: Monday, January 29, 2018 9:02:21 PM
> > Subject: Re: [Gluster-users] Run away memory with gluster mount
> > 
> > 
> > 
> > On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:
> > 
> > 
> > 
> > - Original Message -
> > 
> > From: "Ravishankar N" <ravishan...@redhat.com
> > <mailto:ravishan...@redhat.com>>
> > To: "Dan Ragle" <dan...@biblestuph.com>,
> > gluster-users@gluster.org
> >     <mailto:gluster-users@gluster.org>
> > Cc: "Csaba Henk" <ch...@redhat.com
> > <mailto:ch...@redhat.com>>, "Niels de Vos"
> > <nde...@redhat.com <mailto:nde...@redhat.com>>,
> > "Nithya Balachandran" <nbala...@redhat.com
> > <mailto:nbala...@redhat.com>>,
> > "Raghavendra Gowdappa" <rgowd...@redhat.com
> > <mailto:rgowd...@redhat.com>>
> > Sent: Saturday, January 27, 2018 10:23:38 AM
> > Subject: Re: [Gluster-users] Run away memory with
> > gluster mount
> > 
> > 
> > 
> > On 01/27/2018 02:29 AM, Dan Ragle wrote:
> > 
> > 
> > On 1/25/2018 8:21 PM, Ravishankar N wrote:
> > 
> > 
> > 
> > On 01/25/2018 11:04 PM, Dan Ragle wrote:
> > 
> > *sigh* trying again to correct
> > 

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 2/2/2018 2:13 AM, Nithya Balachandran wrote:

Hi Dan,

It sounds like you might be running into [1]. The patch has been posted 
upstream and the fix should be in the next release.
In the meantime, I'm afraid there is no way to get around this without 
restarting the process.


Regards,
Nithya

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264



Much appreciated. Will watch for the next release and retest then.

Cheers!

Dan



On 2 February 2018 at 02:57, Dan Ragle <dan...@biblestuph.com 
<mailto:dan...@biblestuph.com>> wrote:




On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:



- Original Message -

From: "Dan Ragle" <dan...@biblestuph.com>
To: "Raghavendra Gowdappa" <rgowd...@redhat.com
<mailto:rgowd...@redhat.com>>, "Ravishankar N"
<ravishan...@redhat.com <mailto:ravishan...@redhat.com>>
Cc: gluster-users@gluster.org
<mailto:gluster-users@gluster.org>, "Csaba Henk"
<ch...@redhat.com <mailto:ch...@redhat.com>>, "Niels de Vos"
<nde...@redhat.com <mailto:nde...@redhat.com>>, "Nithya
Balachandran" <nbala...@redhat.com <mailto:nbala...@redhat.com>>
Sent: Monday, January 29, 2018 9:02:21 PM
Subject: Re: [Gluster-users] Run away memory with gluster mount



On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:



- Original Message -

From: "Ravishankar N" <ravishan...@redhat.com
<mailto:ravishan...@redhat.com>>
To: "Dan Ragle" <dan...@biblestuph.com>,
gluster-users@gluster.org
<mailto:gluster-users@gluster.org>
Cc: "Csaba Henk" <ch...@redhat.com
<mailto:ch...@redhat.com>>, "Niels de Vos"
<nde...@redhat.com <mailto:nde...@redhat.com>>,
"Nithya Balachandran" <nbala...@redhat.com
    <mailto:nbala...@redhat.com>>,
    "Raghavendra Gowdappa" <rgowd...@redhat.com
<mailto:rgowd...@redhat.com>>
Sent: Saturday, January 27, 2018 10:23:38 AM
Subject: Re: [Gluster-users] Run away memory with
gluster mount



On 01/27/2018 02:29 AM, Dan Ragle wrote:


On 1/25/2018 8:21 PM, Ravishankar N wrote:



On 01/25/2018 11:04 PM, Dan Ragle wrote:

*sigh* trying again to correct
formatting ... apologize for the
earlier mess.

Having a memory issue with Gluster
3.12.4 and not sure how to
troubleshoot. I don't *think* this is
expected behavior.

This is on an updated CentOS 7 box. The
setup is a simple two node
replicated layout where the two nodes
act as both server and
client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID:
8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1:

vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2:

vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in
there, (increased
cache-size, md invalidation, etc) but
stripped them out in an
attempt to
isolate the issue. Still got the problem
without them.

The volume curren

Re: [Gluster-users] Run away memory with gluster mount

[Nithya Balachandran]

Hi Dan,

It sounds like you might be running into [1]. The patch has been posted
upstream and the fix should be in the next release.
In the meantime, I'm afraid there is no way to get around this without
restarting the process.

Regards,
Nithya

[1]https://bugzilla.redhat.com/show_bug.cgi?id=1541264


On 2 February 2018 at 02:57, Dan Ragle <dan...@biblestuph.com> wrote:

>
>
> On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:
>
>>
>>
>> - Original Message -
>>
>>> From: "Dan Ragle" <dan...@biblestuph.com>
>>> To: "Raghavendra Gowdappa" <rgowd...@redhat.com>, "Ravishankar N" <
>>> ravishan...@redhat.com>
>>> Cc: gluster-users@gluster.org, "Csaba Henk" <ch...@redhat.com>, "Niels
>>> de Vos" <nde...@redhat.com>, "Nithya
>>> Balachandran" <nbala...@redhat.com>
>>> Sent: Monday, January 29, 2018 9:02:21 PM
>>> Subject: Re: [Gluster-users] Run away memory with gluster mount
>>>
>>>
>>>
>>> On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:
>>>
>>>>
>>>>
>>>> - Original Message -
>>>>
>>>>> From: "Ravishankar N" <ravishan...@redhat.com>
>>>>> To: "Dan Ragle" <dan...@biblestuph.com>, gluster-users@gluster.org
>>>>> Cc: "Csaba Henk" <ch...@redhat.com>, "Niels de Vos" <nde...@redhat.com
>>>>> >,
>>>>> "Nithya Balachandran" <nbala...@redhat.com>,
>>>>> "Raghavendra Gowdappa" <rgowd...@redhat.com>
>>>>> Sent: Saturday, January 27, 2018 10:23:38 AM
>>>>> Subject: Re: [Gluster-users] Run away memory with gluster mount
>>>>>
>>>>>
>>>>>
>>>>> On 01/27/2018 02:29 AM, Dan Ragle wrote:
>>>>>
>>>>>>
>>>>>> On 1/25/2018 8:21 PM, Ravishankar N wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 01/25/2018 11:04 PM, Dan Ragle wrote:
>>>>>>>
>>>>>>>> *sigh* trying again to correct formatting ... apologize for the
>>>>>>>> earlier mess.
>>>>>>>>
>>>>>>>> Having a memory issue with Gluster 3.12.4 and not sure how to
>>>>>>>> troubleshoot. I don't *think* this is expected behavior.
>>>>>>>>
>>>>>>>> This is on an updated CentOS 7 box. The setup is a simple two node
>>>>>>>> replicated layout where the two nodes act as both server and
>>>>>>>> client.
>>>>>>>>
>>>>>>>> The volume in question:
>>>>>>>>
>>>>>>>> Volume Name: GlusterWWW
>>>>>>>> Type: Replicate
>>>>>>>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
>>>>>>>> Status: Started
>>>>>>>> Snapshot Count: 0
>>>>>>>> Number of Bricks: 1 x 2 = 2
>>>>>>>> Transport-type: tcp
>>>>>>>> Bricks:
>>>>>>>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
>>>>>>>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
>>>>>>>> Options Reconfigured:
>>>>>>>> nfs.disable: on
>>>>>>>> cluster.favorite-child-policy: mtime
>>>>>>>> transport.address-family: inet
>>>>>>>>
>>>>>>>> I had some other performance options in there, (increased
>>>>>>>> cache-size, md invalidation, etc) but stripped them out in an
>>>>>>>> attempt to
>>>>>>>> isolate the issue. Still got the problem without them.
>>>>>>>>
>>>>>>>> The volume currently contains over 1M files.
>>>>>>>>
>>>>>>>> When mounting the volume, I get (among other things) a process as
>>>>>>>> such:
>>>>>>>>
>>>>>>>> /usr/sbin/glusterfs --volfile-server=localhost
>>>>>>>> --volfile-id=/GlusterWWW /var/www
>>>>>>>>
>>>>>>>> This process begins with little memory, but then as files are
>>>>>>>> accessed in the volume th

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 1/30/2018 6:31 AM, Raghavendra Gowdappa wrote:



- Original Message -

From: "Dan Ragle" <dan...@biblestuph.com>
To: "Raghavendra Gowdappa" <rgowd...@redhat.com>, "Ravishankar N" 
<ravishan...@redhat.com>
Cc: gluster-users@gluster.org, "Csaba Henk" <ch...@redhat.com>, "Niels de Vos" 
<nde...@redhat.com>, "Nithya
Balachandran" <nbala...@redhat.com>
Sent: Monday, January 29, 2018 9:02:21 PM
Subject: Re: [Gluster-users] Run away memory with gluster mount



On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:



- Original Message -

From: "Ravishankar N" <ravishan...@redhat.com>
To: "Dan Ragle" <dan...@biblestuph.com>, gluster-users@gluster.org
Cc: "Csaba Henk" <ch...@redhat.com>, "Niels de Vos" <nde...@redhat.com>,
"Nithya Balachandran" <nbala...@redhat.com>,
"Raghavendra Gowdappa" <rgowd...@redhat.com>
Sent: Saturday, January 27, 2018 10:23:38 AM
Subject: Re: [Gluster-users] Run away memory with gluster mount



On 01/27/2018 02:29 AM, Dan Ragle wrote:


On 1/25/2018 8:21 PM, Ravishankar N wrote:



On 01/25/2018 11:04 PM, Dan Ragle wrote:

*sigh* trying again to correct formatting ... apologize for the
earlier mess.

Having a memory issue with Gluster 3.12.4 and not sure how to
troubleshoot. I don't *think* this is expected behavior.

This is on an updated CentOS 7 box. The setup is a simple two node
replicated layout where the two nodes act as both server and
client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased
cache-size, md invalidation, etc) but stripped them out in an
attempt to
isolate the issue. Still got the problem without them.

The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost
--volfile-id=/GlusterWWW /var/www

This process begins with little memory, but then as files are
accessed in the volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's
been running on and off about 12 hours now and the resident
memory of the above process is already at 7.5G and continues to grow
slowly. If I stop the test script the memory stops growing,
but does not reduce. Restart the test script and the memory begins
slowly growing again.

This is obviously a contrived app environment. With my intended
application load it takes about a week or so for the memory to get
high enough to invoke the oom killer.


Can you try debugging with the statedump
(https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump)
of
the fuse mount process and see what member is leaking? Take the
statedumps in succession, maybe once initially during the I/O and
once the memory gets high enough to hit the OOM mark.
Share the dumps here.

Regards,
Ravi


Thanks for the reply. I noticed yesterday that an update (3.12.5) had
been posted so I went ahead and updated and repeated the test
overnight. The memory usage does not appear to be growing as quickly
as is was with 3.12.4, but does still appear to be growing.

I should also mention that there is another process beyond my test app
that is reading the files from the volume. Specifically, there is an
rsync that runs from the second node 2-4 times an hour that reads from
the GlusterWWW volume mounted on node 1. Since none of the files in
that mount are changing it doesn't actually rsync anything, but
nonetheless it is running and reading the files in addition to my test
script. (It's a part of my intended production setup that I forgot was
still running.)

The mount process appears to be gaining memory at a rate of about 1GB
every 4 hours or so. At that rate it'll take several days before it
runs the box out of memory. But I took your suggestion and made some
statedumps today anyway, about 2 hours apart, 4 total so far. It looks
like there may already be some actionable information. These are the
only registers where the num_allocs have grown with each of the four
samples:

[mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
   ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
   ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
   ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
   ---> num_allocs at Fri Jan 26 14:58:27 2018: 908

[mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
   ---> num_allocs at Fri Jan 26 08:57:

Re: [Gluster-users] Run away memory with gluster mount

[Raghavendra Gowdappa]

- Original Message -
> From: "Dan Ragle" <dan...@biblestuph.com>
> To: "Raghavendra Gowdappa" <rgowd...@redhat.com>, "Ravishankar N" 
> <ravishan...@redhat.com>
> Cc: gluster-users@gluster.org, "Csaba Henk" <ch...@redhat.com>, "Niels de 
> Vos" <nde...@redhat.com>, "Nithya
> Balachandran" <nbala...@redhat.com>
> Sent: Monday, January 29, 2018 9:02:21 PM
> Subject: Re: [Gluster-users] Run away memory with gluster mount
> 
> 
> 
> On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:
> > 
> > 
> > - Original Message -
> >> From: "Ravishankar N" <ravishan...@redhat.com>
> >> To: "Dan Ragle" <dan...@biblestuph.com>, gluster-users@gluster.org
> >> Cc: "Csaba Henk" <ch...@redhat.com>, "Niels de Vos" <nde...@redhat.com>,
> >> "Nithya Balachandran" <nbala...@redhat.com>,
> >> "Raghavendra Gowdappa" <rgowd...@redhat.com>
> >> Sent: Saturday, January 27, 2018 10:23:38 AM
> >> Subject: Re: [Gluster-users] Run away memory with gluster mount
> >>
> >>
> >>
> >> On 01/27/2018 02:29 AM, Dan Ragle wrote:
> >>>
> >>> On 1/25/2018 8:21 PM, Ravishankar N wrote:
> >>>>
> >>>>
> >>>> On 01/25/2018 11:04 PM, Dan Ragle wrote:
> >>>>> *sigh* trying again to correct formatting ... apologize for the
> >>>>> earlier mess.
> >>>>>
> >>>>> Having a memory issue with Gluster 3.12.4 and not sure how to
> >>>>> troubleshoot. I don't *think* this is expected behavior.
> >>>>>
> >>>>> This is on an updated CentOS 7 box. The setup is a simple two node
> >>>>> replicated layout where the two nodes act as both server and
> >>>>> client.
> >>>>>
> >>>>> The volume in question:
> >>>>>
> >>>>> Volume Name: GlusterWWW
> >>>>> Type: Replicate
> >>>>> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
> >>>>> Status: Started
> >>>>> Snapshot Count: 0
> >>>>> Number of Bricks: 1 x 2 = 2
> >>>>> Transport-type: tcp
> >>>>> Bricks:
> >>>>> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
> >>>>> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
> >>>>> Options Reconfigured:
> >>>>> nfs.disable: on
> >>>>> cluster.favorite-child-policy: mtime
> >>>>> transport.address-family: inet
> >>>>>
> >>>>> I had some other performance options in there, (increased
> >>>>> cache-size, md invalidation, etc) but stripped them out in an
> >>>>> attempt to
> >>>>> isolate the issue. Still got the problem without them.
> >>>>>
> >>>>> The volume currently contains over 1M files.
> >>>>>
> >>>>> When mounting the volume, I get (among other things) a process as such:
> >>>>>
> >>>>> /usr/sbin/glusterfs --volfile-server=localhost
> >>>>> --volfile-id=/GlusterWWW /var/www
> >>>>>
> >>>>> This process begins with little memory, but then as files are
> >>>>> accessed in the volume the memory increases. I setup a script that
> >>>>> simply reads the files in the volume one at a time (no writes). It's
> >>>>> been running on and off about 12 hours now and the resident
> >>>>> memory of the above process is already at 7.5G and continues to grow
> >>>>> slowly. If I stop the test script the memory stops growing,
> >>>>> but does not reduce. Restart the test script and the memory begins
> >>>>> slowly growing again.
> >>>>>
> >>>>> This is obviously a contrived app environment. With my intended
> >>>>> application load it takes about a week or so for the memory to get
> >>>>> high enough to invoke the oom killer.
> >>>>
> >>>> Can you try debugging with the statedump
> >>>> (https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump)
> >>>> of
> >>>> the fuse mount process and see what member is leaking? Take the
> >>>> statedum

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 1/26/2018 11:53 PM, Ravishankar N wrote:



On 01/27/2018 02:29 AM, Dan Ragle wrote:


On 1/25/2018 8:21 PM, Ravishankar N wrote:



On 01/25/2018 11:04 PM, Dan Ragle wrote:
*sigh* trying again to correct formatting ... apologize for the 
earlier mess.


Having a memory issue with Gluster 3.12.4 and not sure how to 
troubleshoot. I don't *think* this is expected behavior.


This is on an updated CentOS 7 box. The setup is a simple two node 
replicated layout where the two nodes act as both server and

client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased 
cache-size, md invalidation, etc) but stripped them out in an 
attempt to

isolate the issue. Still got the problem without them.

The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost 
--volfile-id=/GlusterWWW /var/www


This process begins with little memory, but then as files are 
accessed in the volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's 
been running on and off about 12 hours now and the resident
memory of the above process is already at 7.5G and continues to grow 
slowly. If I stop the test script the memory stops growing,
but does not reduce. Restart the test script and the memory begins 
slowly growing again.


This is obviously a contrived app environment. With my intended 
application load it takes about a week or so for the memory to get

high enough to invoke the oom killer.


Can you try debugging with the statedump 
(https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump) 
of
the fuse mount process and see what member is leaking? Take the 
statedumps in succession, maybe once initially during the I/O and

once the memory gets high enough to hit the OOM mark.
Share the dumps here.

Regards,
Ravi


Thanks for the reply. I noticed yesterday that an update (3.12.5) had 
been posted so I went ahead and updated and repeated the test 
overnight. The memory usage does not appear to be growing as quickly 
as is was with 3.12.4, but does still appear to be growing.


I should also mention that there is another process beyond my test app 
that is reading the files from the volume. Specifically, there is an 
rsync that runs from the second node 2-4 times an hour that reads from 
the GlusterWWW volume mounted on node 1. Since none of the files in 
that mount are changing it doesn't actually rsync anything, but 
nonetheless it is running and reading the files in addition to my test 
script. (It's a part of my intended production setup that I forgot was 
still running.)


The mount process appears to be gaining memory at a rate of about 1GB 
every 4 hours or so. At that rate it'll take several days before it 
runs the box out of memory. But I took your suggestion and made some 
statedumps today anyway, about 2 hours apart, 4 total so far. It looks 
like there may already be some actionable information. These are the 
only registers where the num_allocs have grown with each of the four 
samples:


[mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 908

[mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 5
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 10
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 15
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 17

[cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t 
memusage]

 ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036

Not sure the best way to get you the full dumps. They're pretty big, 
over 1G for all four. Also, I noticed some filepath information in 
there that I'd rather not share. What's the recommended next step?


I've CC'd the fuse/ dht devs to see if these data types have potential 
leaks. Could you raise a bug with the volume info and a (dropbox?) link 
from which we can download the dumps? You can remove/replace the 
filepaths from them.


Regards.
Ravi


Filed this bug with links to the tar balled statedumps:

https://bugs.centos.org/view.php?id=14428

Since my testing platform is CentOS I didn't know if 

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 1/29/2018 2:36 AM, Raghavendra Gowdappa wrote:



- Original Message -

From: "Ravishankar N" <ravishan...@redhat.com>
To: "Dan Ragle" <dan...@biblestuph.com>, gluster-users@gluster.org
Cc: "Csaba Henk" <ch...@redhat.com>, "Niels de Vos" <nde...@redhat.com>, "Nithya 
Balachandran" <nbala...@redhat.com>,
"Raghavendra Gowdappa" <rgowd...@redhat.com>
Sent: Saturday, January 27, 2018 10:23:38 AM
Subject: Re: [Gluster-users] Run away memory with gluster mount



On 01/27/2018 02:29 AM, Dan Ragle wrote:


On 1/25/2018 8:21 PM, Ravishankar N wrote:



On 01/25/2018 11:04 PM, Dan Ragle wrote:

*sigh* trying again to correct formatting ... apologize for the
earlier mess.

Having a memory issue with Gluster 3.12.4 and not sure how to
troubleshoot. I don't *think* this is expected behavior.

This is on an updated CentOS 7 box. The setup is a simple two node
replicated layout where the two nodes act as both server and
client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased
cache-size, md invalidation, etc) but stripped them out in an
attempt to
isolate the issue. Still got the problem without them.

The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost
--volfile-id=/GlusterWWW /var/www

This process begins with little memory, but then as files are
accessed in the volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's
been running on and off about 12 hours now and the resident
memory of the above process is already at 7.5G and continues to grow
slowly. If I stop the test script the memory stops growing,
but does not reduce. Restart the test script and the memory begins
slowly growing again.

This is obviously a contrived app environment. With my intended
application load it takes about a week or so for the memory to get
high enough to invoke the oom killer.


Can you try debugging with the statedump
(https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump)
of
the fuse mount process and see what member is leaking? Take the
statedumps in succession, maybe once initially during the I/O and
once the memory gets high enough to hit the OOM mark.
Share the dumps here.

Regards,
Ravi


Thanks for the reply. I noticed yesterday that an update (3.12.5) had
been posted so I went ahead and updated and repeated the test
overnight. The memory usage does not appear to be growing as quickly
as is was with 3.12.4, but does still appear to be growing.

I should also mention that there is another process beyond my test app
that is reading the files from the volume. Specifically, there is an
rsync that runs from the second node 2-4 times an hour that reads from
the GlusterWWW volume mounted on node 1. Since none of the files in
that mount are changing it doesn't actually rsync anything, but
nonetheless it is running and reading the files in addition to my test
script. (It's a part of my intended production setup that I forgot was
still running.)

The mount process appears to be gaining memory at a rate of about 1GB
every 4 hours or so. At that rate it'll take several days before it
runs the box out of memory. But I took your suggestion and made some
statedumps today anyway, about 2 hours apart, 4 total so far. It looks
like there may already be some actionable information. These are the
only registers where the num_allocs have grown with each of the four
samples:

[mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
  ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
  ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
  ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
  ---> num_allocs at Fri Jan 26 14:58:27 2018: 908

[mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
  ---> num_allocs at Fri Jan 26 08:57:31 2018: 5
  ---> num_allocs at Fri Jan 26 10:55:50 2018: 10
  ---> num_allocs at Fri Jan 26 12:55:15 2018: 15
  ---> num_allocs at Fri Jan 26 14:58:27 2018: 17

[cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t
memusage]
  ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596
  ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622
  ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066
  ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036

Not sure the best way to get you the full dumps. They're pretty big,
over 1G for al

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 1/29/2018 12:19 AM, Nithya Balachandran wrote:

Csaba,

Could this be the problem of the inodes not getting freed in the fuse
process?

Daniel,
as Ravi requested, please provide access to the statedumps. You can strip
out the filepath information.


Working on filing a bug report and getting you the dumps now. Will 
update soon.



Does your data set include a lot of directories?


The volume in question has 1M+ files and 77k+ directories.

Cheers!

Dan




Thanks,
Nithya

On 27 January 2018 at 10:23, Ravishankar N  wrote:




On 01/27/2018 02:29 AM, Dan Ragle wrote:



On 1/25/2018 8:21 PM, Ravishankar N wrote:




On 01/25/2018 11:04 PM, Dan Ragle wrote:


*sigh* trying again to correct formatting ... apologize for the earlier
mess.

Having a memory issue with Gluster 3.12.4 and not sure how to
troubleshoot. I don't *think* this is expected behavior.

This is on an updated CentOS 7 box. The setup is a simple two node
replicated layout where the two nodes act as both server and
client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased cache-size,
md invalidation, etc) but stripped them out in an attempt to
isolate the issue. Still got the problem without them.

The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW
/var/www

This process begins with little memory, but then as files are accessed
in the volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's
been running on and off about 12 hours now and the resident
memory of the above process is already at 7.5G and continues to grow
slowly. If I stop the test script the memory stops growing,
but does not reduce. Restart the test script and the memory begins
slowly growing again.

This is obviously a contrived app environment. With my intended
application load it takes about a week or so for the memory to get
high enough to invoke the oom killer.



Can you try debugging with the statedump (https://gluster.readthedocs.i
o/en/latest/Troubleshooting/statedump/#read-a-statedump) of
the fuse mount process and see what member is leaking? Take the
statedumps in succession, maybe once initially during the I/O and
once the memory gets high enough to hit the OOM mark.
Share the dumps here.

Regards,
Ravi



Thanks for the reply. I noticed yesterday that an update (3.12.5) had
been posted so I went ahead and updated and repeated the test overnight.
The memory usage does not appear to be growing as quickly as is was with
3.12.4, but does still appear to be growing.

I should also mention that there is another process beyond my test app
that is reading the files from the volume. Specifically, there is an rsync
that runs from the second node 2-4 times an hour that reads from the
GlusterWWW volume mounted on node 1. Since none of the files in that mount
are changing it doesn't actually rsync anything, but nonetheless it is
running and reading the files in addition to my test script. (It's a part
of my intended production setup that I forgot was still running.)

The mount process appears to be gaining memory at a rate of about 1GB
every 4 hours or so. At that rate it'll take several days before it runs
the box out of memory. But I took your suggestion and made some statedumps
today anyway, about 2 hours apart, 4 total so far. It looks like there may
already be some actionable information. These are the only registers where
the num_allocs have grown with each of the four samples:

[mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
  ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
  ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
  ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
  ---> num_allocs at Fri Jan 26 14:58:27 2018: 908

[mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
  ---> num_allocs at Fri Jan 26 08:57:31 2018: 5
  ---> num_allocs at Fri Jan 26 10:55:50 2018: 10
  ---> num_allocs at Fri Jan 26 12:55:15 2018: 15
  ---> num_allocs at Fri Jan 26 14:58:27 2018: 17

[cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t
memusage]
  ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596
  ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622
  ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066
  ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036

Not sure the best way to get you the full dumps. They're pretty big, over
1G for all four. Also, I noticed some filepath 

Re: [Gluster-users] Run away memory with gluster mount

[Raghavendra Gowdappa]

- Original Message -
> From: "Nithya Balachandran" <nbala...@redhat.com>
> To: "Ravishankar N" <ravishan...@redhat.com>
> Cc: "Csaba Henk" <ch...@redhat.com>, "gluster-users" 
> <gluster-users@gluster.org>
> Sent: Monday, January 29, 2018 10:49:43 AM
> Subject: Re: [Gluster-users] Run away memory with gluster mount
> 
> Csaba,
> 
> Could this be the problem of the inodes not getting freed in the fuse
> process?

We can answer that question only after looking into statedumps. If we find too 
many inodes in fuse inode table's lru list (with refcount 0, lookup count > 0), 
it could be because sub-optimal garbage collection of inodes.

> 
> Daniel,
> as Ravi requested, please provide access to the statedumps. You can strip out
> the filepath information.
> Does your data set include a lot of directories?
> 
> 
> Thanks,
> Nithya
> 
> On 27 January 2018 at 10:23, Ravishankar N < ravishan...@redhat.com > wrote:
> 
> 
> 
> 
> 
> On 01/27/2018 02:29 AM, Dan Ragle wrote:
> 
> 
> 
> On 1/25/2018 8:21 PM, Ravishankar N wrote:
> 
> 
> 
> 
> On 01/25/2018 11:04 PM, Dan Ragle wrote:
> 
> 
> *sigh* trying again to correct formatting ... apologize for the earlier mess.
> 
> Having a memory issue with Gluster 3.12.4 and not sure how to troubleshoot. I
> don't *think* this is expected behavior.
> 
> This is on an updated CentOS 7 box. The setup is a simple two node replicated
> layout where the two nodes act as both server and
> client.
> 
> The volume in question:
> 
> Volume Name: GlusterWWW
> Type: Replicate
> Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
> Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
> Options Reconfigured:
> nfs.disable: on
> cluster.favorite-child-policy: mtime
> transport.address-family: inet
> 
> I had some other performance options in there, (increased cache-size, md
> invalidation, etc) but stripped them out in an attempt to
> isolate the issue. Still got the problem without them.
> 
> The volume currently contains over 1M files.
> 
> When mounting the volume, I get (among other things) a process as such:
> 
> /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW
> /var/www
> 
> This process begins with little memory, but then as files are accessed in the
> volume the memory increases. I setup a script that
> simply reads the files in the volume one at a time (no writes). It's been
> running on and off about 12 hours now and the resident
> memory of the above process is already at 7.5G and continues to grow slowly.
> If I stop the test script the memory stops growing,
> but does not reduce. Restart the test script and the memory begins slowly
> growing again.
> 
> This is obviously a contrived app environment. With my intended application
> load it takes about a week or so for the memory to get
> high enough to invoke the oom killer.
> 
> Can you try debugging with the statedump (
> https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump
> ) of
> the fuse mount process and see what member is leaking? Take the statedumps in
> succession, maybe once initially during the I/O and
> once the memory gets high enough to hit the OOM mark.
> Share the dumps here.
> 
> Regards,
> Ravi
> 
> Thanks for the reply. I noticed yesterday that an update (3.12.5) had been
> posted so I went ahead and updated and repeated the test overnight. The
> memory usage does not appear to be growing as quickly as is was with 3.12.4,
> but does still appear to be growing.
> 
> I should also mention that there is another process beyond my test app that
> is reading the files from the volume. Specifically, there is an rsync that
> runs from the second node 2-4 times an hour that reads from the GlusterWWW
> volume mounted on node 1. Since none of the files in that mount are changing
> it doesn't actually rsync anything, but nonetheless it is running and
> reading the files in addition to my test script. (It's a part of my intended
> production setup that I forgot was still running.)
> 
> The mount process appears to be gaining memory at a rate of about 1GB every 4
> hours or so. At that rate it'll take several days before it runs the box out
> of memory. But I took your suggestion and made some statedumps today anyway,
> about 2 hours apart, 4 total so far. It looks like there may already be some
> actionable information. These are the only registers where the num_allocs
&g

Re: [Gluster-users] Run away memory with gluster mount

[Nithya Balachandran]

Csaba,

Could this be the problem of the inodes not getting freed in the fuse
process?

Daniel,
as Ravi requested, please provide access to the statedumps. You can strip
out the filepath information.
Does your data set include a lot of directories?


Thanks,
Nithya

On 27 January 2018 at 10:23, Ravishankar N  wrote:

>
>
> On 01/27/2018 02:29 AM, Dan Ragle wrote:
>
>>
>> On 1/25/2018 8:21 PM, Ravishankar N wrote:
>>
>>>
>>>
>>> On 01/25/2018 11:04 PM, Dan Ragle wrote:
>>>
 *sigh* trying again to correct formatting ... apologize for the earlier
 mess.

 Having a memory issue with Gluster 3.12.4 and not sure how to
 troubleshoot. I don't *think* this is expected behavior.

 This is on an updated CentOS 7 box. The setup is a simple two node
 replicated layout where the two nodes act as both server and
 client.

 The volume in question:

 Volume Name: GlusterWWW
 Type: Replicate
 Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
 Status: Started
 Snapshot Count: 0
 Number of Bricks: 1 x 2 = 2
 Transport-type: tcp
 Bricks:
 Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
 Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
 Options Reconfigured:
 nfs.disable: on
 cluster.favorite-child-policy: mtime
 transport.address-family: inet

 I had some other performance options in there, (increased cache-size,
 md invalidation, etc) but stripped them out in an attempt to
 isolate the issue. Still got the problem without them.

 The volume currently contains over 1M files.

 When mounting the volume, I get (among other things) a process as such:

 /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW
 /var/www

 This process begins with little memory, but then as files are accessed
 in the volume the memory increases. I setup a script that
 simply reads the files in the volume one at a time (no writes). It's
 been running on and off about 12 hours now and the resident
 memory of the above process is already at 7.5G and continues to grow
 slowly. If I stop the test script the memory stops growing,
 but does not reduce. Restart the test script and the memory begins
 slowly growing again.

 This is obviously a contrived app environment. With my intended
 application load it takes about a week or so for the memory to get
 high enough to invoke the oom killer.

>>>
>>> Can you try debugging with the statedump (https://gluster.readthedocs.i
>>> o/en/latest/Troubleshooting/statedump/#read-a-statedump) of
>>> the fuse mount process and see what member is leaking? Take the
>>> statedumps in succession, maybe once initially during the I/O and
>>> once the memory gets high enough to hit the OOM mark.
>>> Share the dumps here.
>>>
>>> Regards,
>>> Ravi
>>>
>>
>> Thanks for the reply. I noticed yesterday that an update (3.12.5) had
>> been posted so I went ahead and updated and repeated the test overnight.
>> The memory usage does not appear to be growing as quickly as is was with
>> 3.12.4, but does still appear to be growing.
>>
>> I should also mention that there is another process beyond my test app
>> that is reading the files from the volume. Specifically, there is an rsync
>> that runs from the second node 2-4 times an hour that reads from the
>> GlusterWWW volume mounted on node 1. Since none of the files in that mount
>> are changing it doesn't actually rsync anything, but nonetheless it is
>> running and reading the files in addition to my test script. (It's a part
>> of my intended production setup that I forgot was still running.)
>>
>> The mount process appears to be gaining memory at a rate of about 1GB
>> every 4 hours or so. At that rate it'll take several days before it runs
>> the box out of memory. But I took your suggestion and made some statedumps
>> today anyway, about 2 hours apart, 4 total so far. It looks like there may
>> already be some actionable information. These are the only registers where
>> the num_allocs have grown with each of the four samples:
>>
>> [mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
>>  ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
>>  ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
>>  ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
>>  ---> num_allocs at Fri Jan 26 14:58:27 2018: 908
>>
>> [mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
>>  ---> num_allocs at Fri Jan 26 08:57:31 2018: 5
>>  ---> num_allocs at Fri Jan 26 10:55:50 2018: 10
>>  ---> num_allocs at Fri Jan 26 12:55:15 2018: 15
>>  ---> num_allocs at Fri Jan 26 14:58:27 2018: 17
>>
>> [cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t
>> memusage]
>>  ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596
>>  ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622
>>  ---> num_allocs at Fri Jan 26 12:55:15 2018: 

Re: [Gluster-users] Run away memory with gluster mount

[Ravishankar N]

On 01/27/2018 02:29 AM, Dan Ragle wrote:


On 1/25/2018 8:21 PM, Ravishankar N wrote:



On 01/25/2018 11:04 PM, Dan Ragle wrote:
*sigh* trying again to correct formatting ... apologize for the 
earlier mess.


Having a memory issue with Gluster 3.12.4 and not sure how to 
troubleshoot. I don't *think* this is expected behavior.


This is on an updated CentOS 7 box. The setup is a simple two node 
replicated layout where the two nodes act as both server and

client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased 
cache-size, md invalidation, etc) but stripped them out in an 
attempt to

isolate the issue. Still got the problem without them.

The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost 
--volfile-id=/GlusterWWW /var/www


This process begins with little memory, but then as files are 
accessed in the volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's 
been running on and off about 12 hours now and the resident
memory of the above process is already at 7.5G and continues to grow 
slowly. If I stop the test script the memory stops growing,
but does not reduce. Restart the test script and the memory begins 
slowly growing again.


This is obviously a contrived app environment. With my intended 
application load it takes about a week or so for the memory to get

high enough to invoke the oom killer.


Can you try debugging with the statedump 
(https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump) 
of
the fuse mount process and see what member is leaking? Take the 
statedumps in succession, maybe once initially during the I/O and

once the memory gets high enough to hit the OOM mark.
Share the dumps here.

Regards,
Ravi


Thanks for the reply. I noticed yesterday that an update (3.12.5) had 
been posted so I went ahead and updated and repeated the test 
overnight. The memory usage does not appear to be growing as quickly 
as is was with 3.12.4, but does still appear to be growing.


I should also mention that there is another process beyond my test app 
that is reading the files from the volume. Specifically, there is an 
rsync that runs from the second node 2-4 times an hour that reads from 
the GlusterWWW volume mounted on node 1. Since none of the files in 
that mount are changing it doesn't actually rsync anything, but 
nonetheless it is running and reading the files in addition to my test 
script. (It's a part of my intended production setup that I forgot was 
still running.)


The mount process appears to be gaining memory at a rate of about 1GB 
every 4 hours or so. At that rate it'll take several days before it 
runs the box out of memory. But I took your suggestion and made some 
statedumps today anyway, about 2 hours apart, 4 total so far. It looks 
like there may already be some actionable information. These are the 
only registers where the num_allocs have grown with each of the four 
samples:


[mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 908

[mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 5
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 10
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 15
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 17

[cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t 
memusage]

 ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036

Not sure the best way to get you the full dumps. They're pretty big, 
over 1G for all four. Also, I noticed some filepath information in 
there that I'd rather not share. What's the recommended next step?


I've CC'd the fuse/ dht devs to see if these data types have potential 
leaks. Could you raise a bug with the volume info and a (dropbox?) link 
from which we can download the dumps? You can remove/replace the 
filepaths from them.


Regards.
Ravi



Cheers!

Dan



Is there potentially something misconfigured here?

I did see a reference to a memory leak in another thread in this 
list, but that had to do with the setting of quotas, I don't have

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

On 1/25/2018 8:21 PM, Ravishankar N wrote:



On 01/25/2018 11:04 PM, Dan Ragle wrote:

*sigh* trying again to correct formatting ... apologize for the earlier mess.

Having a memory issue with Gluster 3.12.4 and not sure how to troubleshoot. I 
don't *think* this is expected behavior.

This is on an updated CentOS 7 box. The setup is a simple two node replicated 
layout where the two nodes act as both server and
client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased cache-size, md 
invalidation, etc) but stripped them out in an attempt to
isolate the issue. Still got the problem without them.

The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW /var/www

This process begins with little memory, but then as files are accessed in the 
volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's been 
running on and off about 12 hours now and the resident
memory of the above process is already at 7.5G and continues to grow slowly. If 
I stop the test script the memory stops growing,
but does not reduce. Restart the test script and the memory begins slowly 
growing again.

This is obviously a contrived app environment. With my intended application 
load it takes about a week or so for the memory to get
high enough to invoke the oom killer.


Can you try debugging with the statedump 
(https://gluster.readthedocs.io/en/latest/Troubleshooting/statedump/#read-a-statedump)
 of
the fuse mount process and see what member is leaking? Take the statedumps in 
succession, maybe once initially during the I/O and
once the memory gets high enough to hit the OOM mark.
Share the dumps here.

Regards,
Ravi


Thanks for the reply. I noticed yesterday that an update (3.12.5) had been posted so I went ahead and updated and repeated the test 
overnight. The memory usage does not appear to be growing as quickly as is was with 3.12.4, but does still appear to be growing.


I should also mention that there is another process beyond my test app that is reading the files from the volume. Specifically, 
there is an rsync that runs from the second node 2-4 times an hour that reads from the GlusterWWW volume mounted on node 1. Since 
none of the files in that mount are changing it doesn't actually rsync anything, but nonetheless it is running and reading the files 
in addition to my test script. (It's a part of my intended production setup that I forgot was still running.)


The mount process appears to be gaining memory at a rate of about 1GB every 4 hours or so. At that rate it'll take several days 
before it runs the box out of memory. But I took your suggestion and made some statedumps today anyway, about 2 hours apart, 4 total 
so far. It looks like there may already be some actionable information. These are the only registers where the num_allocs have grown 
with each of the four samples:


[mount/fuse.fuse - usage-type gf_fuse_mt_gids_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 784
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 831
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 877
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 908

[mount/fuse.fuse - usage-type gf_common_mt_fd_lk_ctx_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 5
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 10
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 15
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 17

[cluster/distribute.GlusterWWW-dht - usage-type gf_dht_mt_dht_layout_t memusage]
 ---> num_allocs at Fri Jan 26 08:57:31 2018: 24243596
 ---> num_allocs at Fri Jan 26 10:55:50 2018: 27902622
 ---> num_allocs at Fri Jan 26 12:55:15 2018: 30678066
 ---> num_allocs at Fri Jan 26 14:58:27 2018: 33801036

Not sure the best way to get you the full dumps. They're pretty big, over 1G for all four. Also, I noticed some filepath information 
in there that I'd rather not share. What's the recommended next step?


Cheers!

Dan



Is there potentially something misconfigured here?

I did see a reference to a memory leak in another thread in this list, but that 
had to do with the setting of quotas, I don't have
any quotas set on my system.

Thanks,

Dan Ragle
dan...@biblestuph.com

On 1/25/2018 11:04 AM, Dan Ragle wrote:

Having a memory issue with Gluster 3.12.4 and not sure how to
troubleshoot. I don't *think* this is expected behavior. This is on an
updated CentOS 7 box. The setup is a simple two node replicated layout

Re: [Gluster-users] Run away memory with gluster mount

[Dan Ragle]

*sigh* trying again to correct formatting ... apologize for the earlier mess.

Having a memory issue with Gluster 3.12.4 and not sure how to troubleshoot. I 
don't *think* this is expected behavior.

This is on an updated CentOS 7 box. The setup is a simple two node replicated 
layout where the two nodes act as both server and client.

The volume in question:

Volume Name: GlusterWWW
Type: Replicate
Volume ID: 8e9b0e79-f309-4d9b-a5bb-45d065f3
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www
Brick2: vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www
Options Reconfigured:
nfs.disable: on
cluster.favorite-child-policy: mtime
transport.address-family: inet

I had some other performance options in there, (increased cache-size, md invalidation, etc) but stripped them out in an attempt to 
isolate the issue. Still got the problem without them.


The volume currently contains over 1M files.

When mounting the volume, I get (among other things) a process as such:

/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW /var/www

This process begins with little memory, but then as files are accessed in the volume the memory increases. I setup a script that 
simply reads the files in the volume one at a time (no writes). It's been running on and off about 12 hours now and the resident 
memory of the above process is already at 7.5G and continues to grow slowly. If I stop the test script the memory stops growing, but 
does not reduce. Restart the test script and the memory begins slowly growing again.


This is obviously a contrived app environment. With my intended application load it takes about a week or so for the memory to get 
high enough to invoke the oom killer.


Is there potentially something misconfigured here?

I did see a reference to a memory leak in another thread in this list, but that had to do with the setting of quotas, I don't have 
any quotas set on my system.


Thanks,

Dan Ragle
dan...@biblestuph.com

On 1/25/2018 11:04 AM, Dan Ragle wrote:

Having a memory issue with Gluster 3.12.4 and not sure how to
troubleshoot. I don't *think* this is expected behavior. This is on an
updated CentOS 7 box. The setup is a simple two node replicated layout
where the two nodes act as both server and client. The volume in
question: Volume Name: GlusterWWW Type: Replicate Volume ID:
8e9b0e79-f309-4d9b-a5bb-45d065f3 Status: Started Snapshot Count: 0
Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1:
vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2:
vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options Reconfigured:
nfs.disable: on cluster.favorite-child-policy: mtime
transport.address-family: inet I had some other performance options in
there, (increased cache-size, md invalidation, etc) but stripped them
out in an attempt to isolate the issue. Still got the problem without
them. The volume currently contains over 1M files. When mounting the
volume, I get (among other things) a process as such:
/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW
/var/www This process begins with little memory, but then as files are
accessed in the volume the memory increases. I setup a script that
simply reads the files in the volume one at a time (no writes). It's
been running on and off about 12 hours now and the resident memory of
the above process is already at 7.5G and continues to grow slowly. If I
stop the test script the memory stops growing, but does not reduce.
Restart the test script and the memory begins slowly growing again. This
is obviously a contrived app environment. With my intended application
load it takes about a week or so for the memory to get high enough to
invoke the oom killer. Is there potentially something misconfigured
here? Thanks, Dan Ragle dan...@biblestuph.com




___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users

[Gluster-users] Run away memory with gluster mount

[Dan Ragle]

Having a memory issue with Gluster 3.12.4 and not sure how to 
troubleshoot. I don't *think* this is expected behavior. This is on an 
updated CentOS 7 box. The setup is a simple two node replicated layout 
where the two nodes act as both server and client. The volume in 
question: Volume Name: GlusterWWW Type: Replicate Volume ID: 
8e9b0e79-f309-4d9b-a5bb-45d065f3 Status: Started Snapshot Count: 0 
Number of Bricks: 1 x 2 = 2 Transport-type: tcp Bricks: Brick1: 
vs1dlan.mydomain.com:/glusterfs_bricks/brick1/www Brick2: 
vs2dlan.mydomain.com:/glusterfs_bricks/brick1/www Options Reconfigured: 
nfs.disable: on cluster.favorite-child-policy: mtime 
transport.address-family: inet I had some other performance options in 
there, (increased cache-size, md invalidation, etc) but stripped them 
out in an attempt to isolate the issue. Still got the problem without 
them. The volume currently contains over 1M files. When mounting the 
volume, I get (among other things) a process as such: 
/usr/sbin/glusterfs --volfile-server=localhost --volfile-id=/GlusterWWW 
/var/www This process begins with little memory, but then as files are 
accessed in the volume the memory increases. I setup a script that 
simply reads the files in the volume one at a time (no writes). It's 
been running on and off about 12 hours now and the resident memory of 
the above process is already at 7.5G and continues to grow slowly. If I 
stop the test script the memory stops growing, but does not reduce. 
Restart the test script and the memory begins slowly growing again. This 
is obviously a contrived app environment. With my intended application 
load it takes about a week or so for the memory to get high enough to 
invoke the oom killer. Is there potentially something misconfigured 
here? Thanks, Dan Ragle dan...@biblestuph.com


___
Gluster-users mailing list
Gluster-users@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users