Re: httpd memory consumption

[William A Rowe Jr]

Have you tried bisecting the config directives to see which is triggering
the memory abuse?

Sounds like the module might not be async-ready, but should httpd really be
doing many thread swaps before the listener thread is tripped?

Does one of your modules load a large table al la Geo IP mapping?

Are you instantiating a scripting runtime, eg PHP or mod_perl?


On Oct 6, 2017 08:04, "Ruediger Pluem"  wrote:

>
>
> On 10/06/2017 10:26 AM, Bert Huijben wrote:
> >
> >
> >> -Original Message-
> >> From: Ruediger Pluem [mailto:rpl...@apache.org]
> >> Sent: vrijdag 6 oktober 2017 09:47
> >> To: Apache HTTP Server Development List 
> >> Subject: httpd memory consumption
> >>
> >> I am currently looking at a core of a httpd 2.4 process using the event
> MPM
> >> that consumed a lot of memory (core dump file size about 1.4 GB).
> >> While taking the core actually no request was processed by this process.
> >> I suspect a memory leak in a 3rd party closed source module.
> >> As I have no view in the 3rd party module I tried to find out the heap
> >> memory usage of httpd and the included modules.
> >>
> >> For this I used the new dump_pool_and_children I added to .gdbinit which
> >> delivers me the memory used by all pools below 'apr_global_pool' and the
> >> amount of memory in the allocator free lists associated to these pools.
> >> This delivered a usage of only a few MB. Of course I am aware that the
> >> process consumes additional memory for stack, static data and text
> >> segments, but this usage should be static.
> >>
> >> Is there any heap usage by httpd that I could have missed?
> >>
> >> - We do not use the unmanaged pools from APR. So I should have caught
> all
> >> pools.
> >> - I do no think that we use allocators that are not used by a pool.
> Hence I
> >> should have caught all allocators and
> >>   its free lists.
> >> - As no request was processed when I captured the core (all worker
> threads
> >> were waiting for work) I doubt that missed
> >>   any or at least large memory consuming buckets.
> >
> > Is APR configured to return memory to the OS? Otherwise you might just
> see all the allocated memory in the free list of the pool allocator(s).
> >
>
> MaxMemFree is set to 2 MB in httpd and the allocator free list's that I
> have checked do not contain much memory (around
> 400 KB in total for all of them).
>
> Regards
>
> Rüdiger
>
>
>

Re: httpd memory consumption

[Ruediger Pluem]

On 10/06/2017 10:26 AM, Bert Huijben wrote:
> 
> 
>> -Original Message-
>> From: Ruediger Pluem [mailto:rpl...@apache.org]
>> Sent: vrijdag 6 oktober 2017 09:47
>> To: Apache HTTP Server Development List 
>> Subject: httpd memory consumption
>>
>> I am currently looking at a core of a httpd 2.4 process using the event MPM
>> that consumed a lot of memory (core dump file size about 1.4 GB).
>> While taking the core actually no request was processed by this process.
>> I suspect a memory leak in a 3rd party closed source module.
>> As I have no view in the 3rd party module I tried to find out the heap
>> memory usage of httpd and the included modules.
>>
>> For this I used the new dump_pool_and_children I added to .gdbinit which
>> delivers me the memory used by all pools below 'apr_global_pool' and the
>> amount of memory in the allocator free lists associated to these pools.
>> This delivered a usage of only a few MB. Of course I am aware that the
>> process consumes additional memory for stack, static data and text
>> segments, but this usage should be static.
>>
>> Is there any heap usage by httpd that I could have missed?
>>
>> - We do not use the unmanaged pools from APR. So I should have caught all
>> pools.
>> - I do no think that we use allocators that are not used by a pool. Hence I
>> should have caught all allocators and
>>   its free lists.
>> - As no request was processed when I captured the core (all worker threads
>> were waiting for work) I doubt that missed
>>   any or at least large memory consuming buckets.
> 
> Is APR configured to return memory to the OS? Otherwise you might just see 
> all the allocated memory in the free list of the pool allocator(s).
> 

MaxMemFree is set to 2 MB in httpd and the allocator free list's that I have 
checked do not contain much memory (around
400 KB in total for all of them).

Regards

Rüdiger

Re: httpd memory consumption

[Ruediger Pluem]

On 10/06/2017 10:12 AM, Joe Orton wrote:
> On Fri, Oct 06, 2017 at 09:46:39AM +0200, Ruediger Pluem wrote:
>> For this I used the new dump_pool_and_children I added to .gdbinit which 
>> delivers me the memory used by all
>> pools below 'apr_global_pool' and the amount of memory in the allocator free 
>> lists associated to these pools.
>> This delivered a usage of only a few MB. Of course I am aware that the 
>> process consumes additional memory for stack,
>> static data and text segments, but this usage should be static.
> 
> Very nice!
> 
>>
>> Is there any heap usage by httpd that I could have missed?
> 
> OpenSSL?

Good pointer. Possibly not the case here as the same openssl version is used 
with other similar setups, but I have no
hard prove for that assumption. Any pointers how to gather the memory usage of 
openssl from the core dump?

Regards

Rüdiger

RE: httpd memory consumption

[Bert Huijben]

> -Original Message-
> From: Ruediger Pluem [mailto:rpl...@apache.org]
> Sent: vrijdag 6 oktober 2017 09:47
> To: Apache HTTP Server Development List 
> Subject: httpd memory consumption
> 
> I am currently looking at a core of a httpd 2.4 process using the event MPM
> that consumed a lot of memory (core dump file size about 1.4 GB).
> While taking the core actually no request was processed by this process.
> I suspect a memory leak in a 3rd party closed source module.
> As I have no view in the 3rd party module I tried to find out the heap
> memory usage of httpd and the included modules.
> 
> For this I used the new dump_pool_and_children I added to .gdbinit which
> delivers me the memory used by all pools below 'apr_global_pool' and the
> amount of memory in the allocator free lists associated to these pools.
> This delivered a usage of only a few MB. Of course I am aware that the
> process consumes additional memory for stack, static data and text
> segments, but this usage should be static.
> 
> Is there any heap usage by httpd that I could have missed?
> 
> - We do not use the unmanaged pools from APR. So I should have caught all
> pools.
> - I do no think that we use allocators that are not used by a pool. Hence I
> should have caught all allocators and
>   its free lists.
> - As no request was processed when I captured the core (all worker threads
> were waiting for work) I doubt that missed
>   any or at least large memory consuming buckets.

Is APR configured to return memory to the OS? Otherwise you might just see all 
the allocated memory in the free list of the pool allocator(s).

Bert

Re: httpd memory consumption

[Joe Orton]

On Fri, Oct 06, 2017 at 09:46:39AM +0200, Ruediger Pluem wrote:
> For this I used the new dump_pool_and_children I added to .gdbinit which 
> delivers me the memory used by all
> pools below 'apr_global_pool' and the amount of memory in the allocator free 
> lists associated to these pools.
> This delivered a usage of only a few MB. Of course I am aware that the 
> process consumes additional memory for stack,
> static data and text segments, but this usage should be static.

Very nice!

> 
> Is there any heap usage by httpd that I could have missed?

OpenSSL?

Re: httpd memory consumption

[Ruediger Pluem]

On 10/06/2017 09:56 AM, Stefan Eissing wrote:
> Were many threads spawned? Asking for a friend...

17 by httpd and 12 by the 3rd party module.

Additional information: no mod_http2 loaded :-)

Regards

Rüdiger

> 
>> Am 06.10.2017 um 09:46 schrieb Ruediger Pluem :
>>
>> I am currently looking at a core of a httpd 2.4 process using the event MPM 
>> that consumed a lot of memory
>> (core dump file size about 1.4 GB).
>> While taking the core actually no request was processed by this process.
>> I suspect a memory leak in a 3rd party closed source module.
>> As I have no view in the 3rd party module I tried to find out the heap 
>> memory usage of httpd and the included modules.
>>
>> For this I used the new dump_pool_and_children I added to .gdbinit which 
>> delivers me the memory used by all
>> pools below 'apr_global_pool' and the amount of memory in the allocator free 
>> lists associated to these pools.
>> This delivered a usage of only a few MB. Of course I am aware that the 
>> process consumes additional memory for stack,
>> static data and text segments, but this usage should be static.
>>
>> Is there any heap usage by httpd that I could have missed?
>>
>> - We do not use the unmanaged pools from APR. So I should have caught all 
>> pools.
>> - I do no think that we use allocators that are not used by a pool. Hence I 
>> should have caught all allocators and
>>  its free lists.
>> - As no request was processed when I captured the core (all worker threads 
>> were waiting for work) I doubt that missed
>>  any or at least large memory consuming buckets.
>>
>> So possible other consumers of memory apart from the 3rd party module that 
>> come to my mind:
>>
>> - Could any of the glibc (httpd was running on a Linux system) functions 
>> wrapped by APR consume a lot of memory?
>> - As far as I researched glibc by default should return memory to the OS if 
>> more then 128 KB are free at the top of
>>  the heap.
>>
>> Regards
>>
>> Rüdiger
> 
> 

Re: httpd memory consumption

[Stefan Eissing]

Were many threads spawned? Asking for a friend...

> Am 06.10.2017 um 09:46 schrieb Ruediger Pluem :
> 
> I am currently looking at a core of a httpd 2.4 process using the event MPM 
> that consumed a lot of memory
> (core dump file size about 1.4 GB).
> While taking the core actually no request was processed by this process.
> I suspect a memory leak in a 3rd party closed source module.
> As I have no view in the 3rd party module I tried to find out the heap memory 
> usage of httpd and the included modules.
> 
> For this I used the new dump_pool_and_children I added to .gdbinit which 
> delivers me the memory used by all
> pools below 'apr_global_pool' and the amount of memory in the allocator free 
> lists associated to these pools.
> This delivered a usage of only a few MB. Of course I am aware that the 
> process consumes additional memory for stack,
> static data and text segments, but this usage should be static.
> 
> Is there any heap usage by httpd that I could have missed?
> 
> - We do not use the unmanaged pools from APR. So I should have caught all 
> pools.
> - I do no think that we use allocators that are not used by a pool. Hence I 
> should have caught all allocators and
>  its free lists.
> - As no request was processed when I captured the core (all worker threads 
> were waiting for work) I doubt that missed
>  any or at least large memory consuming buckets.
> 
> So possible other consumers of memory apart from the 3rd party module that 
> come to my mind:
> 
> - Could any of the glibc (httpd was running on a Linux system) functions 
> wrapped by APR consume a lot of memory?
> - As far as I researched glibc by default should return memory to the OS if 
> more then 128 KB are free at the top of
>  the heap.
> 
> Regards
> 
> Rüdiger