Re: Apache growing (memory)
On 25 Apr 2001, Joe Schaefer wrote: > Kurt George Gjerde <[EMAIL PROTECTED]> writes: Hi, > Even if your script were coded perfectly, it is still possible for this > to happen in modperl. > Personally, I would consider an average growth rate of only .5kB/hit > absolutely wonderful :) As far I ever monitored memory usage, apache itself seems to leak some memory itself as well... This was when using apache 1.3.9 (debian/potato), and took up to a couple hunderds of bytes every requests (thus growing a mem-page (4k) every dozen or so requests per process) serving just static files. It is very hard to write perfect code, and some work arounds or often the best solution... You can consider limiting MaxRequestPerChild in combination with Apache::GTopLimit, or whatever limits the apache-docs, mod_perl guide or CPAN modules propose. Bye, Remco
Re: Apache growing (memory)
Hi all,
On Wed, 25 Apr 2001, Remco Schaar wrote:
> It is very hard to write perfect code,
True, but it's not hard to write code that doesn't leak memory.
void *p = NULL;
...
...
if( p ) { exit(POINTER_ERROR); }
void *p = malloc(n);
...
...
free( p );
p = NULL;
...
...
By which I mean that I use a certain set of pointers for memory
allocation and nothing else. The pointers are declared and
initialized to NULL in the same statement. Whenever I free some
memory I set the pointer which was used to point to it back to NULL.
Whenever I use a pointer for memory I first check that its value is
NULL. Then if I try to use it when it's already being used for
something else my program tells me in a fairly unceremonious fashion.
This is one of a number of defensive techniques I've developed over
the years. It's better to wrap this kind of stuff in a library of
defensive memory management routines (I expect you can buy one:) as
there's less room for errors and it saves a lot of typing.
Just my $0.02.
Geoff, I expect this will turn out to be one of those threads...
73,
Ged.
Re: [OT] writing code (was: Apache growing (memory))
On Wed, 25 Apr 2001, G.W. Haywood wrote:
> Hi all,
Hi again,
> On Wed, 25 Apr 2001, Remco Schaar wrote:
>
> > It is very hard to write perfect code,
>
> True, but it's not hard to write code that doesn't leak memory.
>
> void *p = NULL;
> ...
> ...
> if( p ) { exit(POINTER_ERROR); }
> void *p = malloc(n);
> ...
> ...
> free( p );
> p = NULL;
> ...
> ...
Of course I did not refer to "hello world", but more complex program that
generate new code which is executed etc... I was not even thinking about
starting a discussion or flame war about code "correctness" :-)
> By which I mean that I use a certain set of pointers for memory
> allocation and nothing else. The pointers are declared and
> initialized to NULL in the same statement. Whenever I free some
> memory I set the pointer which was used to point to it back to NULL.
> Whenever I use a pointer for memory I first check that its value is
> NULL. Then if I try to use it when it's already being used for
> something else my program tells me in a fairly unceremonious fashion.
Ok, but this requires all programmers to write clean and _consequent_
code, with all proper checks and a well-thought design
and... and... and... Theory is great, but in practice, this seems to
happen only with small groups/projects and then even rarely (all
software-enginering literature will confirm this, sadly). Certainly with
continuous running programs (as apache/mod_perl), even the smallest
mistake or misplacement in freeing memory (or not-freeing on purpose) will
cause growth of memory consumption per request.
> This is one of a number of defensive techniques I've developed over
> the years. It's better to wrap this kind of stuff in a library of
> defensive memory management routines (I expect you can buy one:) as
> there's less room for errors and it saves a lot of typing.
>
> Just my $0.02.
Go, and tell those guys in redmond... please :-)))
> Geoff, I expect this will turn out to be one of those threads...
No intensions !-)
> 73,
> Ged.
Bye,
Remco
Re: [OT] Apache growing (memory)
Hi Matt, On Wed, 25 Apr 2001, Matt Sergeant wrote: > Easier still, just use boehm gc. ;-) Can you get that for MS-DOS? 73, Ged.
Re: Apache growing (memory)
On Wed, 25 Apr 2001, G.W. Haywood wrote:
> Hi all,
>
> On Wed, 25 Apr 2001, Remco Schaar wrote:
>
> > It is very hard to write perfect code,
>
> True, but it's not hard to write code that doesn't leak memory.
>
> void *p = NULL;
> ...
> ...
> if( p ) { exit(POINTER_ERROR); }
> void *p = malloc(n);
> ...
> ...
> free( p );
> p = NULL;
> ...
> ...
>
> By which I mean that I use a certain set of pointers for memory
> allocation and nothing else. The pointers are declared and
> initialized to NULL in the same statement. Whenever I free some
> memory I set the pointer which was used to point to it back to NULL.
> Whenever I use a pointer for memory I first check that its value is
> NULL. Then if I try to use it when it's already being used for
> something else my program tells me in a fairly unceremonious fashion.
>
> This is one of a number of defensive techniques I've developed over
> the years. It's better to wrap this kind of stuff in a library of
> defensive memory management routines (I expect you can buy one:) as
> there's less room for errors and it saves a lot of typing.
Easier still, just use boehm gc. ;-)
(I've found boehm gc to not only be easier to work with, but also to be
*faster* than "hand" de-allocating memory)
--
/||** Founder and CTO ** ** http://axkit.com/ **
//||** AxKit.com Ltd ** ** XML Application Serving **
// ||** http://axkit.org ** ** XSLT, XPathScript, XSP **
// \\| // ** mod_perl news and resources: http://take23.org **
\\//
//\\
// \\
Re: Apache growing (memory)
Kurt George Gjerde <[EMAIL PROTECTED]> writes: > Each time a page is downloaded the Apache service process claims more > memory. Well, not each time but like for every 20th download the task > manager shows Apache using 20-30K more... > > A test showed that reloading the same page 2000 times raised Apaches > memory by approx 1000k. > Even if your script were coded perfectly, it is still possible for this to happen in modperl. Three main reasons are 1) the copy-on-write effect will cause memory growth (~4K per page), 2) Perl's garbage collection and overall memory management does not really mesh well with modperl, 3) Some XS modules do leak, and occasionally so does perl itself :( Personally, I would consider an average growth rate of only .5kB/hit absolutely wonderful :) However, odds are that if you are just starting out with mod_perl, your script can be written in a more memory-friendly way. Be sure to consult the guide for tips: http://perl.apache.org/guide/ -- Joe Schaefer
Re: Apache growing (memory)
On Wed, Apr 25, 2001 at 10:02:06AM -0400, Brendan McAdams wrote: Our application performance actually > improved across the board when we implements MaxRequests... (This Do you have numbers to back this up? How does reading in a new script every now and then IMPROVE anything compared to keeping in memory all the time? It's not like perl gets "tired" after using the same memory all the time. Memory leakage is a separate issue which I don't seem to be able to solve any how with use strict and I'm not using any global variables at all.
RE: Apache growing (memory)
It sounds like you are globalising some variables and never freeing them, to start with. You definitely need to run in strict and keep your namespaces managed lest you have processes keep variables in Memory. I've found that setting MaxRequestsPerChild in apache to a reasonable amount minimises this leakage also: Our application performance actually improved across the board when we implements MaxRequests... (This basically kills a child process and restarts it after it serves $x number of requests: Starting with a fresh memory space and freeing the old). - Brendan W. McAdams | [EMAIL PROTECTED] Senior Applications Developer | (212) 208-9116 TheMuniCenter, LLC | www.themunicenter.com "Always listen to experts. They'll tell you what can't be done, and why. Then do it." - Robert A. Heinlein -Original Message- From: Kurt George Gjerde [mailto:[EMAIL PROTECTED]] Sent: Wednesday, April 25, 2001 08:13 To: [EMAIL PROTECTED] Subject: Apache growing (memory) Hi, I recently discovered the following on an Apache/mod_perl server. I have Apache for win32, ActiveState's Perl and the mod_perl PPM installed (on a Windows 2000 Pro OS, but read on anyway ;). All were downloaded and installed last week so they are the latest versions. Each time a page is downloaded the Apache service process claims more memory. Well, not each time but like for every 20th download the task manager shows Apache using 20-30K more... A test showed that reloading the same page 2000 times raised Apaches memory by approx 1000k. At first I thought that this was due to some sloppy programming (of mine) in my recent mod_perl handler, but not so. When using an empty handler (just the handler sub shell without any code inside it) the same thing occurred. This does not happen when loading non-script files (plain html). Then I did the same test on an ordinary CGI Perl script (no mod_perl) and the same thing happened then as well (so why am I posting this here?!). Since this is on win32 Apache will continue using the same (sub-)process until it's restarted (unlike Unix; though it IS possible to set up a 'retirement plan' on win32 as well). My fear is that Apache will end up using all the RAM... Does anyone know why this is happening? Is it the win32 build of Apache or does this happen on other platforms also? Thanks, -Kurt. __ kurt george gjerde <[EMAIL PROTECTED]> dept. of media studies, university of bergen
Apache growing (memory)
Hi, I recently discovered the following on an Apache/mod_perl server. I have Apache for win32, ActiveState's Perl and the mod_perl PPM installed (on a Windows 2000 Pro OS, but read on anyway ;). All were downloaded and installed last week so they are the latest versions. Each time a page is downloaded the Apache service process claims more memory. Well, not each time but like for every 20th download the task manager shows Apache using 20-30K more... A test showed that reloading the same page 2000 times raised Apaches memory by approx 1000k. At first I thought that this was due to some sloppy programming (of mine) in my recent mod_perl handler, but not so. When using an empty handler (just the handler sub shell without any code inside it) the same thing occurred. This does not happen when loading non-script files (plain html). Then I did the same test on an ordinary CGI Perl script (no mod_perl) and the same thing happened then as well (so why am I posting this here?!). Since this is on win32 Apache will continue using the same (sub-)process until it's restarted (unlike Unix; though it IS possible to set up a 'retirement plan' on win32 as well). My fear is that Apache will end up using all the RAM... Does anyone know why this is happening? Is it the win32 build of Apache or does this happen on other platforms also? Thanks, -Kurt. __ kurt george gjerde <[EMAIL PROTECTED]> dept. of media studies, university of bergen
