Re: Memory leak on reload when the 'Pg' driver is preloaded
KGM == Keith G Murphy [EMAIL PROTECTED] writes: KGM When using a modular mod_perl, I get a huge leak if I preload the 'Pg' KGM driver in my startup perl script thus: Hmmm. Interesting theory. I shall have to investigate it. I also see a multi-megabyte memory leak in my app when DBD::Pg is preloaded, but I didn't track it down to just that module. Does it happen for you for any perl module that loads another XS compiled (ie, .so) file in addition to the .pm file? I'm on FreeBSD. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Vivek Khera, Ph.D.Khera Communications, Inc. Internet: [EMAIL PROTECTED] Rockville, MD +1-240-453-8497 AIM: vivekkhera Y!: vivek_khera http://www.khera.org/~vivek/
Re: Memory leak on reload when the 'Pg' driver is preloaded
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
OK, it gets weirder. The following script produces the leak. If I
comment out the install_driver line, I get a big old segfault! Same
if I comment out the Apache::DBI line in addition. This works with
plain apache, or apache-ssl.
#!/usr/bin/perl
use strict;
use Apache::Status ();
use Apache::DBI ();
DBI-install_driver('Pg');
1; #return true value
I looked into it with the following setup:
apache 1.3.26-0woody1
libapache-mod-perl 1.27-2
postgresql 7.2.1-2woody2
There was a Test.pm module handling all requests for /. It opened a
connection to the database and fetched a couple of rows.
With DBI-install_driver('Pg'), the Apache processes stayed at 8.4 MB. I
bombarded the Apache with ApacheBench and multiple SIGUSR1 but I found
no memory leaks. The process sizes did not change.
If I comment it all out including the Apache::Status line, I get
neither the segfault or the leak. (Well, small leak, a few bytes).
Without the install_driver() in place, everything worked as well. The
Apache processes stayed at near 6 MB. No segfaults. No problems.
So the install_driver masks a bigger problem with Apache::Status?
That's just too weird. But true, I swear.
Is there anyone else out there with Debian stable w/
apache+libapache-mod-perl who can try to reproduce this simple case?
I'd really like either a confirmation or a counterexample. If the
latter, I'd chalk it up to a weirdness on my system (and probably not
even pursue it, since I've moved on to the static version). I'm
pursuing it this far so Dan J. might get some useful feedback.
I'm pretty tired at the moment, so there might have been something I
mislooked. I'll give it another look tomorrow. At work, where I don't
have any distractions :)
It's pretty odd, in my view, that the Apache processes which did not
preload the Pg driver were smaller.
Keith, could you post results to modperl list if people at debian-isp
come to a different (or to the same) solution. I'm not subscribed to
the list.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE9rxEXnksV4Ys/z5gRAqlBAJ9xKgkCmCxsg8huo/C1N0uk2XKSowCgoCdr
FdyuRGCkZvDifk5fyqfOcdo=
=im3K
-END PGP SIGNATURE-
Re: Memory leak on reload when the 'Pg' driver is preloaded
Juha-Mikko Ahonen wrote:
I looked into it with the following setup:
apache 1.3.26-0woody1
libapache-mod-perl 1.27-2
postgresql 7.2.1-2woody2
There was a Test.pm module handling all requests for /. It opened a
connection to the database and fetched a couple of rows.
With DBI-install_driver('Pg'), the Apache processes stayed at 8.4 MB. I
bombarded the Apache with ApacheBench and multiple SIGUSR1 but I found
no memory leaks. The process sizes did not change.
The interesting thing I see here is that you are using a different
libapache-mod-perl version. I'm using the one from stable. I hesitate
to install yours (from unstable), since apt-get wants to upgrade all of
perl (a whole *bunch* of shit), and I'm not keen on the idea of
permanently tracking unstable with something as complex as Perl and its
dependents. I mean, crap, it even wants to upgrade libc6. Last time I
tried, downgrading libc6 was not fun. I'm guessing you might find it
difficult to downgrade as well, since you probably have Perl 5.8. Any
ideas?
I'm pretty tired at the moment, so there might have been something I
mislooked. I'll give it another look tomorrow. At work, where I don't
have any distractions :)
I appreciate you looking at this. But it's mainly for the maintainer at
this point...
It's pretty odd, in my view, that the Apache processes which did not
preload the Pg driver were smaller.
Why would that be odd? Seems intuitively right to me.
Keith, could you post results to modperl list if people at debian-isp
come to a different (or to the same) solution. I'm not subscribed to
the list.
Will do, for sure.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Using Debian's static-mod_perled apache-perl eliminates the problem. Do you mean you are using the 'so' version that comes with Debian? You should be using the static build of apache/mod_perl
Re: Memory leak on reload when the 'Pg' driver is preloaded
Stathy G. Touloumis wrote: Using Debian's static-mod_perled apache-perl eliminates the problem. Do you mean you are using the 'so' version that comes with Debian? Yes, in the case that failed. The package is called 'libapache-mod-perl'. You should be using the static build of apache/mod_perl You're telling me. :-( By should, do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? Might it be related to the Perl-malloc problem cited here: http://apache.perl.org/docs/1.0/guide/install.html#When_DSO_can_be_Used If either of the above or true, seems like Debian needs to quit distributing libapache-mod-perl. Thanks.
Re: Memory leak on reload when the 'Pg' driver is preloaded
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wednesday 16 October 2002 20:25, Keith G. Murphy wrote: By should, do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? Yes. The problems with DSO mod_perl are well known. It works perfectly when build staticly into Apache, though. Install apache-perl package instead of apache and libapache-mod-perl. Everything works with it as it would on normal Apache. After migrating to apache-perl, remove the LoadModule for mod_perl from your Apache configuration. Might it be related to the Perl-malloc problem cited here: http://apache.perl.org/docs/1.0/guide/install.html#When_DSO_can_be_Us ed If either of the above or true, seems like Debian needs to quit distributing libapache-mod-perl. The problem is not Debian-related, so it won't justify removing the package from Debian. -BEGIN PGP SIGNATURE- Version: GnuPG v1.0.6 (GNU/Linux) Comment: For info see http://www.gnupg.org iD8DBQE9raVtnksV4Ys/z5gRAt+TAJwOZXcdnVNoLWLza7NU6eOpCOB+dQCdEliU VDvG6M+qI88/RN/jj7KsCC0= =0/Bl -END PGP SIGNATURE-
Re: Memory leak on reload when the 'Pg' driver is preloaded
Hi there, On Wed, 16 Oct 2002, Keith G. Murphy wrote: do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? It's not as bad as that. Significant improvements have been made in the reliability of mod_perl as DSO and nowadays there is much less discussion about it on this list. There are still one or two dusty corners but in general thesedays I'd say try it. If it doesn't seem to give you problems then stay with it. If at first you don't succeed, try again. Then give up. No sense being a damned fool about it. (Somebody remind me who said that:) 73, Ged.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Ged Haywood wrote: Hi there, On Wed, 16 Oct 2002, Keith G. Murphy wrote: do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? It's not as bad as that. Significant improvements have been made in the reliability of mod_perl as DSO and nowadays there is much less discussion about it on this list. Are you sure it's not because 'most everyone has silently given up on it? There are still one or two dusty corners but in general thesedays I'd say try it. I had not expected Debian stable to be one of the dusty corners. But I did find, upon investigating the bug reports, that there were *very* old reports about memory leaks, etc., with libapache-mod-perl. My own bug report is now 47 days old, without apparent followup. If it doesn't seem to give you problems then stay with it. If at first you don't succeed, try again. Then give up. Actually, that is what I have done already, several months ago. Seeing several reports of memory leak problems in the list made me think: Hmmm, wonder if someone else has seen this?
Re: Memory leak on reload when the 'Pg' driver is preloaded
Juha-Mikko Ahonen wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wednesday 16 October 2002 20:25, Keith G. Murphy wrote: By should, do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? Yes. The problems with DSO mod_perl are well known. It works perfectly when build staticly into Apache, though. Install apache-perl package instead of apache and libapache-mod-perl. Everything works with it as it would on normal Apache. After migrating to apache-perl, remove the LoadModule for mod_perl from your Apache configuration. I already have, months ago. Might it be related to the Perl-malloc problem cited here: http://apache.perl.org/docs/1.0/guide/install.html#When_DSO_can_be_Us ed If either of the above or true, seems like Debian needs to quit distributing libapache-mod-perl. The problem is not Debian-related, so it won't justify removing the package from Debian. Huh? I would think that would be a really good reason to yank it. If it were Debian-related, the maintainer could fix it and keep it.
RE: Memory leak on reload when the 'Pg' driver is preloaded
For what it is worth, we use a DSO mod_perl/apache that we compiled ourselves, on RedHat thought, that is very stable and does not have noticeable memory leaks and have been using it for over 3 years. Just thought I would throw that out there. -Original Message- From: Keith G. Murphy [mailto:[EMAIL PROTECTED]] Sent: Wednesday, October 16, 2002 2:02 PM To: mod_perl Mailing List Subject: Re: Memory leak on reload when the 'Pg' driver is preloaded Ged Haywood wrote: Hi there, On Wed, 16 Oct 2002, Keith G. Murphy wrote: do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? It's not as bad as that. Significant improvements have been made in the reliability of mod_perl as DSO and nowadays there is much less discussion about it on this list. Are you sure it's not because 'most everyone has silently given up on it? There are still one or two dusty corners but in general thesedays I'd say try it. I had not expected Debian stable to be one of the dusty corners. But I did find, upon investigating the bug reports, that there were *very* old reports about memory leaks, etc., with libapache-mod-perl. My own bug report is now 47 days old, without apparent followup. If it doesn't seem to give you problems then stay with it. If at first you don't succeed, try again. Then give up. Actually, that is what I have done already, several months ago. Seeing several reports of memory leak problems in the list made me think: Hmmm, wonder if someone else has seen this?
Re: Memory leak on reload when the 'Pg' driver is preloaded
Hi there, On Wed, 16 Oct 2002, Keith G. Murphy wrote: Significant improvements have been made in the reliability of mod_perl as DSO and nowadays there is much less discussion about it on this list. Are you sure it's not because 'most everyone has silently given up on it? Yes, I'm sure. Red Hat comes with Apache prebuilt for DSO I believe. I have worked on contracts for a number of organizations which use DSO mod_perl in high volume environments (hundreds of thousands of users). There are still one or two dusty corners but in general thesedays I'd say try it. I had not expected Debian stable to be one of the dusty corners. What I meant was that some (perhaps slightly unusual in some way) configurations seem to give more problems than others. For example I think that servers using SSL have perhaps been prone to troubles, but it could just as easily be that the people who experienced the problems were not careful to follow what you might say are slightly more complex than usual build procedures. 73, Ged.
Re: Memory leak on reload when the 'Pg' driver is preloaded
Ged Haywood wrote: Hi there, On Wed, 16 Oct 2002, Keith G. Murphy wrote: Significant improvements have been made in the reliability of mod_perl as DSO and nowadays there is much less discussion about it on this list. Are you sure it's not because 'most everyone has silently given up on it? Yes, I'm sure. Red Hat comes with Apache prebuilt for DSO I believe. I have worked on contracts for a number of organizations which use DSO mod_perl in high volume environments (hundreds of thousands of users). Then I take your word for it. (My question was sincere, not argumentative, by the way). There are still one or two dusty corners but in general thesedays I'd say try it. I had not expected Debian stable to be one of the dusty corners. What I meant was that some (perhaps slightly unusual in some way) configurations seem to give more problems than others. For example I think that servers using SSL have perhaps been prone to troubles, but it could just as easily be that the people who experienced the problems were not careful to follow what you might say are slightly more complex than usual build procedures. Hmmm. In my case, I replicated the problem with Debian's plain vanilla Apache as well, for the very purpose of simplification. So SSL does not seem to be involved.
Re: Memory leak on reload when the 'Pg' driver is preloaded
On Wed, Oct 16, 2002 at 02:01:33PM -0500, Keith G. Murphy wrote: Ged Haywood wrote: Hi there, On Wed, 16 Oct 2002, Keith G. Murphy wrote: do you mean that the problems with the loadable module overall are so well-known that no one in his right mind should ever use it? It's not as bad as that. Significant improvements have been made in the reliability of mod_perl as DSO and nowadays there is much less discussion about it on this list. Are you sure it's not because 'most everyone has silently given up on it? There are still one or two dusty corners but in general thesedays I'd say try it. I had not expected Debian stable to be one of the dusty corners. But I did find, upon investigating the bug reports, that there were *very* old reports about memory leaks, etc., with libapache-mod-perl. My own bug report is now 47 days old, without apparent followup. That's because I'm having an attack of real life. I haven't had time to look at my Debian bug reports in a month or so now. I don't know why the DSO is leaking in a case when the static module isn't, but it should be fixable; I just need to find five or six hours to sit down with it. -- Daniel Jacobowitz MontaVista Software Debian GNU/Linux Developer
Re: Memory leak on reload when the 'Pg' driver is preloaded
Daniel Jacobowitz wrote: On Wed, Oct 16, 2002 at 02:01:33PM -0500, Keith G. Murphy wrote: My own bug report is now 47 days old, without apparent followup. Hmmm, I probably should not have posted that. Sounds like a major whine. That's because I'm having an attack of real life. I haven't had time to look at my Debian bug reports in a month or so now. I don't know why the DSO is leaking in a case when the static module isn't, but it should be fixable; I just need to find five or six hours to sit down with it. Thanks, Dan, this really wasn't meant to be a slam-the-developer session. I really, *really* appreciate all the work you've done already. Now I feel tacky. :-( It's not like it was an obvious problem: I only got the DSO to leak when loading the Pg driver. That's pretty obscure.
Re: Memory leak on reload when the 'Pg' driver is preloaded
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wednesday 16 October 2002 22:52, Keith G. Murphy wrote:
It's not like it was an obvious problem: I only got the DSO to leak
when loading the Pg driver. That's pretty obscure.
Have you tried to connect() without loading the Pg driver first? E.g.
$dbh = DBI-connect('dbi:Pg:dbname=foo;host=bar', 'user', 'pass');
It would help eliminate if the leak is caused by your startup script. If
it is, it may be the DBI module that's leaking memory.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org
iD8DBQE9rcZ5nksV4Ys/z5gRAjBQAJwKOUuu4B14yLe1msSNekI0mnPvdACdFMOe
TFLM0arS4L177Y+OuJMAUgM=
=GsvK
-END PGP SIGNATURE-
Re: Memory leak on reload when the 'Pg' driver is preloaded
[debian-isp readers, to recap, I'm trying to confirm a
memory-leak/segfault problem with Debian stable plus apache(-ssl) plus
libapache-mod-perl. The memory leak happens upon
/etc/init.d/apache(-ssl) reload. You can see my startup script and my
other comments below.]
Juha-Mikko Ahonen wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
On Wednesday 16 October 2002 22:52, Keith G. Murphy wrote:
It's not like it was an obvious problem: I only got the DSO to leak
when loading the Pg driver. That's pretty obscure.
Have you tried to connect() without loading the Pg driver first? E.g.
$dbh = DBI-connect('dbi:Pg:dbname=foo;host=bar', 'user', 'pass');
It would help eliminate if the leak is caused by your startup script. If
it is, it may be the DBI module that's leaking memory.
OK, it gets weirder. The following script produces the leak. If I
comment out the install_driver line, I get a big old segfault! Same if
I comment out the Apache::DBI line in addition. This works with plain
apache, or apache-ssl.
#!/usr/bin/perl
use strict;
use Apache::Status ();
use Apache::DBI ();
DBI-install_driver('Pg');
1; #return true value
If I comment it all out including the Apache::Status line, I get neither
the segfault or the leak. (Well, small leak, a few bytes).
So the install_driver masks a bigger problem with Apache::Status?
That's just too weird. But true, I swear.
Is there anyone else out there with Debian stable w/
apache+libapache-mod-perl who can try to reproduce this simple case?
I'd really like either a confirmation or a counterexample. If the
latter, I'd chalk it up to a weirdness on my system (and probably not
even pursue it, since I've moved on to the static version). I'm
pursuing it this far so Dan J. might get some useful feedback.
I've submitted this to debian-isp as well; maybe someone there can try it.
Re: Memory leak
script offers database search facilities on the web. If a search is performed which results in many (namely 400) rows being returned, then the httpd child that serves the request grows by 2 MB. Have a child serve that request ten times, and its size will grow from an initial 8 MB to 28 MB. Another ten times and it will go to around 50 MB. Depending on how the return is processed this looks more like an unused reference left behind. You might want to try, e.g., calling the handler from a harness and see where the memory is going in the debugger. It also depends on how the rows are being retrieved. If you are in a while( nextrow ) loop vs. a for( fetchall_arrayref ) type the memory use gets pretty large. Depending on the size of your queries this could also just be perl's buffering the input. Remember that perl doesn't free the space it uses but keeps it internally for later use. If you are running large enough queries over time the memory use will track the high-water mark of the rows returned (which obviuosly pertaint to fetchall-type returns). If the queries get large enough you can suck up lotsaMB. -- Steven Lembark 2930 W. Palmer St. Chicago, IL 60647 [EMAIL PROTECTED] 800-762-1582
Re: Memory leak
Hi there, On Mon, 28 May 2001, Antonios Christofides wrote: script offers database search facilities on the web. If a search is performed which results in many (namely 400) rows being returned, then the httpd child that serves the request grows by 2 MB. Have a child serve that request ten times, and its size will grow from an initial 8 MB to 28 MB. Another ten times and it will go to around 50 MB. If a row is around 5kbytes then I don't think it's a Perl/mod_perl problem, I think you're keeping your variables hanging around in memory. Don't forget that when you use mod_perl the interpreter doesn't exit, so a lot of memory cleanup which happens at exit doesn't happen under mod_perl. What happens if you set maxrequestsperchild to ten? 73, Ged.
Re: memory leak in win32
I've had memory leak issues on Linux [so I don't know how appropriate this advise is for Windows], and the fix that worked the best for others myself was to actually kill the apache server restart it. The process takes seconds, and can be automated scheduled for off-hours. If you search the archives you'll even find a log-rotation program that someone was kind enough to post. Just change it to suit your needs. Son Chang wrote: Hi, I'm experiencing a memory leak with Apache 1.3.14 with mod_perl 1.23 and perl5.6.0 on Windows NT4.0 and Windows2000. I've been running some test with a very simple ModPerl handler which just outputs an HTML page with hello world in it. Also, I made sure to undef all variables that I use. In general, the child processes grow with each request. Also, if I set MaxRequestsPerChild to 20 so the child process dies and restarts after every 20 request, this fixes the problem for the child processes. However, with this configuration, there's a memory leak problem with the parent process. After the child process dies and restarts, the parent process grows in size. This happens every time the child process dies and restarts. So with either configuration, I have a memory leak problem. What can I do to fix this problem? -son
RE: memory leak in win32
Son Chang wrote: Hi, I'm experiencing a memory leak with Apache 1.3.14 with mod_perl 1.23 and perl5.6.0 on Windows NT4.0 and Windows2000. I've been running some test with a very simple ModPerl handler which just outputs an HTML page with hello world in it. Also, I made sure to undef all variables that I use. In general, the child processes grow with each request. Also, if I set MaxRequestsPerChild to 20 so the child process dies and restarts after every 20 request, this fixes the problem for the child processes. However, with this configuration, there's a memory leak problem with the parent process. After the child process dies and restarts, the parent process grows in size. This happens every time the child process dies and restarts. So with either configuration, I have a memory leak problem. What can I do to fix this problem? -son Heh, I just unsubscribed from the mailing list, almost missed your question. We ran into the exact same problem here. Try: http://forum.swarthmore.edu/epigone/modperl/groxbloiglim/03AB3CB11CBED3118A4 F009027B0C2B16B71A4@HAPPY Our fix stops the child process growing, but not the parent. -harshy -- Harshy Wanigasekara, Software guy, Omneon Video Networks.
RE: memory leak under WinNT - fixed
Hi again,
We managed to fix the problem described
below. The issue was the win32
porting code in perl. In the file
perl-5.6.0/win32/win32.h
commenting out the line:
#define ENV_IS_CASELESS
fixes the memory leak problem.
-harshy
-Original Message-
From: Harshy Wanigasekara
Sent: Wednesday, October 25, 2000 3:11 PM
To: [EMAIL PROTECTED]
Subject: memory leak under WinNT
Hi All,
I've looked through the maillist archives,
and FAQs and guides and docs, with no help.
I've got a memory leak problem running embedded
perl under Apache on Windows NT. I've reduced
the problem set as much as I can, to the
following Apache::Registry script:
---
# Filename: test.pl
print("html
head
titleApache test/title
/head
/body
Hello world.
script LANGUAGE=\"JavaScript\"
function ReloadPageFunc()
{
var newloc = \"test.pl\";
window.location.href = newloc;
}
var myTimeOut = setTimeout(\"ReloadPageFunc()\", 1);
/script
/body
/html\n");
The script just generates a page which causes the
browser to reload the same page every 10 seconds.
The browser has no complaints about the page it receives.
When I leave this page running I get memory leaks in
the apache process.
If I remove the enclosing print(""); statement so I
just have the HTML code in a test.html file
(ie: Apache::Registry handler is NOT called), there
are no memory leaks.
In the above example, the apache process' memory usage
keeps increasing continuously until all virtual memory
is exhausted (~1 GByte). I see 32K byte jumps for
about every 100 reloads, it's worse for more complicated
scripts. Since this is Windoze, I can't limit the number
of requests a child can handle.
My setup:
Windows NT 4.0 SP 5
Apache 1.3.12
Perl 5.6.0
Modperl 1.24
(also embperl 1.3b5, but not used in above example)
All source code compiled with VC++ 6.0
I ran BoundsChecker on the apache executable in the
hopes of finding a single culprit, but found a
multitude of perl core routines allocate memory using
win32_malloc() in /perl5.6.0/win32/win32.c and
don't free it.
Anyone seen this before?
-harshy
--
Harshy Wanigasekara, Omneon Video Networks.
Re: Memory leak hell...
On Mon, 11 Sep 2000, Stas Bekman wrote: I was thinking about going thru the symbol table and dumping all the variables. And run diff between the dumps of the two requests. Be careful though that Devel::Peek doesn't show a complete dump for the whole structure, and I couldn't find the interface for changing the deepness of datastructure traversal. I think Doug has patched Apache::Peek to make this configurable. Do you think it still won't work? Yes. Because this only seems to give you access to package variables, not trapped lexicals or anything in closures. -- Matt/ Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Memory leak hell...
On Mon, 11 Sep 2000, Matt Sergeant wrote: On Mon, 11 Sep 2000, Stas Bekman wrote: I was thinking about going thru the symbol table and dumping all the variables. And run diff between the dumps of the two requests. Be careful though that Devel::Peek doesn't show a complete dump for the whole structure, and I couldn't find the interface for changing the deepness of datastructure traversal. I think Doug has patched Apache::Peek to make this configurable. Do you think it still won't work? Yes. Because this only seems to give you access to package variables, not trapped lexicals or anything in closures. Well, you don't know where the leakage is. May be the problem *is* in the package variables... At least if you prove that it's not, you narrow your search path. What I'd do is taking the module that prints the Stack trace and patch it to print memory foot print after each trace line with GTop or else, and then just grep for the first place where the memory bumps up and find the offending code. Of course you have to take into account Frank's reply. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Memory leak hell...
On Mon, 11 Sep 2000, Stas Bekman wrote: On Mon, 11 Sep 2000, Matt Sergeant wrote: On Mon, 11 Sep 2000, Stas Bekman wrote: I was thinking about going thru the symbol table and dumping all the variables. And run diff between the dumps of the two requests. Be careful though that Devel::Peek doesn't show a complete dump for the whole structure, and I couldn't find the interface for changing the deepness of datastructure traversal. I think Doug has patched Apache::Peek to make this configurable. Do you think it still won't work? Yes. Because this only seems to give you access to package variables, not trapped lexicals or anything in closures. Well, you don't know where the leakage is. May be the problem *is* in the package variables... At least if you prove that it's not, you narrow your search path. I think there are about 3 package variables in total. Its not in them. :-) What I'd do is taking the module that prints the Stack trace and patch it to print memory foot print after each trace line with GTop or else, and then just grep for the first place where the memory bumps up and find the offending code. Of course you have to take into account Frank's reply. I'm going to take a more radical approach. A re-write of major portions of the code to make it easier to track it down. *sigh*... -- Matt/ Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Memory leak hell...
On Sun, 10 Sep 2000, Matt Sergeant wrote:
For 2 days solid now I've been trying to track down a very bizarre memory
leak in AxKit.
I've checked everything I can think of - all circular references are now
gone, all closures clean up carefully after themselves, and I've reduced
the usage of some external modules. But still the processes grow.
Now to the wierd bit. I could track this down if it wasn't for this:
The memory leak starts after the Nth hit, usually around 35. This is
running under httpd -X.
So it goes along very happily for 35 hits - memory usage is rock solid
stable. Then after the 35th hit the memory usage starts to grow about 4k
per hit. Obviously thats an impossible rate of growth to sustain for any
amount of time, and soon the server is swamped.
you're leaking on every request, consider this example:
use strict;
my $i = 0;
my @a;
my $old_size = proc_size();
while (++$i) {
my $size = proc_size();
if ($size != $old_size) {
printf "Size changed from $old_size to $size, i=$i\n";
$old_size = $size;
; #pause
}
push @a, [];
}
sub proc_size {
my $size = 0;
open my $fh, "/proc/self/status";
while ($fh) {
last if (($size) = (/^VmRSS:\s+(\d+)/));
}
close $fh;
$size;
}
outputs:
Size changed from 1376 to 1436, i=1
Size changed from 1436 to 1472, i=2
Size changed from 1472 to 1476, i=55
Size changed from 1476 to 1480, i=99
Size changed from 1480 to 1484, i=158
Size changed from 1484 to 1488, i=204
Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5,
Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8,
Unicode::String, MIME::Base64, Storable (loaded but not used),
XML::Sablotron (loaded but not used).
look for xsubs in those modules you're using that are calling
new{SV,AV,HV,RV} or SvREFCNT_inc(). sounds like botched refcnting or
failure to mortalize a new*V.
actually, my first guess is Apache::ModuleConfig-get, if you don't
explicitly pass __PACKAGE__ as the last argument, perl_eval_pv() is
called, which has a leak plugged in perl-current:
[ 6201] By: gsar on 2000/06/06 00:42:59
Log: Perl_eval_pv() leaks 4 bytes every time it is called because it
does a PUSHMARK that's never ever POPMARKed; in general, only
Perl_call_[sp]v() need a PUSHMARK for incoming arguments;
Perl_eval_[sp]v() don't because they don't take any incoming
arguments (this leak has been around since the original version
of perl_eval_pv() in 5.003_97e)
Branch: perl
! perl.c
so first try changing:
Apache::ModuleConfig-get($r)
to
Apache::ModuleConfig-get($r, __PACKAGE__)
RE: Memory leak hell...
-Original Message-
From: Doug MacEachern [mailto:[EMAIL PROTECTED]]
Sent: Monday, September 11, 2000 2:54 PM
To: Matt Sergeant
Cc: [EMAIL PROTECTED]
Subject: Re: Memory leak hell...
[snip]
look for xsubs in those modules you're using that are calling
new{SV,AV,HV,RV} or SvREFCNT_inc(). sounds like botched refcnting or
failure to mortalize a new*V.
actually, my first guess is Apache::ModuleConfig-get, if you don't
explicitly pass __PACKAGE__ as the last argument, perl_eval_pv() is
called, which has a leak plugged in perl-current:
[ 6201] By: gsar on
2000/06/06 00:42:59
Log: Perl_eval_pv() leaks 4 bytes every time it is
called because it
does a PUSHMARK that's never ever POPMARKed; in
general, only
Perl_call_[sp]v() need a PUSHMARK for incoming arguments;
Perl_eval_[sp]v() don't because they don't take
any incoming
arguments (this leak has been around since the
original version
of perl_eval_pv() in 5.003_97e)
Branch: perl
! perl.c
so first try changing:
Apache::ModuleConfig-get($r)
to
Apache::ModuleConfig-get($r, __PACKAGE__)
that's a good bit of info to know - thanks...
--Geoff
Re: Memory leak hell...
On Sun, 10 Sep 2000, Matt Sergeant wrote: For 2 days solid now I've been trying to track down a very bizarre memory leak in AxKit. I've checked everything I can think of - all circular references are now gone, all closures clean up carefully after themselves, and I've reduced the usage of some external modules. But still the processes grow. Now to the wierd bit. I could track this down if it wasn't for this: The memory leak starts after the Nth hit, usually around 35. This is running under httpd -X. So it goes along very happily for 35 hits - memory usage is rock solid stable. Then after the 35th hit the memory usage starts to grow about 4k per hit. Obviously thats an impossible rate of growth to sustain for any amount of time, and soon the server is swamped. Can anyone give me _any_ help in figuring out where this might be coming from? I've tried several things, including adding the ability to display the memory usage in the debug statements (by getting the VmSize out of /proc/self/status). This is driving me absolutely mad, as you might tell. The only thing I can come close to in locating it is some remote possibility that it is in one particular section of the code, but that doesn't actually seem to be the case - sometimes the memory increase goes outside that section of code. The modules I'm using are: Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5, Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8, Unicode::String, MIME::Base64, Storable (loaded but not used), XML::Sablotron (loaded but not used). mod_perl is 1.24 and Perl is 5.00503 (Mandrake release). First, Apache::VMonitor and GTop will make your debugging work much easier. Second, The only reason that I can think of that things start happening after Nth hit is that one of the modules that you use does an internal accounting that goes astrey after Nth hit. Using Apache::Peek will help you discover the leakage by dumping and comparing the complete package variables tables. You will find some examples of using Apache::Peek in the guide. This will definitely allow you to spot the offending module. Of course the complete list of loaded modules can be found in /perl-status :) Apache::Leak is another thing to try, but it's not really good unless you know the exact offensive code. Otherwise it reports leakages for things which are Perl optimizations in fact. Yet another thing I'd think about is monitoring the code with GTop, and running it under strace/gdb. So it goes like this: You run a daemon that watches the processes and prints its memory size via Gtop. You complete 35 hits and now run the server under Apache::DB debugger. So you do 's' by 's' (step by step) and watch the printouts of the GTop daemon. Once you see the growth -- you know the offending module/line - and voila... of course this should be automized... And while you work on this it would be really really cool to have this documented. I can imagine that you will have to write a few snippets of code to automate things... But this would be a treasure for those in the same trouble like you. Hope this helps. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Memory leak hell...
Matt Sergeant wrote: Can anyone give me _any_ help in figuring out where this might be coming from? When I'm working on problems like this, there are two basic things I try. They're not rocket science, but they usually work. The first is removing sections of code until the leak goes away. Then you get to tear your hair out wondering why that subrotuine leaks, but at least you'll know where it is. The second is to run under the debugger, with top running in strobe-light mode (refresh 0) in another window, watching for which line pushes the memory up. You've already named the usual suspects - closures and circular refs - but some closure problems can be very subtle. There's also the memory-related stuff that Apache::Status provides, but I found it difficult to get useful info out of it. - Perrin
Re: Memory leak hell...
On Sun, 10 Sep 2000, Stas Bekman wrote: On Sun, 10 Sep 2000, Matt Sergeant wrote: For 2 days solid now I've been trying to track down a very bizarre memory leak in AxKit. I've checked everything I can think of - all circular references are now gone, all closures clean up carefully after themselves, and I've reduced the usage of some external modules. But still the processes grow. Now to the wierd bit. I could track this down if it wasn't for this: The memory leak starts after the Nth hit, usually around 35. This is running under httpd -X. So it goes along very happily for 35 hits - memory usage is rock solid stable. Then after the 35th hit the memory usage starts to grow about 4k per hit. Obviously thats an impossible rate of growth to sustain for any amount of time, and soon the server is swamped. Can anyone give me _any_ help in figuring out where this might be coming from? I've tried several things, including adding the ability to display the memory usage in the debug statements (by getting the VmSize out of /proc/self/status). This is driving me absolutely mad, as you might tell. The only thing I can come close to in locating it is some remote possibility that it is in one particular section of the code, but that doesn't actually seem to be the case - sometimes the memory increase goes outside that section of code. The modules I'm using are: Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5, Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8, Unicode::String, MIME::Base64, Storable (loaded but not used), XML::Sablotron (loaded but not used). mod_perl is 1.24 and Perl is 5.00503 (Mandrake release). First, Apache::VMonitor and GTop will make your debugging work much easier. Not really. :-( Second, The only reason that I can think of that things start happening after Nth hit is that one of the modules that you use does an internal accounting that goes astrey after Nth hit. Using Apache::Peek will help you discover the leakage by dumping and comparing the complete package variables tables. You will find some examples of using Apache::Peek in the guide. This will definitely allow you to spot the offending module. Of course the complete list of loaded modules can be found in /perl-status :) I don't see this example. It doesn't seem like ::Peek gives me any useful information at all unless I recompile Perl with DEBUGGING_MSTATS, and that seems like a nightmare... Apache::Leak is another thing to try, but it's not really good unless you know the exact offensive code. Otherwise it reports leakages for things which are Perl optimizations in fact. I've tried ::Leak before and its not very helpful. Doesn't even give you a clue where the variables that are leaking are located. Yet another thing I'd think about is monitoring the code with GTop, and running it under strace/gdb. So it goes like this: You run a daemon that watches the processes and prints its memory size via Gtop. You complete 35 hits and now run the server under Apache::DB debugger. So you do 's' by 's' (step by step) and watch the printouts of the GTop daemon. Once you see the growth -- you know the offending module/line - and voila... of course this should be automized... While I may now have started to hallucinate due to working on this non-stop for about 40 hours, this hasn't helped. The leak still moves around. I think maybe I have to go back to the drawing board on this one :-( -- Matt/ Fastnet Software Ltd. High Performance Web Specialists Providing mod_perl, XML, Sybase and Oracle solutions Email for training and consultancy availability. http://sergeant.org | AxKit: http://axkit.org
Re: Memory leak hell...
On Sun, 10 Sep 2000, Matt Sergeant wrote: On Sun, 10 Sep 2000, Stas Bekman wrote: On Sun, 10 Sep 2000, Matt Sergeant wrote: For 2 days solid now I've been trying to track down a very bizarre memory leak in AxKit. I've checked everything I can think of - all circular references are now gone, all closures clean up carefully after themselves, and I've reduced the usage of some external modules. But still the processes grow. Now to the wierd bit. I could track this down if it wasn't for this: The memory leak starts after the Nth hit, usually around 35. This is running under httpd -X. So it goes along very happily for 35 hits - memory usage is rock solid stable. Then after the 35th hit the memory usage starts to grow about 4k per hit. Obviously thats an impossible rate of growth to sustain for any amount of time, and soon the server is swamped. Can anyone give me _any_ help in figuring out where this might be coming from? I've tried several things, including adding the ability to display the memory usage in the debug statements (by getting the VmSize out of /proc/self/status). This is driving me absolutely mad, as you might tell. The only thing I can come close to in locating it is some remote possibility that it is in one particular section of the code, but that doesn't actually seem to be the case - sometimes the memory increase goes outside that section of code. The modules I'm using are: Time::HiRes, Apache::* (core apache stuff only), AxKit, Digest::MD5, Compress::Zlib, Fcntl, XML::Parser, XML::XPath, Unicode::Map8, Unicode::String, MIME::Base64, Storable (loaded but not used), XML::Sablotron (loaded but not used). mod_perl is 1.24 and Perl is 5.00503 (Mandrake release). First, Apache::VMonitor and GTop will make your debugging work much easier. Not really. :-( I meant that GTop lets you an easier access to memory usage. Of course you don't need it if you read directly the /proc/mem Second, The only reason that I can think of that things start happening after Nth hit is that one of the modules that you use does an internal accounting that goes astrey after Nth hit. Using Apache::Peek will help you discover the leakage by dumping and comparing the complete package variables tables. You will find some examples of using Apache::Peek in the guide. This will definitely allow you to spot the offending module. Of course the complete list of loaded modules can be found in /perl-status :) I don't see this example. It doesn't seem like ::Peek gives me any useful information at all unless I recompile Perl with DEBUGGING_MSTATS, and that seems like a nightmare... I was thinking about going thru the symbol table and dumping all the variables. And run diff between the dumps of the two requests. Be careful though that Devel::Peek doesn't show a complete dump for the whole structure, and I couldn't find the interface for changing the deepness of datastructure traversal. I think Doug has patched Apache::Peek to make this configurable. Do you think it still won't work? Apache::Leak is another thing to try, but it's not really good unless you know the exact offensive code. Otherwise it reports leakages for things which are Perl optimizations in fact. I've tried ::Leak before and its not very helpful. Doesn't even give you a clue where the variables that are leaking are located. Yup, that's what I've said :( Yet another thing I'd think about is monitoring the code with GTop, and running it under strace/gdb. So it goes like this: You run a daemon that watches the processes and prints its memory size via Gtop. You complete 35 hits and now run the server under Apache::DB debugger. So you do 's' by 's' (step by step) and watch the printouts of the GTop daemon. Once you see the growth -- you know the offending module/line - and voila... of course this should be automized... While I may now have started to hallucinate due to working on this non-stop for about 40 hours, this hasn't helped. The leak still moves around. I think maybe I have to go back to the drawing board on this one :-( BTW, Frank's comment seems cool, I didn't think about it. What happens if you create some small variable and start making it bigger (string?) and see whether you still get this event on hit #36. May be you at least could move it from #36 to #3. It's a documented fact that the data structures change from hit #1 and #2, but not #3 assumming that it's the same request. _ Stas Bekman JAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide http://perl.apache.org/guide mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Memory Leak
On Tue, 15 Aug 2000, vegan.star wrote:
I have some mod_perl modules. I suspect that it has a memory
leak. I'm running that in a Sun Solaris 2.6 machine with apache
1.3.9. I read that exists Apache::Leak to test for leaks.
How it works?
I have some packages and I put into them like this:
package package_name;
use Apache::Leak;
leak_test{
global vars
sub handler
{
statements
}
sub another_function
{
statements
}
};
Is it correct? I thought not, because the answers are always no
leaks.
Anybody can helps me? Another idea to detect the leaks?
Well yes,
http://thingy.kcilink.com/modperlguide/debug/How_can_I_find_out_if_a_mod_perl.html
but Apache::Leak is not an easy thing to use for leakage detection, since
manytimes what's reported as a leakage is in fact a Perl internal
optimization.
Look at the end of the section from the above link and use the
Apache::Status module with StatusLexInfo option enabled -- this should
reveal more info. See the manpage for more info.
BTW, Solaris has been known to have memory leakages in its C libraries,
but I'm not sure whether it's still true. I think it was prior to 2.6. I
don't use Solaris so I cannot confirm, but I remember the reports about
this from the previous years.
_
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:[EMAIL PROTECTED] http://apachetoday.com http://jazzvalley.com
http://singlesheaven.com http://perlmonth.com perl.org apache.org
Re: Memory leak in DBD::Connect()
Of course, the short term fix, is to lower MaxRequestsPerChild which will hopefully reduce the leakage but won't help performance. Leaks in DBD::Oracle and Oracle's own libraries are not entirely rare. You should send your simple, stand-alone example to the dbi-user's mailing list with your exact version and platform info. John [EMAIL PROTECTED]
Re: memory leak during server graceful restarts
On Fri, 7 Apr 2000, Nikki Chumakov wrote: mod_perl probaly have memory leakage during rereading configs (e.g. on the apachectl graceful) do you have PerlFreshRestart On? if so, try turning it off. i think dso leaks on restart too, try linking static if that's the problem. a static httpd that doesn't have PerlFreshRestart On doesn't run any Perl code, so it can't leak. unless you have Perl sections in httpd.conf, do you?
Re: memory leak during server graceful restarts
Doug MacEachern wrote: On Fri, 7 Apr 2000, Nikki Chumakov wrote: mod_perl probaly have memory leakage during rereading configs (e.g. on the apachectl graceful) do you have PerlFreshRestart On? if so, try turning it off. i think dso leaks on restart too, try linking static if that's the problem. a static httpd that doesn't have PerlFreshRestart On doesn't run any Perl code, so it can't leak. unless you have Perl sections in httpd.conf, do you? Well, actually I checked without any perl_mod specific config statements in httpd.conf and without any http requests. With LoadModule/AddModule mod_perl the server grows in size during every reload, and doesn't grow with AddModule mod_perl commented out. So, this is apparently mod_perl DSO problem, that happens before compiling any Perl code.
Re: memory leak during server graceful restarts
I have run into the same issue also both with mod_perl as a DSO module or compiled staticly. I have tested it using apache-1.3.6 and apache-1.3.12 w/ mod_perl-1.22 and mod_perl-1.21. The memory growth also occurs if I HUP the server rather than sending USR1. Since the problem only occurs if mod_perl is loaded (DSO) or staticly loaded, I assume the problem lies in memory cleanup by modperl. Even if I do nothing other than have mod_perl installed and HUP the parent, the process will grow at a relatively rapid rate of over 1MB per HUP instance. The reason I noticed this is that I cache some relatively static database information in the parent process by assignment to a package's scoped hash variables in my startup.pl. I wanted to refresh this cache hourly just in case changes did happen to occur in the database tables I was caching. I am curious as to whether there is a better way to go about caching this data and refreshing it without the problem of watching your apache process size grow every time you HUP it. Any helpful hints would be appreciated. The platforms are various Linux kernels ranging from 2.0x to 2.3x. The majority of which are linux-2.2.10. Douglas Leonard [EMAIL PROTECTED] On Fri, 7 Apr 2000, Nikki Chumakov wrote: mod_perl probaly have memory leakage during rereading configs (e.g. on the apachectl graceful) My system: RH 6.1, linux-2.2.15pre17 glibc-2.1 apache-1.3.12 mod_ssl-2.6.2 russian patches (ftp.lexa.ru/pub/apache-rus/) PL29.4 mm-1.0.12 mod_perl-1.22 With mod_perl module enabled in httpd.conf every 'apachectl graceful' increase the data memory segment of parent server (and thus the child's as well).
Re: Memory leak/server crashes
there are hints in the SUPPORT doc on how to debug such problems. there
was also several "Hanging process" threads in the past weeks with more
tips, search in the archives for keywords gdb, .gdbinit, curinfo
if you can get more insight from those tips, we can help more.
On Sun, 9 Jan 2000, James Furness wrote:
I'm looking for some help getting apache to run reliably. Apache 1.3.9 with
mod_perl 1.21 and Perl 5.005_03 is running on a dual P500 with 1 Gb of RAM
running Redhat 6.1. We run about 5 sites off the box, most of which are
fairly high traffic, and use a lot of CGI and
MySQL 3.22.25 is used with Apache::DBI.
The major problem seems to be a memory leak of some sort, identical to that
described in the "memory leak in mod_perl" thread on this list from October
1997 and the "httpd, mod_perl and memory consumption (long)" thread from
July 1997.
The server runs normally for several hours, then suddenly a httpd process
starts growing exponentially, the swapfile usage grows massively and the
server starts to become sluggish (I assume due to disk thrashing caused by
the heavy swap usage). Usually when this started to happen I would log in
and use apachectl stop to shutdown the server, then type 'killall httpd'
several times till the processes finally died off, and then use apachectl
start to restart apache. If I was not around or did not catch this, the
server would eventually become unresponsive and lock up, requiring a manual
reboot by the datacentre staff. Messages such as "Out of memory" and
"Callback called exit" would appear in the error log as the server spiralled
down and MySQL would start to have trouble running.
To combat this, I created a script to monitor load and swapfile usage, and
restart apache as described above if load was above 7 and swapfile usage
above 150Mb. This script has kept the server online and we now have an
uptime of something like 22 days (previously no more than 1 day), but the
script is getting triggered several times a day and no more "Out of memory"
messages are appearing, but the situation is not ideal.
I have tried adding:
sub UNIVERSAL::AUTOLOAD {
my $class = shift;
Carp::cluck "$class can't \$UNIVERSAL::AUTOLOAD!\n";
}
As recommended by the developers guide, which flooded the error log with the
text below being printed roughly once a second in the error log:
-
Apache=SCALAR(0x830937c) can't $UNIVERSAL::AUTOLOAD!
Apache=SCALAR(0x8309364) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
--
I've pretty much exhausted any ways I can think of to trace this problem,
such as i've tried to eliminate memory leaks in code by removing some
scripts from mod_perl and running them under mod_cgi and i've tried tweaking
MaxRequestsPerChild both without any success.
One thing that was mentioned in a previous thread was that using 'exit'
could confuse perl, and exit() is used fairly heavily in the scripts since
most are converted to mod_perl from standard CGIs, but i'd prefer not to
have to remove these since the structure of the scripts is reliant on some
form of exit statement. Is there some alternative to exit()?
I've also had a look at some of the patches to Apache.pm and Apache.xs
suggested in the previous threads, and these seem to have been incorporated
into mod_perl 1.21.
Are there any other solutions I could try to this problem? Does anyone know
what might be causing this?
The second problem I have is when loading pages, usually CGI, but I think
this has happened on some static pages, what IE5 describes as "Server not
found or DNS error" is experienced. Originally I thought this was the server
hitting MaxClients (150) since it usually occurs at the same time as massive
surges of hits, and /server-status usually shows 150 httpd processes have
been spawned, however I increased MaxClients to 200 recently and the error
has continued to happen, even though /server-status doesn't show any more
than about 170 processes spawned. I have not ruled out DNS server troubles
or backbone problems (We've had a few routing troubles recently that slowed
things down, but not actually cut off traffic or anything like that), but I
am at a loss as to what else could be causing this so I thought i'd ask
whilst i'm on the subject of server problems :)
Thanks in advance,
--
James Furness [EMAIL PROTECTED]
ICQ #: 4663650
Re: Memory leak/server crashes
Why, reinvent the wheel? I wrote Apache::VMonitor (grab from CPAN) that does all this and more (all but tail -f) I use it all the time, saves me a lot of time, since I don't have to telnet! Ok - I will try to look into that when I get time. 2) Open up and hack Apache::SizeLimit and have it do a stack dump (Carp::croak) of what's going on... there may be some clue there. I've done this (In a PerlRequire'd file): -- # Use apache process size limitation use Apache::SizeLimit; $Apache::SizeLimit::MAX_PROCESS_SIZE = 9000; $Apache::SizeLimit::CHECK_EVERY_N_REQUESTS = 3; --- and 'PerlCleanupHandler Apache::SizeLimit' in httpd.conf. The server is still getting restarted by the uptime/swapfile monitor, so I'm not sure if this is having an effect. I'll look into opening up sizelimit and doing a stack dump as soon as I get time. Apache::GTopLimit is an advanced one :) (you are on Linux, right?) but Apache::SizeLimit is just file I had some problems with GTop, I was trying to use Apache::VMonitor, I downloaded and installed libgtop and the other packages needed (Forget which now) and tried to install VMonitor, but it failed on make test, couldn't locate one of the packages I definitely installed, a graphics manipulation one from memory, but i'm writing this e-mail offline so I can't check :) 3) try running in single mode with 'strace' (probably not a good idea for a production server), but you can still strace all the processes into a log file Well I might be able to get a server running in single mode on a different port and try that, it would be worth the information gained if I can sort this problem out :) 4) Apache::Leak ? Ok, will look at that too. -- James Furness [EMAIL PROTECTED] ICQ #: 4663650
Re: Memory leak/server crashes
I'm looking for some help getting apache to run reliably. Apache 1.3.9 with mod_perl 1.21 and Perl 5.005_03 is running on a dual P500 with 1 Gb of RAM running Redhat 6.1. We run about 5 sites off the box, most of which are fairly high traffic, and use a lot of CGI and MySQL 3.22.25 is used with Apache::DBI. The major problem seems to be a memory leak of some sort, identical to that described in the "memory leak in mod_perl" thread on this list from October 1997 and the "httpd, mod_perl and memory consumption (long)" thread from July 1997. [snip] I too have had this problem and haven't found a suitable solution. In my case, though, I think the leaks are primarily do to old perl scripts being run under Registry and not bugs in mod_perl or perl. Well as I said, there is a lot of old code, which I guess could be the culprit. The first thing to do is to try to discover if the problem is a mod_perl problem or a bad script problem. If your server can handle it, you could try a binary search to find which (if any) scripts make the problem worse. Basically pick half your registry scripts and use mod_cgi. If leaks persist, you know that you have some problem scripts in the ones you didn't make mod_cgi. If leaks stop, then you know the problem scripts are in the ones you made mod_cgi. Repeat as necessary until you have narrowed it down to a single script. This is tedious though and may not be practical. Ok, i'll try this when I get time. Now, let's assume the problem is in fact in mod_perl or apache or perl itself. In this case I'm not sure what the best way to proceed is. I think mod_perl and perl have shown themselves to be pretty good about not leaking memory, as has apache. IMO it's much, much more likely a problem concerning Registry and impolite scripts that are misbehaving and leaving parts of themselves around. Yeah, i'm willing to believe my scripts could be a cause. Have you tried correlating the memory surges with any page accesses? That may help narrow down the culprit. I'm not really sure how I could go about doing that, any suggestions? :) -- James Furness [EMAIL PROTECTED] ICQ #: 4663650
Re: Memory leak/server crashes
Try using Apache::SizeLimit as a way of controlling your
processes. Sounds like a recursive page that performs infinite internal
requests.
--
Sean Chittenden [EMAIL PROTECTED]
fingerprint = 6988 8952 0030 D640 3138 C82F 0E9A DEF1 8F45 0466
My mother once said to me, "Elwood," (she always called me Elwood)
"Elwood, in this world you must be oh so smart or oh so pleasant."
For years I tried smart. I recommend pleasant.
-- Elwood P. Dowde, "Harvey"
On Sun, 9 Jan 2000, James Furness wrote:
Date: Sun, 9 Jan 2000 19:58:00 -
From: James Furness [EMAIL PROTECTED]
Reply-To: James Furness [EMAIL PROTECTED]
To: [EMAIL PROTECTED]
Subject: Memory leak/server crashes
I'm looking for some help getting apache to run reliably. Apache 1.3.9 with
mod_perl 1.21 and Perl 5.005_03 is running on a dual P500 with 1 Gb of RAM
running Redhat 6.1. We run about 5 sites off the box, most of which are
fairly high traffic, and use a lot of CGI and
MySQL 3.22.25 is used with Apache::DBI.
The major problem seems to be a memory leak of some sort, identical to that
described in the "memory leak in mod_perl" thread on this list from October
1997 and the "httpd, mod_perl and memory consumption (long)" thread from
July 1997.
The server runs normally for several hours, then suddenly a httpd process
starts growing exponentially, the swapfile usage grows massively and the
server starts to become sluggish (I assume due to disk thrashing caused by
the heavy swap usage). Usually when this started to happen I would log in
and use apachectl stop to shutdown the server, then type 'killall httpd'
several times till the processes finally died off, and then use apachectl
start to restart apache. If I was not around or did not catch this, the
server would eventually become unresponsive and lock up, requiring a manual
reboot by the datacentre staff. Messages such as "Out of memory" and
"Callback called exit" would appear in the error log as the server spiralled
down and MySQL would start to have trouble running.
To combat this, I created a script to monitor load and swapfile usage, and
restart apache as described above if load was above 7 and swapfile usage
above 150Mb. This script has kept the server online and we now have an
uptime of something like 22 days (previously no more than 1 day), but the
script is getting triggered several times a day and no more "Out of memory"
messages are appearing, but the situation is not ideal.
I have tried adding:
sub UNIVERSAL::AUTOLOAD {
my $class = shift;
Carp::cluck "$class can't \$UNIVERSAL::AUTOLOAD!\n";
}
As recommended by the developers guide, which flooded the error log with the
text below being printed roughly once a second in the error log:
-
Apache=SCALAR(0x830937c) can't $UNIVERSAL::AUTOLOAD!
Apache=SCALAR(0x8309364) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
DBI::DBI_tie=HASH(0x82dd16c) can't $UNIVERSAL::AUTOLOAD!
IO::Handle=IO(0x820aabc) can't $UNIVERSAL::AUTOLOAD!
--
I've pretty much exhausted any ways I can think of to trace this problem,
such as i've tried to eliminate memory leaks in code by removing some
scripts from mod_perl and running them under mod_cgi and i've tried tweaking
MaxRequestsPerChild both without any success.
One thing that was mentioned in a previous thread was that using 'exit'
could confuse perl, and exit() is used fairly heavily in the scripts since
most are converted to mod_perl from standard CGIs, but i'd prefer not to
have to remove these since the structure of the scripts is reliant on some
form of exit statement. Is there some alternative to exit()?
I've also had a look at some of the patches to Apache.pm and Apache.xs
suggested in the previous threads, and these seem to have been incorporated
into mod_perl 1.21.
Are there any other solutions I could try to this problem? Does anyone know
what might be causing this?
The second problem I have is when loading pages, usually CGI, but I think
this has happened on some static pages, what IE5 describes as "Server not
found or DNS error" is experienced. Originally I thought this was the server
hitting MaxClients (150) since it usually occurs at the same time as massive
surges of hits, and /server-status usually shows 150 httpd processes have
been spawned, however I increased MaxClients to 200 recently and the error
has continued to happen, even though /server-status doesn't show any more
than about 170 processes spawned. I have not ruled out DNS server troubles
or backbone problems (We've had a few routing troubles
Re: Memory leak/server crashes
Try using Apache::SizeLimit as a way of controlling your processes. Sounds like a recursive page that performs infinite internal requests. Ok, sounds like a good solution, but it still seems to me I should be eliminating the problem at the source. Any ideas as to how I could narrow down the location of whatever's causing the recursion? -- James Furness [EMAIL PROTECTED] ICQ #: 4663650
Re: Memory leak/server crashes
"James Furness" [EMAIL PROTECTED] writes: I'm looking for some help getting apache to run reliably. Apache 1.3.9 with mod_perl 1.21 and Perl 5.005_03 is running on a dual P500 with 1 Gb of RAM running Redhat 6.1. We run about 5 sites off the box, most of which are fairly high traffic, and use a lot of CGI and MySQL 3.22.25 is used with Apache::DBI. The major problem seems to be a memory leak of some sort, identical to that described in the "memory leak in mod_perl" thread on this list from October 1997 and the "httpd, mod_perl and memory consumption (long)" thread from July 1997. [snip] I too have had this problem and haven't found a suitable solution. In my case, though, I think the leaks are primarily do to old perl scripts being run under Registry and not bugs in mod_perl or perl. The first thing to do is to try to discover if the problem is a mod_perl problem or a bad script problem. If your server can handle it, you could try a binary search to find which (if any) scripts make the problem worse. Basically pick half your registry scripts and use mod_cgi. If leaks persist, you know that you have some problem scripts in the ones you didn't make mod_cgi. If leaks stop, then you know the problem scripts are in the ones you made mod_cgi. Repeat as necessary until you have narrowed it down to a single script. This is tedious though and may not be practical. Depending on how old the scripts are, I would check for non-closed filehandles, excessive global variables, not using strict, etc. perl-status is your friend (hopefully you have it enabled!) so you can see the namespaces of each httpd and see if you have any candidate variables, file handles, functions, etc that could be clogging memory. As a last resort, you could try Apache::SizeLimit to cap the size of each httpd daemon. This works reasonably well for us. Something to the effect: use Apache::SizeLimit; $Apache::SizeLimit::MAX_PROCESS_SIZE = 16384; $Apache::SizeLimit::CHECK_EVERY_N_REQUESTS = 3; should help cap your processes at 16meg each. Tweak as necessary. Read the perldoc for Apache::SizeLimit for all the info you need. Now, let's assume the problem is in fact in mod_perl or apache or perl itself. In this case I'm not sure what the best way to procede is. I think mod_perl and perl have shown themselves to be pretty good about not leaking memory, as has apache. IMO it's much, much more likely a problem concerning Registry and impolite scripts that are misbehaving and leaving parts of themselves around. Have you tried correlating the memory surges with any page accesses? That may help narrow down the culprit. Good luck! Chip -- Chip Turner [EMAIL PROTECTED] Programmer, ZFx, Inc. www.zfx.com PGP key available at wwwkeys.us.pgp.net
Re: Memory leak/server crashes
Yeah... two things I'd do: 1) Open two telnet sessions to the box. One for top that is monitoring processes for your web user (www typically) and is sorting by memory usage w/ a 1 second refresh. I'd change the size of the window and make it pretty short so that the refreshes happen quicker, but that depends on your connection speed. The second telnet window is a window that tails your access log (tail -f). It sounds boring, but by watching the two, you should have an idea as to when the problem happens. 2) Open up and hack Apache::SizeLimit and have it do a stack dump (Carp::croak) of what's going on... there may be some clue there. Solution #1 will probably be your best bet... Good luck (cool site too!). --SC -- Sean Chittenden [EMAIL PROTECTED] fingerprint = 6988 8952 0030 D640 3138 C82F 0E9A DEF1 8F45 0466 The faster I go, the behinder I get. -- Lewis Carroll On Sun, 9 Jan 2000, James Furness wrote: Date: Sun, 9 Jan 2000 21:47:03 - From: James Furness [EMAIL PROTECTED] Reply-To: James Furness [EMAIL PROTECTED] To: Sean Chittenden [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Subject: Re: Memory leak/server crashes Try using Apache::SizeLimit as a way of controlling your processes. Sounds like a recursive page that performs infinite internal requests. Ok, sounds like a good solution, but it still seems to me I should be eliminating the problem at the source. Any ideas as to how I could narrow down the location of whatever's causing the recursion? -- James Furness [EMAIL PROTECTED] ICQ #: 4663650
Re: Memory leak/server crashes
On Sun, 9 Jan 2000, Sean Chittenden wrote: Yeah... two things I'd do: 1) Open two telnet sessions to the box. One for top that is monitoring processes for your web user (www typically) and is sorting by memory usage w/ a 1 second refresh. I'd change the size of the window and make it pretty short so that the refreshes happen quicker, but that depends on your connection speed. The second telnet window is a window that tails your access log (tail -f). It sounds boring, but by watching the two, you should have an idea as to when the problem happens. Why, reinvent the wheel? I wrote Apache::VMonitor (grab from CPAN) that does all this and more (all but tail -f) I use it all the time, saves me a lot of time, since I don't have to telnet! 2) Open up and hack Apache::SizeLimit and have it do a stack dump (Carp::croak) of what's going on... there may be some clue there. Apache::GTopLimit is an advanced one :) (you are on Linux, right?) but Apache::SizeLimit is just file 3) try running in single mode with 'strace' (probably not a good idea for a production server), but you can still strace all the processes into a log file 4) Apache::Leak ? ___ Stas Bekmanmailto:[EMAIL PROTECTED] http://www.stason.org/stas Perl,CGI,Apache,Linux,Web,Java,PC http://www.stason.org/stas/TULARC perl.apache.orgmodperl.sourcegarden.org perlmonth.comperl.org single o- + single o-+ = singlesheavenhttp://www.singlesheaven.com
