Re: Memory Leaks
On Mon, 20 May 2002 16:23:40 -0500 Gregory Matthews [EMAIL PROTECTED] wrote: : Unfortunately, we only have one machine. If we did employ the cron job as : a clean-up utility once per day, wouldn't the potential impact of a site : being unavailable only be for a few seconds (until Apache restarted)? ... provided one is not using memory as unique storage for sessions or some other data maybe? -- fxn
Re: Memory Leaks
On Sun, 19 May 2002 23:34:24 -0400
Perrin Harkins [EMAIL PROTECTED] wrote:
: Leaks are caused by circular references, the string form of eval (at
: least it used to leak a little), nested closures (sometimes created
: accidentally with the Error module)
I am using the Error module in my current project, what kind of
constructs should one avoid? Is this safe?
my $um = UserManager-new;
# ...
try {
$um-write_user($user);
$um-dbh-commit;
} catch Exception::DB with {
my $e = shift;
debug Exception: $e;
$um-dbh-rollback;
};
-- fxn
Re: Memory Leaks
On Mon, 20 May 2002, F. Xavier Noria wrote:
On Sun, 19 May 2002 23:34:24 -0400
Perrin Harkins [EMAIL PROTECTED] wrote:
: Leaks are caused by circular references, the string form of eval (at
: least it used to leak a little), nested closures (sometimes created
: accidentally with the Error module)
I am using the Error module in my current project, what kind of
constructs should one avoid? Is this safe?
my $um = UserManager-new;
# ...
try {
$um-write_user($user);
$um-dbh-commit;
} catch Exception::DB with {
my $e = shift;
debug Exception: $e;
$um-dbh-rollback;
};
No. $um is caught in a closure, which could potentially leak.
--
!-- Matt --
:-Get a smart net/:-
Re: Memory Leaks
On Mon, 20 May 2002 10:15:02 +0100 (BST)
Matt Sergeant [EMAIL PROTECTED] wrote:
: my $um = UserManager-new;
: # ...
: try {
: $um-write_user($user);
: $um-dbh-commit;
: } catch Exception::DB with {
: my $e = shift;
: debug Exception: $e;
: $um-dbh-rollback;
: };
:
: No. $um is caught in a closure, which could potentially leak.
Wow, thank you, I have that pattern repeated in the code many times.
That is the way I would write that try/catch in Java, where you need to
have $um in the scope of the try and the catch blocks, what is the right
way to write that in Perl/Error.pm?
-- fxn
Re: Memory Leaks
On Mon, 20 May 2002, F. Xavier Noria wrote:
On Mon, 20 May 2002 10:15:02 +0100 (BST)
Matt Sergeant [EMAIL PROTECTED] wrote:
: my $um = UserManager-new;
: # ...
: try {
: $um-write_user($user);
:$um-dbh-commit;
: } catch Exception::DB with {
: my $e = shift;
: debug Exception: $e;
: $um-dbh-rollback;
: };
:
: No. $um is caught in a closure, which could potentially leak.
Wow, thank you, I have that pattern repeated in the code many times.
That is the way I would write that try/catch in Java, where you need to
have $um in the scope of the try and the catch blocks, what is the right
way to write that in Perl/Error.pm?
I gave up on Error.pm's try/catch syntax a long time ago - I think it's
hidden closure system combined with perl bugs is just too broken for
production use. Instead I use good old eval:
my $um = UserManager-new;
...
eval {
$um-write_user($user);
$um-dbh-commit;
};
if ($@ $@-isa('Exception::DB')) {
debug Exception: $@;
$um-dbh-rollback;
}
(note: if you expect all exceptions to be references like this, you had
better have a $SIG{__DIE__} handler installed to bless non-blessed
exceptions before re-throwing them - ask me if you need an example of
that)
--
!-- Matt --
:-Get a smart net/:-
Re: Memory Leaks
On Mon, 20 May 2002, Matt Sergeant wrote:
if ($@ $@-isa('Exception::DB')) {
debug Exception: $@;
$um-dbh-rollback;
}
(note: if you expect all exceptions to be references like this, you had
better have a $SIG{__DIE__} handler installed to bless non-blessed
exceptions before re-throwing them
Can't you just use UNIVERSAL's ISA method directly?
if (UNIVERSAL::isa($@,'Exception::DB')) {
This of course might fail if you got the string Exception::DB or
likewise back as an error message.
Alternativly, check if it's blessed
use Scalar::Util qw(blessed);
if (blessed($@) $@-isa('Exception::DB')) {
Later.
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t-Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t-Tgoto(cm,$_,$y). $w;select$k,$k,$k,.03}$y+=2}
Re: Memory Leaks
On Mon, 20 May 2002, Mark Fowler wrote:
On Mon, 20 May 2002, Matt Sergeant wrote:
if ($ $@-isa('Exception::DB')) {
debug Exception: $;
$um-dbh-rollback;
}
(note: if you expect all exceptions to be references like this, you had
better have a $SIG{__DIE__} handler installed to bless non-blessed
exceptions before re-throwing them
Can't you just use UNIVERSAL's ISA method directly?
if (UNIVERSAL::isa($,'Exception::DB')) {
This of course might fail if you got the string Exception::DB or
likewise back as an error message.
Alternativly, check if it's blessed
use Scalar::Util qw(blessed);
if (blessed($) $@-isa('Exception::DB')) {
Yeah, I know all the tricks. Ultimately it's a matter of how ugly you want
your code to get, and how many external modules you want to rely on (I
believe Scalar::Util is going to be part of 5.8 though).
--
!-- Matt --
:-Get a smart net/:-
Re: Memory Leaks
I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? Thanks. Gregory At 08:25 AM 5/20/2002 -0400, you wrote: It is more an issue of it being worth tracking down a small memory leak vs a large memory leak. Our software still has some very small leaks, on the order of 10kv every hour... it would probably take us a month to track down and solve all these problems. I find it easier to restart the web servers daily. We did have some enourmous leaks as well, based on circular reference, and those ate up 1 GB of memory in about 30 minutes... It took us about three weeks to find it. Gregory Matthews writes: So am I being overly paranoid concerning the leak potential of mod_perl programming? If I start with strict code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Keep in mind, although my application is not designed to launch the space shuttle, I do want it to be solid/stable/peformance-packed from the ground up. I will be also be using MySql with the Apache::DBI module. Thanks in advance. Gregory At 11:34 PM 5/19/2002 -0400, you wrote: I have a couple of questions regarding leaking memory in mod_perl: 1. What are the main culprits, in order of severity, of memory leaks, i.e.: a. global variables (NOT lexically scoped via my) b. ... c. ... 2. When writing code from scratch (a new application), what is the best way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like my $x = 7 if $y;. Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin -- C Wayne Huling [EMAIL PROTECTED]
Re: Memory Leaks
Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? There have been some discussion on the list about this in the past. The ideal situation is to have a cluster that you can do a rolling restart on without making the service totally unavailable. - Perrin
Re: Memory Leaks
Unfortunately, we only have one machine. If we did employ the cron job as a clean-up utility once per day, wouldn't the potential impact of a site being unavailable only be for a few seconds (until Apache restarted)? Gregory At 05:12 PM 5/20/2002 -0400, you wrote: Like another suggestion, we have a cluster of machines and roll the restarts every hour. Each machine is offset but 10 minutes. Gregory Matthews writes: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? Thanks. Gregory At 08:25 AM 5/20/2002 -0400, you wrote: It is more an issue of it being worth tracking down a small memory leak vs a large memory leak. Our software still has some very small leaks, on the order of 10kv every hour... it would probably take us a month to track down and solve all these problems. I find it easier to restart the web servers daily. We did have some enourmous leaks as well, based on circular reference, and those ate up 1 GB of memory in about 30 minutes... It took us about three weeks to find it. Gregory Matthews writes: So am I being overly paranoid concerning the leak potential of mod_perl programming? If I start with strict code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Keep in mind, although my application is not designed to launch the space shuttle, I do want it to be solid/stable/peformance-packed from the ground up. I will be also be using MySql with the Apache::DBI module. Thanks in advance. Gregory At 11:34 PM 5/19/2002 -0400, you wrote: I have a couple of questions regarding leaking memory in mod_perl: 1. What are the main culprits, in order of severity, of memory leaks, i.e.: a. global variables (NOT lexically scoped via my) b. ... c. ... 2. When writing code from scratch (a new application), what is the best way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like my $x = 7 if $y;. Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin -- C Wayne Huling [EMAIL PROTECTED] -- C Wayne Huling [EMAIL PROTECTED]
Re: Memory Leaks
At 23:23 20.05.2002, Gregory Matthews wrote: Unfortunately, we only have one machine. If we did employ the cron job as a clean-up utility once per day, wouldn't the potential impact of a site being unavailable only be for a few seconds (until Apache restarted)? And if something goes wrong? You'd be having a server offline with noone knowing about it. At 05:12 PM 5/20/2002 -0400, you wrote: Like another suggestion, we have a cluster of machines and roll the restarts every hour. Each machine is offset but 10 minutes. Gregory Matthews writes: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? Thanks. Gregory At 08:25 AM 5/20/2002 -0400, you wrote: It is more an issue of it being worth tracking down a small memory leak vs a large memory leak. Our software still has some very small leaks, on the order of 10kv every hour... it would probably take us a month to track down and solve all these problems. I find it easier to restart the web servers daily. We did have some enourmous leaks as well, based on circular reference, and those ate up 1 GB of memory in about 30 minutes... It took us about three weeks to find it. Gregory Matthews writes: So am I being overly paranoid concerning the leak potential of mod_perl programming? If I start with strict code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Keep in mind, although my application is not designed to launch the space shuttle, I do want it to be solid/stable/peformance-packed from the ground up. I will be also be using MySql with the Apache::DBI module. Thanks in advance. Gregory At 11:34 PM 5/19/2002 -0400, you wrote: I have a couple of questions regarding leaking memory in mod_perl: 1. What are the main culprits, in order of severity, of memory leaks, i.e.: a. global variables (NOT lexically scoped via my) b. ... c. ... 2. When writing code from scratch (a new application), what is the best way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like my $x = 7 if $y;. Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin -- C Wayne Huling [EMAIL PROTECTED] -- C Wayne Huling [EMAIL PROTECTED] -- Per Einar Ellefsen [EMAIL PROTECTED]
Re: Memory Leaks
If you don't want to restart the server then don't do this instead, it should help prevent small leaks from being a problem. http://httpd.apache.org/docs-2.0/mod/mpm_common.html#maxrequestsperchild - Original Message - From: Per Einar Ellefsen [EMAIL PROTECTED] To: Gregory Matthews [EMAIL PROTECTED] Cc: [EMAIL PROTECTED] Sent: Monday, May 20, 2002 3:30 PM Subject: Re: Memory Leaks At 23:23 20.05.2002, Gregory Matthews wrote: Unfortunately, we only have one machine. If we did employ the cron job as a clean-up utility once per day, wouldn't the potential impact of a site being unavailable only be for a few seconds (until Apache restarted)? And if something goes wrong? You'd be having a server offline with noone knowing about it. At 05:12 PM 5/20/2002 -0400, you wrote: Like another suggestion, we have a cluster of machines and roll the restarts every hour. Each machine is offset but 10 minutes. Gregory Matthews writes: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? Thanks. Gregory At 08:25 AM 5/20/2002 -0400, you wrote: It is more an issue of it being worth tracking down a small memory leak vs a large memory leak. Our software still has some very small leaks, on the order of 10kv every hour... it would probably take us a month to track down and solve all these problems. I find it easier to restart the web servers daily. We did have some enourmous leaks as well, based on circular reference, and those ate up 1 GB of memory in about 30 minutes... It took us about three weeks to find it. Gregory Matthews writes: So am I being overly paranoid concerning the leak potential of mod_perl programming? If I start with strict code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Keep in mind, although my application is not designed to launch the space shuttle, I do want it to be solid/stable/peformance-packed from the ground up. I will be also be using MySql with the Apache::DBI module. Thanks in advance. Gregory At 11:34 PM 5/19/2002 -0400, you wrote: I have a couple of questions regarding leaking memory in mod_perl: 1. What are the main culprits, in order of severity, of memory leaks, i.e.: a. global variables (NOT lexically scoped via my) b. ... c. ... 2. When writing code from scratch (a new application), what is the best way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like my $x = 7 if $y;. Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin -- C Wayne Huling [EMAIL PROTECTED] -- C Wayne Huling [EMAIL PROTECTED] -- Per Einar Ellefsen [EMAIL PROTECTED]
Re: Memory Leaks
On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of reliable computing. Matt.
Re: Memory Leaks
I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. -Allen On Mon, 20 May 2002, Matt Sergeant wrote: On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of reliable computing. Matt.
Re: Memory Leaks
Per Einar Ellefsen wrote: And if something goes wrong? You'd be having a server offline with noone knowing about it. You can easilly set up mon (http://www.kernel.org/software/mon/) to page you if the server doesn't come back up within a certain amount of time. - Perrin
Re: Memory Leaks
At 23:54 20.05.2002, Allen Day wrote: I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. Hmm, if you are serving big files off of mod_perl, memory leaks are the least of your problems :) That doesn't apply to Apache::MP3 of course, for which it's normal, but in no case should your mod_perl server be serving your big files. On Mon, 20 May 2002, Matt Sergeant wrote: On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of reliable computing. -- Per Einar Ellefsen [EMAIL PROTECTED]
Re: Memory Leaks
Jason wrote: If you don't want to restart the server then don't do this instead, it should help prevent small leaks from being a problem. http://httpd.apache.org/docs-2.0/mod/mpm_common.html#maxrequestsperchild Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since it results in fewer unnecessary restarts. However, it's still a good idea to restart periodically, because some of the shared memory seems to become unshared over time no matter what you do, and restarting fixes that. - Perrin
Re: Memory Leaks
I mentioned the connection closing as a drawback of restarting the server -- it was slightly OT for the thread. Yes, it is a subclass of Apache::MP3 that can stream video and audio. There is an old version called Apache::Jukebox in the Apache::MP3 CVS at namp.sourceforge.net in case anyone is interested. -Allen On Mon, 20 May 2002, Per Einar Ellefsen wrote: At 23:54 20.05.2002, Allen Day wrote: I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. Hmm, if you are serving big files off of mod_perl, memory leaks are the least of your problems :) That doesn't apply to Apache::MP3 of course, for which it's normal, but in no case should your mod_perl server be serving your big files. On Mon, 20 May 2002, Matt Sergeant wrote: On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of reliable computing.
Re: Memory Leaks
On Mon, 20 May 2002, Perrin Harkins wrote: Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since it results in fewer unnecessary restarts. However, it's still a good idea to restart periodically, because some of the shared memory seems to become unshared over time no matter what you do, and restarting fixes that. that reminds me, i wrote a c version of Apache::GTopLimit 2+ years ago (at somepoint in the 5 months i worked at backflip.com), but never released it. if somebody wants to maintain/release it, the package is here: http://perl.apache.org/~dougm/mod_gtop-0.02.tar.gz
Re: Memory Leaks
I'd like to try to disagree here. I have built several file-related webapps where I have implemented virtual filesystems which require special perl modules to access the files at all. mod_perl takes care of serving the requests. If I need a restart, then I can still safely use graceful. Admittedly there are times when something could very well get screwed up, but my solution to that is to develop a better front-end server with it's own buffer so that the back-end can swiftly serve the files leaving much more idle time (in comparison to directly connecting remote client to fileserver) for backend restarts if needed. Issac Per Einar Ellefsen wrote: At 23:54 20.05.2002, Allen Day wrote: I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. Hmm, if you are serving big files off of mod_perl, memory leaks are the least of your problems :) That doesn't apply to Apache::MP3 of course, for which it's normal, but in no case should your mod_perl server be serving your big files. On Mon, 20 May 2002, Matt Sergeant wrote: On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of reliable computing.
Re: Memory Leaks
At 00:45 21.05.2002, Issac Goldstand wrote: I'd like to try to disagree here. I have built several file-related webapps where I have implemented virtual filesystems which require special perl modules to access the files at all. mod_perl takes care of serving the requests. If I need a restart, then I can still safely use graceful. Admittedly there are times when something could very well get screwed up, but my solution to that is to develop a better front-end server with it's own buffer so that the back-end can swiftly serve the files leaving much more idle time (in comparison to directly connecting remote client to fileserver) for backend restarts if needed. In the case that you need advanced logic such as that, I clearly agree with both you and Allen. And a proxy server is very needed in such a case :) Per Einar Ellefsen wrote: At 23:54 20.05.2002, Allen Day wrote: I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. Hmm, if you are serving big files off of mod_perl, memory leaks are the least of your problems :) That doesn't apply to Apache::MP3 of course, for which it's normal, but in no case should your mod_perl server be serving your big files. On Mon, 20 May 2002, Matt Sergeant wrote: On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: I too thought of setting a cron job to restart the server once per day in order to keep the memory fresh. In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of reliable computing. -- Per Einar Ellefsen [EMAIL PROTECTED]
Re: Memory Leaks
Per Einar Ellefsen wrote: At 23:54 20.05.2002, Allen Day wrote: I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. Hmm, if you are serving big files off of mod_perl, memory leaks are the least of your problems :) Well, you can serve big files without reading them into a memory at once. Why there would be memory leaks? That doesn't apply to Apache::MP3 of course, for which it's normal, but in no case should your mod_perl server be serving your big files. The reason for not serving big files with mod_perl, is that you don't want to tie heavy and snappy mod_perl servers to wait indefinitely for the client to grab their data. If you have plenty of memory or you have just a few clients (intranet?) that's just fine. This is all discussed here: http://perl.apache.org/release/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory Leaks
Gregory Matthews wrote: Does using the Apache::GTopLimit module have the same net effect as restarting the server itself by simply killing off the actual processes which are growing beyond the set threshold, and thereby causing new processes to be born? It's not the exactly the same, since it won't pick up any changes in Perl modules on the disk. And that's one of the main reasons for doing restarts. Otherwise yes. __ Stas BekmanJAm_pH -- Just Another mod_perl Hacker http://stason.org/ mod_perl Guide --- http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory Leaks
I have a couple of questions regarding leaking memory in mod_perl: 1. What are the main culprits, in order of severity, of memory leaks, i.e.: a. global variables (NOT lexically scoped via my) b. ... c. ... 2. When writing code from scratch (a new application), what is the best way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like my $x = 7 if $y;. Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin
Re: Memory Leaks
So am I being overly paranoid concerning the leak potential of mod_perl programming? If I start with strict code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Keep in mind, although my application is not designed to launch the space shuttle, I do want it to be solid/stable/peformance-packed from the ground up. I will be also be using MySql with the Apache::DBI module. Thanks in advance. Gregory At 11:34 PM 5/19/2002 -0400, you wrote: I have a couple of questions regarding leaking memory in mod_perl: 1. What are the main culprits, in order of severity, of memory leaks, i.e.: a. global variables (NOT lexically scoped via my) b. ... c. ... 2. When writing code from scratch (a new application), what is the best way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like my $x = 7 if $y;. Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin
Re: Memory Leaks
Not long ago, Gregory Matthews proclaimed... So am I being overly paranoid concerning the leak potential of mod_perl programming? No... But once you do it right it comes natural. The thing that killed me when I first started doing mod_perl development was code that pushed items onto lists in Apache::Registry scripts. I wasn't initializing those list variables and before you knew it, lists were getting populated with values in duplicate, triplicate, and so on. Nothing has helped me become a better Perl programming than mod_perl. :) -=Fozz -- Doran L. Barton [EMAIL PROTECTED] - Chief Super Hero - Iodynamics LLC http://www.iodynamics.com/ - Linux solutions and dynamic websites Dog for sale: eats anything and is fond of children. -- Classified ad in newspaper
Re: Memory Leaks
So am I being overly paranoid concerning the leak potential of mod_perl programming? No, memory management is very important with mod_perl. If I start with strict code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Using strict is an absolute must for many reasons. It won't prevent you from using up tons of memory though. You have to become tuned in to the things that use up memory so that you can avoid them. Start with the Guide. - Perrin
Re: Memory Leaks?
At 15:50 15/03/2001 -0300, Jason Leidigh wrote:
I was able to clean up a number of errors which seemed as
though they were indeed causing leaks. For example:
$regex = qr'xx?'i;
Causes the following error:
(?i-xsm:xx?) can't `Regexp::DESTROY'
AUTOLOADs will catch DESTROYs, the latter being called on every object
destruction (I never noticed that they were called on regexen, that's
nice). Chances are, unless you clearly see a memory leak with DESTROYs
(which are expected not to be found most of the time) then your AUTOLOAD
should probably explicitly ignore them.
Another solution is:
sub Regexp::DESTROY {}
sub Apache::DESTROY {}
etc...
-- robin b.
"Science is like sex: sometimes something useful comes out, but that is not
the reason we are doing it." -- R. Feynman
Re: Memory leaks?
On Tue, 2 Jan 2001, Per 'stripe' Moeller wrote: Hi, I have created a highly configurable content management system in modperl to build websites, which consists of several modules, some preload when apache initiates, some when a request initiates and some when needed. I had the opportunity to test the system with 2000 lan connected users to do some performance checking, and found out that for each request the server received, the child process which handled the request used a little more memory each time. In this test I had around 500.000 hits pr day, and apache/modperl used up all the 1Gb of memory the server had in a very short time. I checked and rechecked my code and could not find any kind of memory leak in it, so finally I had to configure the Apache to kill child processes after answering 100 requests, otherwise they would consume too much memory. Have anybody else experienced this kind of problem? Lots of people. Memory leaks are very easy to introduce, and very hard to find. The most likely causes are closures, circular references, and bad modules. Not necessarily in that order :-) Another problem I encountered... when doing HTTP upload's, apache/modperl uses 7-8 times the size of the uploaded file, of memory. If I uploaded a 10Mb file, the server would typically use up to 170Mb for the child answering the request, and the child would not free all the memory used when the request had been handled. Can this really be true? Yes its true. Try the latest Apache::Request test release - it should fix this problem. -- Matt/ /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: memory leaks redux
On Sat Jan 29 13:11:25 2000 + Mike Whitaker wrote: [EMAIL PROTECTED] (Doug MacEachern) wrote: there are hints in the SUPPORT doc on how to debug such problems. there was also several "Hanging process" threads in the past weeks with more tips, search in the archives for keywords gdb, .gdbinit, curinfo if you can get more insight from those tips, we can help more. I have also seen (and reported via the Debian bugs system) a problem which I think has been observed before where HUP'ing the root httpd causes it to reload every darn PerlModule directive, and bloat accordingly (with our server, that's 3M per SIGHUP, which makes log rotation somewhat painful (when you get 3 million hits in 8 hours, you rotate those logs pretty fast!)). You can log to pipe. -- Ričardas Čepas ~~~ ~~ PGP signature
Re: Memory Leaks in mod_perl
On Wed, Nov 03, 1999 at 10:40:08PM -0500, David Huggins-Daines wrote: I have more or less the same problem as Ben here. mod_perl appears to leak memory on SIGHUP proportionately to the number of extra Perl modules loaded into the interpreter I think to some extent individual modules are to blame (though again - this might be something in the Apache core that they use). When I did some checking I found that PerlModule Apache didn't seem to leak any more memory than just having a perl section (though this may be because it's implicitly loaded by having a Perl section - anyone confirm this?) Apache::Status on the other hand leaked around 936K. Apache::DBI (not PerlSections as I stated in my last post) leaks 404K, though 360K of this seems to be DBI itself. It seems to be the native code Perl Modules that cause the most trouble. Maybe a short term solution to this would be to allow a directive which destroys the Perl interpreter on restarts. I have experienced this with: Apache/1.3.6, mod_perl 1.21, Red Hat 5.2/i386 (perl 5.004_05 and 5.005_03) Well I guess it's not something bizarre about the way tha Debian compiled it then. I have not been able to check earlier versions of mod_perl because I can't seem to find them on CPAN :) Please also note Debian bug reports #34947 and #41946. (http://bugs.debian.org/34947, http://bugs.debian.org/41946) Bug 34947 might be a different problem. But if it is, the poster stated that statically linked 1.3.3/1.16_02 worked well for him. I'll see if I can track down some earlier versions and build them like this. Ben -- +-Ben Bell - "A song, a perl script and the occasional silly sig.-+ /// email: [EMAIL PROTECTED]www: http://www.deus.net/~bjb/ bjbDon't try to drive me crazy... \_/...I'm close enough to walk.
Re: Memory Leaks?
sounds like you have PerlFreshRestart On, try turning it Off. scan the archives for more info. On Fri, 29 Oct 1999, Ben Bell wrote: Hi, I'm using the Debian package of mod_perl (1.21) and apache 1.3.9 and I've noticed quite nasty memory leaks on server restart. I've noticed unresolved bug reports on the Debian pages about this. Is it a known issue with this version? The leak is ca. 2MB each restart (or graceful) with my startup script enabled (which queries a database, and uses the following modules: Apache Apache::PerlSections Apache::DBI Data::Dumper Carp VI::Utils All my vars are declared as "my". When I disable all Perl stuff (my startup script, the perl sections etc) I still see a memory leak, albeit a smaller one. Can anyone shed any light on this? Cheers, Ben -- +-Ben Bell - "A song, a perl script and the occasional silly sig.-+ /// email: [EMAIL PROTECTED]www: http://www.deus.net/~bjb/ bjbDon't try to drive me crazy... \_/...I'm close enough to walk.
