Re: Memory Leaks
On Mon, 20 May 2002 16:23:40 -0500 Gregory Matthews <[EMAIL PROTECTED]> wrote: : Unfortunately, we only have one machine. If we did employ the cron job as : a clean-up utility once per day, wouldn't the potential impact of a site : being unavailable only be for a few seconds (until Apache restarted)? ... provided one is not using memory as unique storage for sessions or some other data maybe? -- fxn
Re: Memory Leaks
Gregory Matthews wrote: > Does using the Apache::GTopLimit module have the same net effect as > restarting the server itself by simply killing off the actual processes > which are growing beyond the set threshold, and thereby causing new > processes to be born? It's not the exactly the same, since it won't pick up any changes in Perl modules on the disk. And that's one of the main reasons for doing restarts. Otherwise yes. __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory Leaks
Per Einar Ellefsen wrote: > At 23:54 20.05.2002, Allen Day wrote: > >> I've noticed that if I restart apache while I'm in the middle of a >> download (MP3 stream), after the buffer in my MP3 player runs out, it >> skips to the next track -- presumably because the connection was closed. >> >> This might cause a problem for you if your users are downloading big >> files. They might have to restart from the beginning if they didn't >> cache >> the partial download somewhere. > > > Hmm, if you are serving big files off of mod_perl, memory leaks are the > least of your problems :) Well, you can serve big files without reading them into a memory at once. Why there would be memory leaks? > That doesn't apply to Apache::MP3 of course, > for which it's normal, but in no case should your mod_perl server be > serving your big files. The reason for not serving big files with mod_perl, is that you don't want to tie heavy and snappy mod_perl servers to wait indefinitely for the client to grab their data. If you have plenty of memory or you have just a few clients (intranet?) that's just fine. This is all discussed here: http://perl.apache.org/release/docs/1.0/guide/strategy.html#Adding_a_Proxy_Server_in_http_Accelerator_Mode __ Stas BekmanJAm_pH --> Just Another mod_perl Hacker http://stason.org/ mod_perl Guide ---> http://perl.apache.org mailto:[EMAIL PROTECTED] http://use.perl.org http://apacheweek.com http://modperlbook.org http://apache.org http://ticketmaster.com
Re: Memory Leaks
At 00:45 21.05.2002, Issac Goldstand wrote: >I'd like to try to disagree here. I have built several file-related >webapps where I have implemented virtual filesystems which require special >perl modules to access the files at all. mod_perl takes care of serving >the requests. If I need a restart, then I can still safely use >graceful. Admittedly there are times when something could very well get >screwed up, but my solution to that is to develop a better front-end >server with it's own buffer so that the back-end can swiftly serve the >files leaving much more idle time (in comparison to directly connecting >remote client to fileserver) for backend restarts if needed. In the case that you need advanced logic such as that, I clearly agree with both you and Allen. And a proxy server is very needed in such a case :) >Per Einar Ellefsen wrote: > >>At 23:54 20.05.2002, Allen Day wrote: >> >>>I've noticed that if I restart apache while I'm in the middle of a >>>download (MP3 stream), after the buffer in my MP3 player runs out, it >>>skips to the next track -- presumably because the connection was closed. >>> >>>This might cause a problem for you if your users are downloading big >>>files. They might have to restart from the beginning if they didn't cache >>>the partial download somewhere. >> >> >>Hmm, if you are serving big files off of mod_perl, memory leaks are the >>least of your problems :) That doesn't apply to Apache::MP3 of course, >>for which it's normal, but in no case should your mod_perl server be >>serving your big files. >> >>>On Mon, 20 May 2002, Matt Sergeant wrote: >>> >>> > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: >>> > > I too thought of setting a cron job to restart the server once per >>> day in >>> > > order to keep the memory "fresh". >>> > > >>> > > In a production environment, are there any downsides to doing this, >>> i.e., >>> > > server inaccessibility, etc..? >>> > >>> > It's very rare to have a site that can't cope with just a few seconds >>> > downtime. Most users won't even notice, save for some slight delay in >>> getting >>> > their request through. Users tend to be pretty used to trying again >>> in this >>> > world of "reliable" computing. >> >>-- >>Per Einar Ellefsen >>[EMAIL PROTECTED]
Re: Memory Leaks
Does using the Apache::GTopLimit module have the same net effect as restarting the server itself by simply killing off the actual processes which are growing beyond the set threshold, and thereby causing new processes to be born? Gregory At 03:27 PM 5/20/2002 -0700, you wrote: >On Mon, 20 May 2002, Perrin Harkins wrote: > > > Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since > > it results in fewer unnecessary restarts. However, it's still a good > > idea to restart periodically, because some of the shared memory seems to > > become unshared over time no matter what you do, and restarting fixes that. > >that reminds me, i wrote a c version of Apache::GTopLimit 2+ years ago (at >somepoint in the 5 months i worked at backflip.com), but never released >it. if somebody wants to maintain/release it, the package is here: >http://perl.apache.org/~dougm/mod_gtop-0.02.tar.gz
Re: Memory Leaks
I'd like to try to disagree here. I have built several file-related webapps where I have implemented virtual filesystems which require special perl modules to access the files at all. mod_perl takes care of serving the requests. If I need a restart, then I can still safely use graceful. Admittedly there are times when something could very well get screwed up, but my solution to that is to develop a better front-end server with it's own buffer so that the back-end can swiftly serve the files leaving much more idle time (in comparison to directly connecting remote client to fileserver) for backend restarts if needed. Issac Per Einar Ellefsen wrote: > At 23:54 20.05.2002, Allen Day wrote: > >> I've noticed that if I restart apache while I'm in the middle of a >> download (MP3 stream), after the buffer in my MP3 player runs out, it >> skips to the next track -- presumably because the connection was closed. >> >> This might cause a problem for you if your users are downloading big >> files. They might have to restart from the beginning if they didn't >> cache >> the partial download somewhere. > > > Hmm, if you are serving big files off of mod_perl, memory leaks are > the least of your problems :) That doesn't apply to Apache::MP3 of > course, for which it's normal, but in no case should your mod_perl > server be serving your big files. > >> On Mon, 20 May 2002, Matt Sergeant wrote: >> >> > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: >> > > I too thought of setting a cron job to restart the server once >> per day in >> > > order to keep the memory "fresh". >> > > >> > > In a production environment, are there any downsides to doing >> this, i.e., >> > > server inaccessibility, etc..? >> > >> > It's very rare to have a site that can't cope with just a few seconds >> > downtime. Most users won't even notice, save for some slight delay >> in getting >> > their request through. Users tend to be pretty used to trying again >> in this >> > world of "reliable" computing. > >
Re: Memory Leaks
On Mon, 20 May 2002, Perrin Harkins wrote: > Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since > it results in fewer unnecessary restarts. However, it's still a good > idea to restart periodically, because some of the shared memory seems to > become unshared over time no matter what you do, and restarting fixes that. that reminds me, i wrote a c version of Apache::GTopLimit 2+ years ago (at somepoint in the 5 months i worked at backflip.com), but never released it. if somebody wants to maintain/release it, the package is here: http://perl.apache.org/~dougm/mod_gtop-0.02.tar.gz
Re: Memory Leaks
I mentioned the connection closing as a drawback of restarting the server -- it was slightly OT for the thread. Yes, it is a subclass of Apache::MP3 that can stream video and audio. There is an old version called Apache::Jukebox in the Apache::MP3 CVS at namp.sourceforge.net in case anyone is interested. -Allen On Mon, 20 May 2002, Per Einar Ellefsen wrote: > At 23:54 20.05.2002, Allen Day wrote: > >I've noticed that if I restart apache while I'm in the middle of a > >download (MP3 stream), after the buffer in my MP3 player runs out, it > >skips to the next track -- presumably because the connection was closed. > > > >This might cause a problem for you if your users are downloading big > >files. They might have to restart from the beginning if they didn't cache > >the partial download somewhere. > > Hmm, if you are serving big files off of mod_perl, memory leaks are the > least of your problems :) That doesn't apply to Apache::MP3 of course, for > which it's normal, but in no case should your mod_perl server be serving > your big files. > > >On Mon, 20 May 2002, Matt Sergeant wrote: > > > > > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: > > > > I too thought of setting a cron job to restart the server once per day in > > > > order to keep the memory "fresh". > > > > > > > > In a production environment, are there any downsides to doing this, i.e., > > > > server inaccessibility, etc..? > > > > > > It's very rare to have a site that can't cope with just a few seconds > > > downtime. Most users won't even notice, save for some slight delay in > > getting > > > their request through. Users tend to be pretty used to trying again in this > > > world of "reliable" computing. > >
Re: Memory Leaks
Jason wrote: > If you don't want to restart the server then don't do this instead, it should >help prevent small leaks from being a problem. > http://httpd.apache.org/docs-2.0/mod/mpm_common.html#maxrequestsperchild Apache::SizeLimit or Apache::GTopLimit is a better way to do it, since it results in fewer unnecessary restarts. However, it's still a good idea to restart periodically, because some of the shared memory seems to become unshared over time no matter what you do, and restarting fixes that. - Perrin
Re: Memory Leaks
At 23:54 20.05.2002, Allen Day wrote: >I've noticed that if I restart apache while I'm in the middle of a >download (MP3 stream), after the buffer in my MP3 player runs out, it >skips to the next track -- presumably because the connection was closed. > >This might cause a problem for you if your users are downloading big >files. They might have to restart from the beginning if they didn't cache >the partial download somewhere. Hmm, if you are serving big files off of mod_perl, memory leaks are the least of your problems :) That doesn't apply to Apache::MP3 of course, for which it's normal, but in no case should your mod_perl server be serving your big files. >On Mon, 20 May 2002, Matt Sergeant wrote: > > > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: > > > I too thought of setting a cron job to restart the server once per day in > > > order to keep the memory "fresh". > > > > > > In a production environment, are there any downsides to doing this, i.e., > > > server inaccessibility, etc..? > > > > It's very rare to have a site that can't cope with just a few seconds > > downtime. Most users won't even notice, save for some slight delay in > getting > > their request through. Users tend to be pretty used to trying again in this > > world of "reliable" computing. -- Per Einar Ellefsen [EMAIL PROTECTED]
Re: Memory Leaks
Per Einar Ellefsen wrote: > And if something goes wrong? You'd be having a server offline with noone > knowing about it. You can easilly set up mon (http://www.kernel.org/software/mon/) to page you if the server doesn't come back up within a certain amount of time. - Perrin
Re: Memory Leaks
I've noticed that if I restart apache while I'm in the middle of a download (MP3 stream), after the buffer in my MP3 player runs out, it skips to the next track -- presumably because the connection was closed. This might cause a problem for you if your users are downloading big files. They might have to restart from the beginning if they didn't cache the partial download somewhere. -Allen On Mon, 20 May 2002, Matt Sergeant wrote: > On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: > > I too thought of setting a cron job to restart the server once per day in > > order to keep the memory "fresh". > > > > In a production environment, are there any downsides to doing this, i.e., > > server inaccessibility, etc..? > > It's very rare to have a site that can't cope with just a few seconds > downtime. Most users won't even notice, save for some slight delay in getting > their request through. Users tend to be pretty used to trying again in this > world of "reliable" computing. > > Matt. >
Re: Memory Leaks
On Monday 20 May 2002 9:30 pm, Gregory Matthews wrote: > I too thought of setting a cron job to restart the server once per day in > order to keep the memory "fresh". > > In a production environment, are there any downsides to doing this, i.e., > server inaccessibility, etc..? It's very rare to have a site that can't cope with just a few seconds downtime. Most users won't even notice, save for some slight delay in getting their request through. Users tend to be pretty used to trying again in this world of "reliable" computing. Matt.
Re: Memory Leaks
If you don't want to restart the server then don't do this instead, it should help prevent small leaks from being a problem. http://httpd.apache.org/docs-2.0/mod/mpm_common.html#maxrequestsperchild - Original Message - From: "Per Einar Ellefsen" <[EMAIL PROTECTED]> To: "Gregory Matthews" <[EMAIL PROTECTED]> Cc: <[EMAIL PROTECTED]> Sent: Monday, May 20, 2002 3:30 PM Subject: Re: Memory Leaks > At 23:23 20.05.2002, Gregory Matthews wrote: > >Unfortunately, we only have one machine. If we did employ the cron job as > >a clean-up utility once per day, wouldn't the potential impact of a site > >being unavailable only be for a few seconds (until Apache restarted)? > > And if something goes wrong? You'd be having a server offline with noone > knowing about it. > > >At 05:12 PM 5/20/2002 -0400, you wrote: > > > >>Like another suggestion, we have a cluster of machines and roll the > >>restarts every hour. Each machine is offset but 10 minutes. > >> > >>Gregory Matthews writes: > >> > I too thought of setting a cron job to restart the server once per day in > >> > order to keep the memory "fresh". > >> > > >> > In a production environment, are there any downsides to doing this, i.e., > >> > server inaccessibility, etc..? > >> > > >> > Thanks. > >> > > >> > Gregory > >> > > >> > At 08:25 AM 5/20/2002 -0400, you wrote: > >> > > >> > >It is more an issue of it being worth tracking down a small memory > >> > >leak vs a large memory leak. Our software still has some very small > >> > >leaks, on the order of 10kv every hour... it would probably take us a > >> > >month to track down and solve all these problems. I find it easier to > >> > >restart the web servers daily. > >> > > > >> > >We did have some enourmous leaks as well, based on circular reference, > >> > >and those ate up 1 GB of memory in about 30 minutes... It took us > >> > >about three weeks to find it. > >> > > > >> > >Gregory Matthews writes: > >> > > > So am I being overly paranoid concerning the "leak" potential of > >> mod_perl > >> > > > programming? > >> > > > > >> > > > If I start with "strict" code to begin with and try my best to > >> stay away > >> > > > from the problems you mentioned, then any potential memory leak/drain > >> > > > issues will be avoided? > >> > > > > >> > > > Keep in mind, although my application is not designed to launch > >> the space > >> > > > shuttle, I do want it to be solid/stable/peformance-packed from the > >> > > ground up. > >> > > > > >> > > > I will be also be using MySql with the Apache::DBI module. > >> > > > > >> > > > Thanks in advance. > >> > > > > >> > > > Gregory > >> > > > > >> > > > > >> > > > At 11:34 PM 5/19/2002 -0400, you wrote: > >> > > > > > I have a couple of questions regarding leaking memory in mod_perl: > >> > > > > > > >> > > > > > 1. What are the main culprits, in order of severity, of > >> memory leaks, > >> > > > >i.e.: > >> > > > > > > >> > > > > > a. global variables (NOT lexically scoped via my) > >> > > > > > b. ... > >> > > > > > c. ... > >> > > > > > > >> > > > > > 2. When writing code from scratch (a new application), what > >> is the > >> > > > >best > >> > > > > > way to avoid creating leaks to begin with, i.e., use strict;, > >> PerlWarn > >> > > > >On, > >> > > > > > etc.. ? > >> > > > > > >> > > > >There are actually not very many ways you can leak memory in Perl > >> (and > >> > > > >thus mod_perl). Most people confuse memory growth with memory > >> leakage. > >> > > > >If you want to know how to avoid memory growth, look at the > >> performance > >> > > > >tuning stuff in the Guide, like passing references, avoiding > >> slurping of > >> > > > >large files, controlling the buffering of DBI result sets, etc. > >> > > > > > >> > > > >Leaks are caused by circular references, the string form of eval (at > >> > > > >least it used to leak a little), nested closures (sometimes created > >> > > > >accidentally with the Error module), and one or two obscure syntax > >> > > > >problems. I think one of them involved code like "my $x = 7 if $y;". > >> > > > >Matt Sergeant got bitten by this in the early stages of AxKit > >> > > > >development, and the details are in the mailing list archive. > >> > > > > > >> > > > >Global variables by themselves are not a source of leaks or > >> growth. If > >> > > > >you slurp a large file into a global, your process will grow, but the > >> > > > >same is true for a lexical. > >> > > > > > >> > > > >- Perrin > >> > > > > >> > > > > >> > > > >> > >-- > >> > >C Wayne Huling <[EMAIL PROTECTED]> > >> > > >> > > >> > >>-- > >>C Wayne Huling <[EMAIL PROTECTED]> > > > > > > -- > Per Einar Ellefsen > [EMAIL PROTECTED] >
Re: Memory Leaks
At 23:23 20.05.2002, Gregory Matthews wrote: >Unfortunately, we only have one machine. If we did employ the cron job as >a clean-up utility once per day, wouldn't the potential impact of a site >being unavailable only be for a few seconds (until Apache restarted)? And if something goes wrong? You'd be having a server offline with noone knowing about it. >At 05:12 PM 5/20/2002 -0400, you wrote: > >>Like another suggestion, we have a cluster of machines and roll the >>restarts every hour. Each machine is offset but 10 minutes. >> >>Gregory Matthews writes: >> > I too thought of setting a cron job to restart the server once per day in >> > order to keep the memory "fresh". >> > >> > In a production environment, are there any downsides to doing this, i.e., >> > server inaccessibility, etc..? >> > >> > Thanks. >> > >> > Gregory >> > >> > At 08:25 AM 5/20/2002 -0400, you wrote: >> > >> > >It is more an issue of it being worth tracking down a small memory >> > >leak vs a large memory leak. Our software still has some very small >> > >leaks, on the order of 10kv every hour... it would probably take us a >> > >month to track down and solve all these problems. I find it easier to >> > >restart the web servers daily. >> > > >> > >We did have some enourmous leaks as well, based on circular reference, >> > >and those ate up 1 GB of memory in about 30 minutes... It took us >> > >about three weeks to find it. >> > > >> > >Gregory Matthews writes: >> > > > So am I being overly paranoid concerning the "leak" potential of >> mod_perl >> > > > programming? >> > > > >> > > > If I start with "strict" code to begin with and try my best to >> stay away >> > > > from the problems you mentioned, then any potential memory leak/drain >> > > > issues will be avoided? >> > > > >> > > > Keep in mind, although my application is not designed to launch >> the space >> > > > shuttle, I do want it to be solid/stable/peformance-packed from the >> > > ground up. >> > > > >> > > > I will be also be using MySql with the Apache::DBI module. >> > > > >> > > > Thanks in advance. >> > > > >> > > > Gregory >> > > > >> > > > >> > > > At 11:34 PM 5/19/2002 -0400, you wrote: >> > > > > > I have a couple of questions regarding leaking memory in mod_perl: >> > > > > > >> > > > > > 1. What are the main culprits, in order of severity, of >> memory leaks, >> > > > >i.e.: >> > > > > > >> > > > > > a. global variables (NOT lexically scoped via my) >> > > > > > b. ... >> > > > > > c. ... >> > > > > > >> > > > > > 2. When writing code from scratch (a new application), what >> is the >> > > > >best >> > > > > > way to avoid creating leaks to begin with, i.e., use strict;, >> PerlWarn >> > > > >On, >> > > > > > etc.. ? >> > > > > >> > > > >There are actually not very many ways you can leak memory in Perl >> (and >> > > > >thus mod_perl). Most people confuse memory growth with memory >> leakage. >> > > > >If you want to know how to avoid memory growth, look at the >> performance >> > > > >tuning stuff in the Guide, like passing references, avoiding >> slurping of >> > > > >large files, controlling the buffering of DBI result sets, etc. >> > > > > >> > > > >Leaks are caused by circular references, the string form of eval (at >> > > > >least it used to leak a little), nested closures (sometimes created >> > > > >accidentally with the Error module), and one or two obscure syntax >> > > > >problems. I think one of them involved code like "my $x = 7 if $y;". >> > > > >Matt Sergeant got bitten by this in the early stages of AxKit >> > > > >development, and the details are in the mailing list archive. >> > > > > >> > > > >Global variables by themselves are not a source of leaks or >> growth. If >> > > > >you slurp a large file into a global, your process will grow, but the >> > > > >same is true for a lexical. >> > > > > >> > > > >- Perrin >> > > > >> > > > >> > > >> > >-- >> > >C Wayne Huling <[EMAIL PROTECTED]> >> > >> > >> >>-- >>C Wayne Huling <[EMAIL PROTECTED]> > > -- Per Einar Ellefsen [EMAIL PROTECTED]
Re: Memory Leaks
Unfortunately, we only have one machine. If we did employ the cron job as a clean-up utility once per day, wouldn't the potential impact of a site being unavailable only be for a few seconds (until Apache restarted)? Gregory At 05:12 PM 5/20/2002 -0400, you wrote: >Like another suggestion, we have a cluster of machines and roll the >restarts every hour. Each machine is offset but 10 minutes. > >Gregory Matthews writes: > > I too thought of setting a cron job to restart the server once per day in > > order to keep the memory "fresh". > > > > In a production environment, are there any downsides to doing this, i.e., > > server inaccessibility, etc..? > > > > Thanks. > > > > Gregory > > > > At 08:25 AM 5/20/2002 -0400, you wrote: > > > > >It is more an issue of it being worth tracking down a small memory > > >leak vs a large memory leak. Our software still has some very small > > >leaks, on the order of 10kv every hour... it would probably take us a > > >month to track down and solve all these problems. I find it easier to > > >restart the web servers daily. > > > > > >We did have some enourmous leaks as well, based on circular reference, > > >and those ate up 1 GB of memory in about 30 minutes... It took us > > >about three weeks to find it. > > > > > >Gregory Matthews writes: > > > > So am I being overly paranoid concerning the "leak" potential of > mod_perl > > > > programming? > > > > > > > > If I start with "strict" code to begin with and try my best to stay > away > > > > from the problems you mentioned, then any potential memory leak/drain > > > > issues will be avoided? > > > > > > > > Keep in mind, although my application is not designed to launch the > space > > > > shuttle, I do want it to be solid/stable/peformance-packed from the > > > ground up. > > > > > > > > I will be also be using MySql with the Apache::DBI module. > > > > > > > > Thanks in advance. > > > > > > > > Gregory > > > > > > > > > > > > At 11:34 PM 5/19/2002 -0400, you wrote: > > > > > > I have a couple of questions regarding leaking memory in mod_perl: > > > > > > > > > > > > 1. What are the main culprits, in order of severity, of memory > leaks, > > > > >i.e.: > > > > > > > > > > > > a. global variables (NOT lexically scoped via my) > > > > > > b. ... > > > > > > c. ... > > > > > > > > > > > > 2. When writing code from scratch (a new application), what is the > > > > >best > > > > > > way to avoid creating leaks to begin with, i.e., use strict;, > PerlWarn > > > > >On, > > > > > > etc.. ? > > > > > > > > > >There are actually not very many ways you can leak memory in Perl (and > > > > >thus mod_perl). Most people confuse memory growth with memory > leakage. > > > > >If you want to know how to avoid memory growth, look at the > performance > > > > >tuning stuff in the Guide, like passing references, avoiding > slurping of > > > > >large files, controlling the buffering of DBI result sets, etc. > > > > > > > > > >Leaks are caused by circular references, the string form of eval (at > > > > >least it used to leak a little), nested closures (sometimes created > > > > >accidentally with the Error module), and one or two obscure syntax > > > > >problems. I think one of them involved code like "my $x = 7 if $y;". > > > > >Matt Sergeant got bitten by this in the early stages of AxKit > > > > >development, and the details are in the mailing list archive. > > > > > > > > > >Global variables by themselves are not a source of leaks or > growth. If > > > > >you slurp a large file into a global, your process will grow, but the > > > > >same is true for a lexical. > > > > > > > > > >- Perrin > > > > > > > > > > > > > >-- > > >C Wayne Huling <[EMAIL PROTECTED]> > > > > > >-- >C Wayne Huling <[EMAIL PROTECTED]>
Re: Memory Leaks
Gregory Matthews wrote: > I too thought of setting a cron job to restart the server once per day > in order to keep the memory "fresh". > > In a production environment, are there any downsides to doing this, > i.e., server inaccessibility, etc..? There have been some discussion on the list about this in the past. The ideal situation is to have a cluster that you can do a rolling restart on without making the service totally unavailable. - Perrin
Re: Memory Leaks
I too thought of setting a cron job to restart the server once per day in order to keep the memory "fresh". In a production environment, are there any downsides to doing this, i.e., server inaccessibility, etc..? Thanks. Gregory At 08:25 AM 5/20/2002 -0400, you wrote: >It is more an issue of it being worth tracking down a small memory >leak vs a large memory leak. Our software still has some very small >leaks, on the order of 10kv every hour... it would probably take us a >month to track down and solve all these problems. I find it easier to >restart the web servers daily. > >We did have some enourmous leaks as well, based on circular reference, >and those ate up 1 GB of memory in about 30 minutes... It took us >about three weeks to find it. > >Gregory Matthews writes: > > So am I being overly paranoid concerning the "leak" potential of mod_perl > > programming? > > > > If I start with "strict" code to begin with and try my best to stay away > > from the problems you mentioned, then any potential memory leak/drain > > issues will be avoided? > > > > Keep in mind, although my application is not designed to launch the space > > shuttle, I do want it to be solid/stable/peformance-packed from the > ground up. > > > > I will be also be using MySql with the Apache::DBI module. > > > > Thanks in advance. > > > > Gregory > > > > > > At 11:34 PM 5/19/2002 -0400, you wrote: > > > > I have a couple of questions regarding leaking memory in mod_perl: > > > > > > > > 1. What are the main culprits, in order of severity, of memory leaks, > > >i.e.: > > > > > > > > a. global variables (NOT lexically scoped via my) > > > > b. ... > > > > c. ... > > > > > > > > 2. When writing code from scratch (a new application), what is the > > >best > > > > way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn > > >On, > > > > etc.. ? > > > > > >There are actually not very many ways you can leak memory in Perl (and > > >thus mod_perl). Most people confuse memory growth with memory leakage. > > >If you want to know how to avoid memory growth, look at the performance > > >tuning stuff in the Guide, like passing references, avoiding slurping of > > >large files, controlling the buffering of DBI result sets, etc. > > > > > >Leaks are caused by circular references, the string form of eval (at > > >least it used to leak a little), nested closures (sometimes created > > >accidentally with the Error module), and one or two obscure syntax > > >problems. I think one of them involved code like "my $x = 7 if $y;". > > >Matt Sergeant got bitten by this in the early stages of AxKit > > >development, and the details are in the mailing list archive. > > > > > >Global variables by themselves are not a source of leaks or growth. If > > >you slurp a large file into a global, your process will grow, but the > > >same is true for a lexical. > > > > > >- Perrin > > > > > >-- >C Wayne Huling <[EMAIL PROTECTED]>
Re: Memory Leaks
On Mon, 20 May 2002, Mark Fowler wrote:
> On Mon, 20 May 2002, Matt Sergeant wrote:
>
> > if ($@ && $@->isa('Exception::DB')) {
> >debug "Exception: $@";
> >$um->dbh->rollback;
> > }
> >
> > (note: if you expect all exceptions to be references like this, you had
> > better have a $SIG{__DIE__} handler installed to bless non-blessed
> > exceptions before re-throwing them
>
> Can't you just use UNIVERSAL's ISA method directly?
>
>if (UNIVERSAL::isa($@,'Exception::DB')) {
>
> This of course might fail if you got the string "Exception::DB" or
> likewise back as an error message.
>
> Alternativly, check if it's blessed
>
>use Scalar::Util qw(blessed);
>
>if (blessed($@) && $@->isa('Exception::DB')) {
Yeah, I know all the tricks. Ultimately it's a matter of how ugly you want
your code to get, and how many external modules you want to rely on (I
believe Scalar::Util is going to be part of 5.8 though).
--
<:->Get a smart net
Re: Memory Leaks
On Mon, 20 May 2002, Matt Sergeant wrote:
> if ($@ && $@->isa('Exception::DB')) {
>debug "Exception: $@";
>$um->dbh->rollback;
> }
>
> (note: if you expect all exceptions to be references like this, you had
> better have a $SIG{__DIE__} handler installed to bless non-blessed
> exceptions before re-throwing them
Can't you just use UNIVERSAL's ISA method directly?
if (UNIVERSAL::isa($@,'Exception::DB')) {
This of course might fail if you got the string "Exception::DB" or
likewise back as an error message.
Alternativly, check if it's blessed
use Scalar::Util qw(blessed);
if (blessed($@) && $@->isa('Exception::DB')) {
Later.
Mark.
--
s'' Mark Fowler London.pm Bath.pm
http://www.twoshortplanks.com/ [EMAIL PROTECTED]
';use Term'Cap;$t=Tgetent Term'Cap{};print$t->Tputs(cl);for$w(split/ +/
){for(0..30){$|=print$t->Tgoto(cm,$_,$y)." $w";select$k,$k,$k,.03}$y+=2}
Re: Memory Leaks
On Mon, 20 May 2002, F. Xavier Noria wrote:
> On Mon, 20 May 2002 10:15:02 +0100 (BST)
> Matt Sergeant <[EMAIL PROTECTED]> wrote:
>
> : > my $um = UserManager->new;
> : > # ...
> : > try {
> : > $um->write_user($user);
> : > $um->dbh->commit;
> : > } catch Exception::DB with {
> : > my $e = shift;
> : > debug "Exception: $e";
> : > $um->dbh->rollback;
> : > };
> :
> : No. $um is caught in a closure, which could potentially leak.
>
> Wow, thank you, I have that pattern repeated in the code many times.
>
> That is the way I would write that try/catch in Java, where you need to
> have $um in the scope of the try and the catch blocks, what is the right
> way to write that in Perl/Error.pm?
I gave up on Error.pm's try/catch syntax a long time ago - I think it's
hidden closure system combined with perl bugs is just too broken for
production use. Instead I use good old eval:
my $um = UserManager->new;
...
eval {
$um->write_user($user);
$um->dbh->commit;
};
if ($@ && $@->isa('Exception::DB')) {
debug "Exception: $@";
$um->dbh->rollback;
}
(note: if you expect all exceptions to be references like this, you had
better have a $SIG{__DIE__} handler installed to bless non-blessed
exceptions before re-throwing them - ask me if you need an example of
that)
--
<:->Get a smart net
Re: Memory Leaks
On Mon, 20 May 2002 10:15:02 +0100 (BST)
Matt Sergeant <[EMAIL PROTECTED]> wrote:
: > my $um = UserManager->new;
: > # ...
: > try {
: > $um->write_user($user);
: > $um->dbh->commit;
: > } catch Exception::DB with {
: > my $e = shift;
: > debug "Exception: $e";
: > $um->dbh->rollback;
: > };
:
: No. $um is caught in a closure, which could potentially leak.
Wow, thank you, I have that pattern repeated in the code many times.
That is the way I would write that try/catch in Java, where you need to
have $um in the scope of the try and the catch blocks, what is the right
way to write that in Perl/Error.pm?
-- fxn
Re: Memory Leaks
On Mon, 20 May 2002, F. Xavier Noria wrote:
> On Sun, 19 May 2002 23:34:24 -0400
> "Perrin Harkins" <[EMAIL PROTECTED]> wrote:
>
> : Leaks are caused by circular references, the string form of eval (at
> : least it used to leak a little), nested closures (sometimes created
> : accidentally with the Error module)
>
> I am using the Error module in my current project, what kind of
> constructs should one avoid? Is this safe?
>
> my $um = UserManager->new;
> # ...
> try {
> $um->write_user($user);
> $um->dbh->commit;
> } catch Exception::DB with {
> my $e = shift;
> debug "Exception: $e";
> $um->dbh->rollback;
> };
No. $um is caught in a closure, which could potentially leak.
--
<:->Get a smart net
Re: Memory Leaks
On Sun, 19 May 2002 23:34:24 -0400
"Perrin Harkins" <[EMAIL PROTECTED]> wrote:
: Leaks are caused by circular references, the string form of eval (at
: least it used to leak a little), nested closures (sometimes created
: accidentally with the Error module)
I am using the Error module in my current project, what kind of
constructs should one avoid? Is this safe?
my $um = UserManager->new;
# ...
try {
$um->write_user($user);
$um->dbh->commit;
} catch Exception::DB with {
my $e = shift;
debug "Exception: $e";
$um->dbh->rollback;
};
-- fxn
Re: Memory Leaks
> So am I being overly paranoid concerning the "leak" potential of mod_perl > programming? No, memory management is very important with mod_perl. > If I start with "strict" code to begin with and try my best to stay away > from the problems you mentioned, then any potential memory leak/drain > issues will be avoided? Using strict is an absolute must for many reasons. It won't prevent you from using up tons of memory though. You have to become tuned in to the things that use up memory so that you can avoid them. Start with the Guide. - Perrin
Re: Memory Leaks
Not long ago, Gregory Matthews proclaimed... > So am I being overly paranoid concerning the "leak" potential of mod_perl > programming? No... But once you do it "right" it comes natural. The thing that killed me when I first started doing mod_perl development was code that pushed items onto lists in Apache::Registry scripts. I wasn't initializing those list variables and before you knew it, lists were getting populated with values in duplicate, triplicate, and so on. Nothing has helped me become a better Perl programming than mod_perl. :) -=Fozz -- Doran L. Barton <[EMAIL PROTECTED]> - Chief Super Hero - Iodynamics LLC < http://www.iodynamics.com/ > - Linux solutions and dynamic websites "Dog for sale: eats anything and is fond of children." -- Classified ad in newspaper
Re: Memory Leaks
So am I being overly paranoid concerning the "leak" potential of mod_perl programming? If I start with "strict" code to begin with and try my best to stay away from the problems you mentioned, then any potential memory leak/drain issues will be avoided? Keep in mind, although my application is not designed to launch the space shuttle, I do want it to be solid/stable/peformance-packed from the ground up. I will be also be using MySql with the Apache::DBI module. Thanks in advance. Gregory At 11:34 PM 5/19/2002 -0400, you wrote: > > I have a couple of questions regarding leaking memory in mod_perl: > > > > 1. What are the main culprits, in order of severity, of memory leaks, >i.e.: > > > > a. global variables (NOT lexically scoped via my) > > b. ... > > c. ... > > > > 2. When writing code from scratch (a new application), what is the >best > > way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn >On, > > etc.. ? > >There are actually not very many ways you can leak memory in Perl (and >thus mod_perl). Most people confuse memory growth with memory leakage. >If you want to know how to avoid memory growth, look at the performance >tuning stuff in the Guide, like passing references, avoiding slurping of >large files, controlling the buffering of DBI result sets, etc. > >Leaks are caused by circular references, the string form of eval (at >least it used to leak a little), nested closures (sometimes created >accidentally with the Error module), and one or two obscure syntax >problems. I think one of them involved code like "my $x = 7 if $y;". >Matt Sergeant got bitten by this in the early stages of AxKit >development, and the details are in the mailing list archive. > >Global variables by themselves are not a source of leaks or growth. If >you slurp a large file into a global, your process will grow, but the >same is true for a lexical. > >- Perrin
Re: Memory Leaks
> I have a couple of questions regarding leaking memory in mod_perl: > > 1. What are the main culprits, in order of severity, of memory leaks, i.e.: > > a. global variables (NOT lexically scoped via my) > b. ... > c. ... > > 2. When writing code from scratch (a new application), what is the best > way to avoid creating leaks to begin with, i.e., use strict;, PerlWarn On, > etc.. ? There are actually not very many ways you can leak memory in Perl (and thus mod_perl). Most people confuse memory growth with memory leakage. If you want to know how to avoid memory growth, look at the performance tuning stuff in the Guide, like passing references, avoiding slurping of large files, controlling the buffering of DBI result sets, etc. Leaks are caused by circular references, the string form of eval (at least it used to leak a little), nested closures (sometimes created accidentally with the Error module), and one or two obscure syntax problems. I think one of them involved code like "my $x = 7 if $y;". Matt Sergeant got bitten by this in the early stages of AxKit development, and the details are in the mailing list archive. Global variables by themselves are not a source of leaks or growth. If you slurp a large file into a global, your process will grow, but the same is true for a lexical. - Perrin
Re: Memory Leaks?
At 15:50 15/03/2001 -0300, Jason Leidigh wrote:
>I was able to clean up a number of errors which seemed as
>though they were indeed causing leaks. For example:
>
>$regex = qr'xx?'i;
>
>Causes the following error:
>
>(?i-xsm:xx?) can't `Regexp::DESTROY'
AUTOLOADs will catch DESTROYs, the latter being called on every object
destruction (I never noticed that they were called on regexen, that's
nice). Chances are, unless you clearly see a memory leak with DESTROYs
(which are expected not to be found most of the time) then your AUTOLOAD
should probably explicitly ignore them.
Another solution is:
sub Regexp::DESTROY {}
sub Apache::DESTROY {}
etc...
-- robin b.
"Science is like sex: sometimes something useful comes out, but that is not
the reason we are doing it." -- R. Feynman
Re: Memory leaks?
On Tue, 2 Jan 2001, Per 'stripe' Moeller wrote: > Hi, > > I have created a highly configurable content management system in modperl to > build websites, which consists of several modules, some preload when apache > initiates, some when a request initiates and some when needed. > > I had the opportunity to test the system with 2000 lan connected users to do > some performance checking, and found out that for each request the server > received, the child process which handled the request used a little more > memory each time. In this test I had around 500.000 hits pr day, and > apache/modperl used up all the 1Gb of memory the server had in a very short > time. I checked and rechecked my code and could not find any kind of memory > leak in it, so finally I had to configure the Apache to kill child processes > after answering 100 requests, otherwise they would consume too much memory. > > Have anybody else experienced this kind of problem? Lots of people. Memory leaks are very easy to introduce, and very hard to find. The most likely causes are closures, circular references, and bad modules. Not necessarily in that order :-) > Another problem I encountered... when doing HTTP upload's, apache/modperl > uses 7-8 times the size of the uploaded file, of memory. If I uploaded a > 10Mb file, the server would typically use up to 170Mb for the child > answering the request, and the child would not free all the memory used when > the request had been handled. Can this really be true? Yes its true. Try the latest Apache::Request test release - it should fix this problem. -- /||** Director and CTO ** //||** AxKit.com Ltd ** ** XML Application Serving ** // ||** http://axkit.org ** ** XSLT, XPathScript, XSP ** // \\| // ** Personal Web Site: http://sergeant.org/ ** \\// //\\ // \\
Re: memory leaks redux
On Sat Jan 29 13:11:25 2000 + Mike Whitaker wrote: > [EMAIL PROTECTED] (Doug MacEachern) wrote: > >there are hints in the SUPPORT doc on how to debug such problems. there > >was also several "Hanging process" threads in the past weeks with more > >tips, search in the archives for keywords gdb, .gdbinit, curinfo > >if you can get more insight from those tips, we can help more. > > I have also seen (and reported via the Debian bugs system) a problem > which I think has been observed before where HUP'ing the root httpd > causes it to reload every darn PerlModule directive, and bloat > accordingly (with our server, that's 3M per SIGHUP, which makes log > rotation somewhat painful (when you get 3 million hits in 8 hours, you > rotate those logs pretty fast!)). > You can log to pipe. -- Ričardas Čepas ~~~ ~~ PGP signature
Re: Memory Leaks in mod_perl
On Wed, Nov 03, 1999 at 10:40:08PM -0500, David Huggins-Daines wrote: > I have more or less the same problem as Ben here. mod_perl appears to leak > memory on SIGHUP proportionately to the number of extra Perl modules loaded > into the interpreter I think to some extent individual modules are to blame (though again - this might be something in the Apache core that they use). When I did some checking I found that PerlModule Apache didn't seem to leak any more memory than just having a section (though this may be because it's implicitly loaded by having a section - anyone confirm this?) Apache::Status on the other hand leaked around 936K. Apache::DBI (not PerlSections as I stated in my last post) leaks 404K, though 360K of this seems to be DBI itself. It seems to be the native code Perl Modules that cause the most trouble. Maybe a short term solution to this would be to allow a directive which destroys the Perl interpreter on restarts. > I have experienced this with: > Apache/1.3.6, mod_perl 1.21, Red Hat 5.2/i386 (perl 5.004_05 and 5.005_03) Well I guess it's not something bizarre about the way tha Debian compiled it then. > I have not been able to check earlier versions of mod_perl because I can't > seem to find them on CPAN :) > > Please also note Debian bug reports #34947 and #41946. > (http://bugs.debian.org/34947, http://bugs.debian.org/41946) Bug 34947 might be a different problem. But if it is, the poster stated that statically linked 1.3.3/1.16_02 worked well for him. I'll see if I can track down some earlier versions and build them like this. Ben -- +-Ben Bell - "A song, a perl script and the occasional silly sig.-+ /// email: [EMAIL PROTECTED]www: http://www.deus.net/~bjb/ bjbDon't try to drive me crazy... \_/...I'm close enough to walk.
Re: Memory Leaks in mod_perl
On Thu, Nov 04, 1999 at 12:32:07PM +, Ben Bell wrote: > Bug 34947 might be a different problem. But if it is, the poster stated > that statically linked 1.3.3/1.16_02 worked well for him. I'll see if > I can track down some earlier versions and build them like this. It's DSO related for sure, I've built a static 1.3.9/1.21 here and it doesn't do this at all. I've hacked the Debian package to build with static mod_perl in the Apache binary, source and binary packages are at: http://elgin.plcom.on.ca/debian/dists/unstable/main/ Or put this in your /etc/apt/sources.list and "apt-get install apache-perl": deb http://elgin.plcom.on.ca/debian unstable main (note: this server may be up and down for a bit, it's also the development machine for my project at work that uses mod_perl...) These aren't official packages, of course. Hopefully the memory leakage on DSO problem can be resolved before we release potato, it is a rather severe bug IMHO.
Re: Memory Leaks?
On Sun, Oct 31, 1999 at 11:33:19PM -0800, Doug MacEachern wrote: > sounds like you have PerlFreshRestart On, try turning it Off. scan the > archives for more info. I have tried both ways. Also, the mail archives say that this is a bug that was fixed in 1.21 (which I am running). I've done some more checks of different configurations and measured the leak. Leak Without mod_perl: 8 PerlFreshRestart On:40 PerlFreshRestart Off: 40 Empty section: 396 PerlModule Apache 396 same plus Apache::Registry 452 + ::Status 1388 + PerlHandler Apache::Status 1388 + PerlModule Apache::PerlSections 1400 + Apache::DBI 1804 + Data::Dumper 1804 + Carp 1804 + PerlRequire mystartup.pl 1896 So it looks like we leak: 8K to Apache 32K to mod_perl itself (though maybe this is a leak in apache's module handling code?) 56K to Apache::Registry 356K to mod_perl when you use it for anything 404K to Apache::PerlSections 936K to Apache::Status (and 92K to something in my code :) Can anyone else with the same config (Apache 1.3.9, mod_perl 1.21) confirm this? I am using the Debian packages (1.3.9-10 and 1.21-5 respectively) so it may be something enabled at compile time. Cheers, Ben -- +-Ben Bell - "A song, a perl script and the occasional silly sig.-+ /// email: [EMAIL PROTECTED]www: http://www.deus.net/~bjb/ bjbDon't try to drive me crazy... \_/...I'm close enough to walk.
Re: Memory Leaks?
sounds like you have PerlFreshRestart On, try turning it Off. scan the archives for more info. On Fri, 29 Oct 1999, Ben Bell wrote: > Hi, > > I'm using the Debian package of mod_perl (1.21) and apache 1.3.9 and > I've noticed quite nasty memory leaks on server restart. I've noticed > unresolved bug reports on the Debian pages about this. Is it a known > issue with this version? > > The leak is ca. 2MB each restart (or graceful) with my startup script > enabled (which queries a database, and uses the following modules: > Apache > Apache::PerlSections > Apache::DBI > Data::Dumper > Carp > VI::Utils > > All my vars are declared as "my". > When I disable all Perl stuff (my startup script, the sections etc) > I still see a memory leak, albeit a smaller one. > > Can anyone shed any light on this? > > Cheers, > Ben > > > -- > +-Ben Bell - "A song, a perl script and the occasional silly sig.-+ > /// email: [EMAIL PROTECTED]www: http://www.deus.net/~bjb/ > bjbDon't try to drive me crazy... > \_/...I'm close enough to walk. >
