RE: [Catalyst] Memory leak under FastCGI?
-Original Message- From: Matt S Trout [mailto:[EMAIL PROTECTED] Sent: Friday, April 11, 2008 10:35 AM To: The elegant MVC web framework Subject: Re: [Catalyst] Memory leak under FastCGI? On Thu, Apr 10, 2008 at 03:11:56PM -0400, Matt Pitts wrote: -Original Message- From: Matt S Trout [mailto:[EMAIL PROTECTED] Sent: Thursday, April 10, 2008 12:33 PM To: The elegant MVC web framework Subject: Re: [Catalyst] Memory leak under FastCGI? On Mon, Apr 07, 2008 at 10:52:54AM -0400, Matt Pitts wrote: As of now, I'm trying to hack up a better ProcManager based on FCGI::Engine::ProcManager that actually recycles its children using options like MaxRequestPerChild. Hopefully, I'll be able to get it done and cleaned up enough to release. Please don't; there's work being done on an app plugin that implements this generically, which is a much better approach. Contact John I see generic from two directions: Catalyst and FastCGI. Implementing this as a subclass of FCGI::Engine::ProcManager makes it available to anyone using FastCGI and is, therefore, more generic to FastCGI. However, implementing this as a Cat plugin makes it generic (or abstract) to all the C::Engine:: modules, so that all Cat apps can enjoy the benefits. I'm not sure that either is the better approach they just have different scopes. Napiorkowski for his current prototype and help get that finished. Once you use something like that, the child exits and ProcManager just spawns another one. I don't like this approach because the dying of children isn't being managed in a single place. If it's done like this and the children don't coordinate their deaths then you could run into a situation where many or all of the children die at the same time - this is bad for obvious reasons, especially in an external FastCGI app. My plan was do all child TERM signaling from within the manager app and use some simple heuristics to prevent a mass-child death situation. Obviously, if it's written as a plugin, then I can choose to use or not to use - that's be beauty of Catalyst - I guess I'm just trying to defend my position that I like the idea of having a MaxRequestsPerChild feature implemented in the manager of my FastCGI processes not within the children themselves. The accept() happens in the child in normal situations though, so the parent can't know how many requests a given child has handled without additional complication. Yeap. My version of the additional complication is a lite IPC messaging system using DBM::Deep. I override post_dispatch in the children to send a message to the queue that says I just finished handling a request and then the manager iterates the message queue during each daemon loop and keeps track of each child's request count. This obviously introduces some overhead to the post_dispatch call, but I'm thinking that since MyApp-handle_request has already been called that all data has been sent to the client (buffering?), the only thing that might not have happened yet is the closing of the FastCGI connection with the socket. Am I wrong? Can anyone elaborate on the effects of doing this in post_dispatch? Basically, you -can't- manage it in a single place. A don't die for a bit, somebody else just did mechanism would be good as a co-ordination between manager and children and IMO should be implemented separately. My plan is to build it as a subclass of FCGI::Engine::ProcManager so folks can use it at their own discretion. v/r -matt ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
RE: [Catalyst] Memory leak under FastCGI?
-Original Message- From: Matt S Trout [mailto:[EMAIL PROTECTED] Sent: Friday, April 11, 2008 1:23 PM To: The elegant MVC web framework Subject: Re: [Catalyst] Memory leak under FastCGI? On Fri, Apr 11, 2008 at 11:20:53AM -0400, Matt Pitts wrote: The accept() happens in the child in normal situations though, so the parent can't know how many requests a given child has handled without additional complication. Yeap. My version of the additional complication is a lite IPC messaging system using DBM::Deep. I override post_dispatch in the children to send a message to the queue that says I just finished handling a request and then the manager iterates the message queue during each daemon loop and keeps track of each child's request count. Yuck. Seems much simpler to me to do something like: Child1 hits request count, sends SIGUSR1 to proc manager, starts exiting Manager sends SIGUSR1 to all other children (child may take a while to clean up, exit happens somewhen) Other children go into must not die mode Manager spawns new child once it gets SIGCHLD New child sends SIGUSR2 to proc manager once it's starting its accept() loop Proc manager sends SIGUSR2 to all children to indicate they're allowed to die now if they want to Seems a lot more unixy to me, and likely much less overhead as well. Neat. I like this approach, especially if the only process limitation is MaxRequestsPerChild. My original motivation behind a message queue was to make it easier to add additional management facilities, but as of now MaxRequestsPerChild is the main one I'm interested in. Although I had thought about using SIGUSR1 or 2 to implement a simple ping to make sure that a child is still there. I'll definitely keep your approach in mind if I decide to only implement MaxRequestsPerChild. Going the SIGUSR approach I think I'd still want the manager to be the one sending SIGTERM to the children and have the children trap SIGTERM and set a flag to exit(0) at the end of the current request during post_dispatch. This way if the manager wants to kill multiple children in a short time period it can. Thanks for the input! v/r -matt pitts ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Memory leak under FastCGI?
On Mon, Apr 07, 2008 at 10:52:54AM -0400, Matt Pitts wrote: As of now, I'm trying to hack up a better ProcManager based on FCGI::Engine::ProcManager that actually recycles its children using options like MaxRequestPerChild. Hopefully, I'll be able to get it done and cleaned up enough to release. Please don't; there's work being done on an app plugin that implements this generically, which is a much better approach. Contact John Napiorkowski for his current prototype and help get that finished. Once you use something like that, the child exits and ProcManager just spawns another one. Note also that the FCGI::ProcManager maintainer is Gareth Kirwan and can be found hopefully via this list or as gbjk on IRC. (moral of the story: RFC on here first before coding lest ye be reinventing a wheel :) -- Matt S Trout Need help with your Catalyst or DBIx::Class project? Technical Directorhttp://www.shadowcat.co.uk/catalyst/ Shadowcat Systems Ltd. Want a managed development or deployment platform? http://chainsawblues.vox.com/http://www.shadowcat.co.uk/servers/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
RE: [Catalyst] Memory leak under FastCGI?
-Original Message- From: Matt S Trout [mailto:[EMAIL PROTECTED] Sent: Sunday, April 06, 2008 1:18 PM To: The elegant MVC web framework Subject: Re: [Catalyst] Memory leak under FastCGI? On Wed, Mar 19, 2008 at 03:32:12PM -0400, Matt Pitts wrote: I have not yet determined if the conversion from mod_fcgid to mod_fastcgi external is when the RAM usage started climbing - still awaiting system reports - but, there haven't been an substantial changes to the app since then that I would consider potential causes of a memory leak. I'm guessing it's been there for a while, but it's only now evident because the app instances were getting automatically recycled by mod_fcgid and now they're not. Right, which is why I'm confused by your mentioning 'under FastCGI'. The problem is almost certainly that your application has a memory leak in it, not the engine side of things - you should try replaying a known set of requests against the application and use things like Devel::Leak to determine if you're leaking SVs. Is it generally an acceptable practice to just restart the external fastcgi process periodically to free its memory? Yep. Other than really gross leaks it's often not worth the developer time involved to track them down. Thanks for the feedback. I got a pointer to: http://www.catalystframework.org/calendar/2007/18 And after looking that over I started feeling like copy-on-write was more the cause than leaking memory. It may still be leaking, but it is so gradual that it's hard to know without a detailed investigation. As of now, I'm trying to hack up a better ProcManager based on FCGI::Engine::ProcManager that actually recycles its children using options like MaxRequestPerChild. Hopefully, I'll be able to get it done and cleaned up enough to release. [OT] Can anyone recommend a good IPC message queue? I was working with IPC::PubSub, but then started getting weird errors out of DBM::Deep. I want something lite that doesn't require a completely separate process and keeps me from having to deal with sockets. v/r -matt pitts ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Memory leak under FastCGI?
On Wed, Mar 19, 2008 at 03:32:12PM -0400, Matt Pitts wrote: I have not yet determined if the conversion from mod_fcgid to mod_fastcgi external is when the RAM usage started climbing - still awaiting system reports - but, there haven't been an substantial changes to the app since then that I would consider potential causes of a memory leak. I'm guessing it's been there for a while, but it's only now evident because the app instances were getting automatically recycled by mod_fcgid and now they're not. Right, which is why I'm confused by your mentioning 'under FastCGI'. The problem is almost certainly that your application has a memory leak in it, not the engine side of things - you should try replaying a known set of requests against the application and use things like Devel::Leak to determine if you're leaking SVs. Is it generally an acceptable practice to just restart the external fastcgi process periodically to free its memory? Yep. Other than really gross leaks it's often not worth the developer time involved to track them down. -- Matt S Trout Need help with your Catalyst or DBIx::Class project? Technical Directorhttp://www.shadowcat.co.uk/catalyst/ Shadowcat Systems Ltd. Want a managed development or deployment platform? http://chainsawblues.vox.com/http://www.shadowcat.co.uk/servers/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Memory leak under FastCGI?
There was something about growing FastCGI processes in the Catalyst advent calendar: http://www.catalystframework.org/calendar/2007/18 In short: Apparently this is normal behavior, and the article recommends you, like you suggested, to periodically restart fastcgi processes to keep memory usage reasonable. On 3/19/08, Matt Pitts [EMAIL PROTECTED] wrote: We have a Catalyst app that I recently (about a month ago) converted from running under mod_fcgid to external under mod_fastcgi and it appears to be leaking memory. There are 2 application backends and they both suffered oom-killer events within six minutes of one another yesterday and one of them again this morning after a cron job ran that experienced some deep recursion calls. Both backends were running out of PARs when the first OOM events occurred. To eliminate the PAR setup itself as a cause, I started one of the backends by directly invoking myapp_fastcgi.pla and its memory behavior is the same, so I'm pretty confident the invocation via PAR is not a cause. I also checked my versions of CGI::FormBuilder to ensure I was up to 3.501 and these all checked out. I have not yet determined if the conversion from mod_fcgid to mod_fastcgi external is when the RAM usage started climbing - still awaiting system reports - but, there haven't been an substantial changes to the app since then that I would consider potential causes of a memory leak. I'm guessing it's been there for a while, but it's only now evident because the app instances were getting automatically recycled by mod_fcgid and now they're not. I'm hoping someone here might have some light to shine on common FastCGI memory leak issues. Is it generally an acceptable practice to just restart the external fastcgi process periodically to free its memory? Any input is greatly appreciated. v/r -matt pitts ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Memory leak under FastCGI?
On Mon, Mar 31, 2008 at 5:22 AM, Matthieu Codron [EMAIL PROTECTED] wrote: There was something about growing FastCGI processes in the Catalyst advent calendar: http://www.catalystframework.org/calendar/2007/18 In short: Apparently this is normal behavior, and the article recommends you, like you suggested, to periodically restart fastcgi processes to keep memory usage reasonable. On 3/19/08, Matt Pitts [EMAIL PROTECTED] wrote: We have a Catalyst app that I recently (about a month ago) converted from running under mod_fcgid to external under mod_fastcgi and it appears to be leaking memory. There are 2 application backends and they both suffered oom-killer events within six minutes of one another yesterday and one of them again this morning after a cron job ran that experienced some deep recursion calls. Both backends were running out of PARs when the first OOM events occurred. To eliminate the PAR setup itself as a cause, I started one of the backends by directly invoking myapp_fastcgi.pla and its memory behavior is the same, so I'm pretty confident the invocation via PAR is not a cause. I also checked my versions of CGI::FormBuilder to ensure I was up to 3.501 and these all checked out. I have not yet determined if the conversion from mod_fcgid to mod_fastcgi external is when the RAM usage started climbing - still awaiting system reports - but, there haven't been an substantial changes to the app since then that I would consider potential causes of a memory leak. I'm guessing it's been there for a while, but it's only now evident because the app instances were getting automatically recycled by mod_fcgid and now they're not. I'm hoping someone here might have some light to shine on common FastCGI memory leak issues. Is it generally an acceptable practice to just restart the external fastcgi process periodically to free its memory? Any input is greatly appreciated. We wrote a plugin at work to deal with this at the handle_request phase, that way nothing dies mid request. You'll just have to bug John Napiorkowski to release it :-) I think its under the heat lamp ready to go.. ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
RE: [Catalyst] Memory leak under FastCGI?
Thanks very much for the pointer - it's good to know that I'm not alone in my troubles. I'm now actually planning on writing a ProcManager based on FCGI::Engine::ProcManager that incorporates some true process management - a la MaxRequestsPerChild - rather than just a loop to startup X childs and then a blocking wait(). v/r -matt pitts From: Matthieu Codron [mailto:[EMAIL PROTECTED] Sent: Monday, March 31, 2008 5:22 AM To: The elegant MVC web framework Subject: Re: [Catalyst] Memory leak under FastCGI? There was something about growing FastCGI processes in the Catalyst advent calendar: http://www.catalystframework.org/calendar/2007/18 In short: Apparently this is normal behavior, and the article recommends you, like you suggested, to periodically restart fastcgi processes to keep memory usage reasonable. On 3/19/08, Matt Pitts [EMAIL PROTECTED] wrote: We have a Catalyst app that I recently (about a month ago) converted from running under mod_fcgid to external under mod_fastcgi and it appears to be leaking memory. There are 2 application backends and they both suffered oom-killer events within six minutes of one another yesterday and one of them again this morning after a cron job ran that experienced some deep recursion calls. Both backends were running out of PARs when the first OOM events occurred. To eliminate the PAR setup itself as a cause, I started one of the backends by directly invoking myapp_fastcgi.pla and its memory behavior is the same, so I'm pretty confident the invocation via PAR is not a cause. I also checked my versions of CGI::FormBuilder to ensure I was up to 3.501 and these all checked out. I have not yet determined if the conversion from mod_fcgid to mod_fastcgi external is when the RAM usage started climbing - still awaiting system reports - but, there haven't been an substantial changes to the app since then that I would consider potential causes of a memory leak. I'm guessing it's been there for a while, but it's only now evident because the app instances were getting automatically recycled by mod_fcgid and now they're not. I'm hoping someone here might have some light to shine on common FastCGI memory leak issues. Is it generally an acceptable practice to just restart the external fastcgi process periodically to free its memory? Any input is greatly appreciated. v/r -matt pitts ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/ ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Memory leak under FastCGI?
We have a Catalyst app that I recently (about a month ago) converted from running under mod_fcgid to external under mod_fastcgi and it appears to be leaking memory. There are 2 application backends and they both suffered oom-killer events within six minutes of one another yesterday and one of them again this morning after a cron job ran that experienced some deep recursion calls. Both backends were running out of PARs when the first OOM events occurred. To eliminate the PAR setup itself as a cause, I started one of the backends by directly invoking myapp_fastcgi.pla and its memory behavior is the same, so I'm pretty confident the invocation via PAR is not a cause. I also checked my versions of CGI::FormBuilder to ensure I was up to 3.501 and these all checked out. I have not yet determined if the conversion from mod_fcgid to mod_fastcgi external is when the RAM usage started climbing - still awaiting system reports - but, there haven't been an substantial changes to the app since then that I would consider potential causes of a memory leak. I'm guessing it's been there for a while, but it's only now evident because the app instances were getting automatically recycled by mod_fcgid and now they're not. I'm hoping someone here might have some light to shine on common FastCGI memory leak issues. Is it generally an acceptable practice to just restart the external fastcgi process periodically to free its memory? Any input is greatly appreciated. v/r -matt pitts ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
