While this is an interesting idea, I have a few concerns about it: 1. This won't take care of requests that balloon memory very rapidly (such as the issue we recently found in katello which would cause OOM in a matter of minutes). Waiting up to an hour to reclaim memory might not be fast enough in those cases. 2. If passenger fails to complete a response within the graceful shutdown period, users will just have a request failed and no indication of what happened. While most requests should finish within 2 minutes, for example a large csv download can take much longer then that, and user will not know why it failed (and it could be completely unrelated, leading to red herrings - the request that failed won't necessarily be the one that caused the leak). 3. We might not know about places where we leak memory, since the process will get killed without anyone noticing it.
On Wed, May 10, 2017 at 10:43 AM, Ohad Levy <[email protected]> wrote: > > > On Wed, May 10, 2017 at 10:28 AM, Lukas Zapletal <[email protected]> wrote: > >> Hello, >> >> I wrote a script that gracefully terminates Passenger processes >> consuming more than 2.5 GB of private RSS memory. Typical consumption >> of Foreman with Katello and other basic plugins is around 1.5 GB and >> since only Passenger Enterprise allows to limit maximum amount of RSS >> memory for processes, this simple script does exactly that: >> >> https://gist.github.com/lzap/8dddbe66ec8d43cbd4277c1de7045c17 >> >> Put it into your /etc/cron.hourly/ and make it executable, make sure >> you read root emails or forward it properly, the script reports all >> terminations performed on STDOUT. I would like you to test this script >> in production and get back to me with feedback about what you think. >> >> Motivation is simple - we often introduce bugs in our Rails codebase >> which performs some eager loading or there are memory leaks in our >> code or dependencies and Passenger processes can grow up to dozens >> gigabytes. Unfortunately, there is no other way of getting out other >> than restarting httpd with passenger. This script could help to avoid >> situations when production instance starts to swap hard thank to some >> small regression we introduced. Also administrator will be noticed >> early via email when this happens so we will keep track of these >> regressions in production. >> >> http://projects.theforeman.org/issues/19496 >> >> Feedback appreciated. >> >> > I gave it a spin, two comments: > > 1. my default memory size was around 400m i had to change the script it in > order to see it in action, maybe we should take into account avail memory > and how many passenger processes are allowed - the default of 2.5gb seems a > bit too high? > 2. in order to use your script in an scl env, you would need to wrap it > with scl enable <collection> passenger-recycler > > Thanks! > Ohad > > >> -- >> Later, >> Lukas @lzap Zapletal >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- Have a nice day, Tomer Brisker Red Hat Engineering -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
