In general threads consume more CPU while processes consume more RAM. Reducing the number of threads may not have any considerable impact on your overall memory usage. You could try a memory profiling tool to identify memory leaks in your application or maybe there is a way to restart your sidekiq master process when it begins to consume too much memory.
— Sent from Mailbox for iPhone On Mon, Nov 11, 2013 at 11:10 AM, Neil Middleton <n...@neilmiddleton.com> wrote: > Yeah, so R14 is more of a warning than an error - your app is still running, > but much slower than normal. > I would suggest looking at what you’re loading into memory during a worker > run - I would guess that you’re loading your *entire* app and all it’s > dependencies in order to do something that I would guess is pretty simple. > - N > On 11 November 2013 at 16:06:32, Jack Royal-Gordon (jac...@pobox.com) wrote: > I have background processes running Sidekiq that run for several hours each > night. They generate a large number of R14 errors (1600 for 2 processes each > running about 6-9 hours). But it seems like the excessive memory is slight > (not more than 520MB vs the 512MB allocated), so it would not be justified to > use jumbo (1024MB) processes. > Threads require pre-loading of the runtime environment, which includes the > Rails library. If there are parts of the library that I'm not using (and > there are), I ought to be able to fine-tune the pre-load process to exclude > them, eventually achieving a point where memory usage comes within 512MB. > Determining which parts are used and which aren't can be a tedious > experience. Can I not just rely on the paging to eventually page out the > parts I'm not using, at which point no more paging will take place because > the out-of-memory pages are never requested? > I tried lowering the number of concurrent jobs per process (i.e. threads) > from 10 to 8, but it seemed like it only made up 4-5MB in excess memory and > did not affect the number of R14 errors. In order to make up for the lost > concurrency, I added another process. The overall effect was to slow > everything down (an additional 2-1/2 hours of processing to get through the > workload), so I concluded that the extra concurrency (3x8=24 vs 2x10 = 20) > put extra load on the Postgre database which became a processing bottleneck. > So my gut feeling is that changing the jobs-per-process count is not the best > way to lower memory usage. > Any other thoughts on how to solve this (or whether it really needs to be > solved) would be appreciated. > -- > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en_US?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Heroku Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to heroku+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > -- > -- > You received this message because you are subscribed to the Google > Groups "Heroku" group. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en_US?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Heroku Community" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to heroku+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. -- -- You received this message because you are subscribed to the Google Groups "Heroku" group. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en_US?hl=en --- You received this message because you are subscribed to the Google Groups "Heroku Community" group. To unsubscribe from this group and stop receiving emails from it, send an email to heroku+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
