On Fri, Jan 21, 2011 at 11:41 PM, Cosimo Streppone <cos...@streppone.it>wrote:
> There seems to be two common approaches. One is to just start up a bunch of >> worker processes independently (managed like any other daemon) killing and >> starting more as needed. >> > > For some time I used this approach, with a "fake" init > script like '/etc/init.d/worker' that would start one of them, > so I could run it multiple times to spawn more workers. > > A munin plugin monitored the number of active workers, > and if it detected less than normal, it would spawn a new > one. > > I thought that'd be neat, then I changed my mind. > It worked just fine, but in the end I didn't like > mixing munin plugins with application-related stuff. > What did you change to using? IMO, we (at work) have tended to over-engineer the workers. For example, for a given task we have servers that run via init.d scripts and they fork off a number of threads to handle connections and also fork off a bunch of worker threads that another manager thread allocates work to. It's a complex beast just to run some code. And as a beast if it breaks there's only one or two people that can fix. When the on-call tech gets a call at 2am it's unlikely they would know how to restart specifying more threads. The advantage I see in using separate scripts, like you describe above, is simplicity. If a human was alerted that a queue was getting large then it's just a matter of finding a machine that has headroom and start a few more workers. Any operator can do that. Plus, that's something that could be automated pretty easily with a shell script (and maybe even run via cron). A disadvantage is Perl is not the fastest in startup time (compared to forking, for example), but hopefully starting extra processes would be a rare situation when load changed dramatically. BTW -- Celery is worth a look: http://celeryq.org/ -- Bill Moseley mose...@hank.org
_______________________________________________ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
