I mentioned it here and there, but now that the code is on CPAN, I'd like to announce it here.
There are a lot of modules on CPAN that create pools of worker processes, but all that I have seen have a fatal flaw: they use fork at runtime (and from perl) and do processing in the child. fork doesn't work safely in programs using pthreads, which is almost every program I write these days, and forking from a running program suffers form other hazards, such as having ltos of event watchers for things that only make sense in one of the process copies. So here's my take on the problem: AnyEvent::Fork (http://software.schmorp.de/pkg/AnyEvent-Fork) This module execs a new perl interpreter in a safe way, avoiding both the fork-in-perl problem and fork-copies-everything problems. It is also surprisingly fast - despite the overhead, creating a process via AnyEvent::Fork is often cheaper than using fork+exec, as fork has to make a (virtual) copy of your whole process, and with AnyEvent::Fork you can usualy make your copies from smaller processes that only contain the code that you really need. Small example: AnyEvent::Fork ->new ->require ("MyModule") ->run ("MyModule::server", my $cv = AE::cv); my $fh = $cv->recv; The module doesn't, however, allow you to talk with the new processes, it only allows you to create them and run your code. To talk with them, there is: AnyEvent::Fork::RPC (http://software.schmorp.de/pkg/AnyEvent-Fork-RPC) This module takes a process created with AnyEvent::Fork, and adds simple RPC functionality, by running a simple server and giving you a code reference that you can use to call a function in the other process. It has a synchronous and an asynchronous backend (the async backend uses AnyEvent and can handle multiple RPC calls in parallel). Supports any serialiser you want. Small example: my $rpc = AnyEvent::Fork ->new ->require ("MyModule") ->AnyEvent::Fork::RPC::run ( "MyModule::server", ); $rpc->(1, 2, 3, sub { print "MyModule::server returned @_\n"; }); It doesn't, however, create some pool of processes for you or anything like that, which is why there is also: AnyEvent::Fork::Pool (http://software.schmorp.de/pkg/AnyEvent-Fork-Pool) This uses a process created via AnyEvent::Fork as a template process to create multiple worker processes. The pool of processes resizes dynamically according to load and allows of variety of configurations. Small example: my $pool = AnyEvent::Fork ->new ->require ("MyWorker") ->AnyEvent::Fork::Pool::run ( "MyWorker::run", max => 4, ); for (1..10) { $pool->(doit => $_, sub { print "MyWorker::run returned @_\n"; }); } Right now, AnyEvent::Fork seems to look pretty stable, and could use some real-world testing. AnyEvent::Fork::RPC is more like early beta, and AnyEvent::Fork::Pool is more like alpha quality at this point. The API might change if I find a bug that requires changing it, which is not at all unlikely at the moment. The modules are also quite small (sloccount says all three together are 604 loc), and since they are completely seperate distributions on CPAN, you only pay for what you actually use. Even windows us supported, to the extent possible (cygwin doesn't support any kind of handle-open or fd passing, and strawberry/activeperl of course are not getting better with time, and require the usual silly workarounds from your side). My idea is not that everybody should go and start using AnyEvent::Fork::Pool - you are invited to do your own pools using AnyEvent::Fork::RPC. Or even ignore AnyEvent::Fork::RPC, and just use AnyEvent::Fork to create processes and do whatever you want with them. It would however, be nice if more modules would use AnyEvent::Fork instead of some home-grown fork-and-hack-around-or-ignore-problems approach. Mostly new modules, as the design of AnyEvent::Fork makes it unsuitable as drop-in replacement for existing process pool managers. However, modules such as AnyEvent::DBI could easily switch to AnyEvent::Fork, or AnyEvent::Fork::RPC, as they don't expose the implementation. And the CVS version of Gtk2::CV already uses AnyEvent::Fork, got a lot faster, and wastes a lot less memory. In any case, as always, I release my code in the hope that is is useful to somebody, and can be safely ignored by the rest :) -- The choice of a Deliantra, the free code+content MORPG -----==- _GNU_ http://www.deliantra.net ----==-- _ generation ---==---(_)__ __ ____ __ Marc Lehmann --==---/ / _ \/ // /\ \/ / [email protected] -=====/_/_//_/\_,_/ /_/\_\ _______________________________________________ anyevent mailing list [email protected] http://lists.schmorp.de/mailman/listinfo/anyevent
