Execute Jobs parallel using JobQueue
Hello, Consider my code below would like to execute the sleep() function in parallel, using POE JobQueue component: #= use strict; use POE qw(Component::JobQueue); # Passive queue waits for enqueue events. POE::Component::JobQueue->spawn( Alias => 'passive', WorkerLimit => 16, Worker => \&spawn_a_worker, Passive => { Prioritizer => sub { 1 } } ); sub spawn_a_worker { my ( $postback, @job_params ) = @_; POE::Session->create( inline_states => { _start => \&start, sleep => \&sleep }, args => [ $postback, @job_params, ], ); } POE::Session->create( inline_states => { _start => \&init, _stop => sub { print "END" } } ); POE::Kernel->run(); exit(0); sub init { my $kernel = $_[KERNEL]; foreach ( 1 .. 10 ) { $kernel->post( passive => enqueue => response => $_ ); } } sub start { my $kernel = $_[ KERNEL ]; print "Starting\n"; $kernel->yield("sleep"); } sub sleep { print "Sleeping for 5 seconds\n"; sleep 5; } #== It seems that tasks still block each other? If not, I would have all message executed at the end of 5 seconds.
Re: Execute Jobs parallel using JobQueue
Good day, Ryan, The problem with using 'sleep' in POE code is that POE isn't true pre-emptive multi-tasking, but rather cooperative multitasking at the process level. Different sessions are still running in the same process/thread, so any actions that will monopolize CPU time for a stretch will block all POE tasks. The general advice for POE tasks is that they should do no blocking calls in any individual event handling, in order to allow other sessions to run when the current one is idle. The best way to have a session delay itself for a time is one of the methods described in http://search.cpan.org/~rcaputo/POE-1.280/lib/POE/Kernel.pm#Timer_Events_%28Delayed_Messages%29 . Take care, Mark. On Tue, Dec 1, 2009 at 4:38 PM, Ryan Chan wrote: > Hello, > > Consider my code below would like to execute the sleep() function in > parallel, using POE JobQueue component: > > > #= > use strict; > use POE qw(Component::JobQueue); > > # Passive queue waits for enqueue events. > POE::Component::JobQueue->spawn( > Alias => 'passive', > WorkerLimit => 16, > Worker => \&spawn_a_worker, > Passive => { > Prioritizer => sub { 1 } > } > ); > > sub spawn_a_worker { > my ( $postback, @job_params ) = @_; > POE::Session->create( > inline_states => { > _start => \&start, > sleep => \&sleep > }, > args => [ > $postback, > �...@job_params, > ], > ); > } > > POE::Session->create( > inline_states => { > _start => \&init, > _stop => sub { print "END" } > } > ); > POE::Kernel->run(); > exit(0); > > sub init { > my $kernel = $_[KERNEL]; > > foreach ( 1 .. 10 ) { > $kernel->post( passive => enqueue => response => $_ ); > } > } > > sub start { > my $kernel = $_[ KERNEL ]; > print "Starting\n"; > $kernel->yield("sleep"); > } > > sub sleep { > print "Sleeping for 5 seconds\n"; > sleep 5; > } > > #== > > It seems that tasks still block each other? If not, I would have all > message executed at the end of 5 seconds. >
Re: Execute Jobs parallel using JobQueue
The other general advice is to use fork(), with or without POE, when you need true parallelism. -- Rocco Caputo - rcap...@pobox.com On Dec 1, 2009, at 08:50, Mark Morgan wrote: Good day, Ryan, The problem with using 'sleep' in POE code is that POE isn't true pre-emptive multi-tasking, but rather cooperative multitasking at the process level. Different sessions are still running in the same process/thread, so any actions that will monopolize CPU time for a stretch will block all POE tasks. The general advice for POE tasks is that they should do no blocking calls in any individual event handling, in order to allow other sessions to run when the current one is idle. The best way to have a session delay itself for a time is one of the methods described in http://search.cpan.org/~rcaputo/POE-1.280/lib/POE/Kernel.pm#Timer_Events_%28Delayed_Messages%29 . Take care, Mark. On Tue, Dec 1, 2009 at 4:38 PM, Ryan Chan wrote: Hello, Consider my code below would like to execute the sleep() function in parallel, using POE JobQueue component: #= use strict; use POE qw(Component::JobQueue); # Passive queue waits for enqueue events. POE::Component::JobQueue->spawn( Alias => 'passive', WorkerLimit => 16, Worker => \&spawn_a_worker, Passive => { Prioritizer => sub { 1 } } ); sub spawn_a_worker { my ( $postback, @job_params ) = @_; POE::Session->create( inline_states => { _start => \&start, sleep => \&sleep }, args => [ $postback, @job_params, ], ); } POE::Session->create( inline_states => { _start => \&init, _stop => sub { print "END" } } ); POE::Kernel->run(); exit(0); sub init { my $kernel = $_[KERNEL]; foreach ( 1 .. 10 ) { $kernel->post( passive => enqueue => response => $_ ); } } sub start { my $kernel = $_[ KERNEL ]; print "Starting\n"; $kernel->yield("sleep"); } sub sleep { print "Sleeping for 5 seconds\n"; sleep 5; } #== It seems that tasks still block each other? If not, I would have all message executed at the end of 5 seconds.
Re: Execute Jobs parallel using JobQueue
On Wed, Dec 2, 2009 at 5:31 AM, Rocco Caputo wrote: > The other general advice is to use fork(), with or without POE, when you > need true parallelism. > It seems POE::Component::Pool::Thread can solve the problem? (But I was unable to install via CPAN in ubuntu) If I goes with the fork solution, any abstraction recommended? e.g. Parallel::Iterator Parallel::ForkManager Proc::Queue or others? Thanks.
Re: Execute Jobs parallel using JobQueue
On Wed, 2 Dec 2009 12:42:34 +0800 Ryan Chan wrote: > If I goes with the fork solution, any abstraction recommended? I don't want to scare you off, but I can also suggest POEx::WorkerPool. -- Nicholas Perez XMPP/Email: n...@nickandperla.net http://search.cpan.org/~nperez/ http://github.com/nperez
Re: Execute Jobs parallel using JobQueue
On Wed, Dec 2, 2009 at 2:56 PM, Nick Perez wrote: > On Wed, 2 Dec 2009 12:42:34 +0800 > Ryan Chan wrote: > >> If I goes with the fork solution, any abstraction recommended? > > > I don't want to scare you off, but I can also suggest POEx::WorkerPool. > > -- unfortunately, make failed in CPAN in ubuntu 8.10
Re: Execute Jobs parallel using JobQueue
On Thu, 3 Dec 2009 22:37:37 +0800 Ryan Chan wrote: > On Wed, Dec 2, 2009 at 2:56 PM, Nick Perez > wrote: > > On Wed, 2 Dec 2009 12:42:34 +0800 > > Ryan Chan wrote: > > > >> If I goes with the fork solution, any abstraction recommended? > > > > > > I don't want to scare you off, but I can also suggest > > POEx::WorkerPool. > > > > -- > > unfortunately, make failed in CPAN in ubuntu 8.10 > Can I get the failure output? What version of Perl? Also, why and operating system so ancient? -- Nicholas Perez XMPP/Email: n...@nickandperla.net http://search.cpan.org/~nperez/ http://github.com/nperez