Execute Jobs parallel using JobQueue
Hello,
Consider my code below would like to execute the sleep() function in
parallel, using POE JobQueue component:
#=
use strict;
use POE qw(Component::JobQueue);
# Passive queue waits for enqueue events.
POE::Component::JobQueue->spawn(
Alias => 'passive',
WorkerLimit => 16,
Worker => \&spawn_a_worker,
Passive => {
Prioritizer => sub { 1 }
}
);
sub spawn_a_worker {
my ( $postback, @job_params ) = @_;
POE::Session->create(
inline_states => {
_start => \&start,
sleep => \&sleep
},
args => [
$postback,
@job_params,
],
);
}
POE::Session->create(
inline_states => {
_start => \&init,
_stop => sub { print "END" }
}
);
POE::Kernel->run();
exit(0);
sub init {
my $kernel = $_[KERNEL];
foreach ( 1 .. 10 ) {
$kernel->post( passive => enqueue => response => $_ );
}
}
sub start {
my $kernel = $_[ KERNEL ];
print "Starting\n";
$kernel->yield("sleep");
}
sub sleep {
print "Sleeping for 5 seconds\n";
sleep 5;
}
#==
It seems that tasks still block each other? If not, I would have all
message executed at the end of 5 seconds.
Re: Execute Jobs parallel using JobQueue
Good day, Ryan,
The problem with using 'sleep' in POE code is that POE isn't true
pre-emptive multi-tasking, but rather cooperative multitasking at the
process level. Different sessions are still running in the same
process/thread, so any actions that will monopolize CPU time for a
stretch will block all POE tasks. The general advice for POE tasks is
that they should do no blocking calls in any individual event
handling, in order to allow other sessions to run when the current one
is idle.
The best way to have a session delay itself for a time is one of the
methods described in
http://search.cpan.org/~rcaputo/POE-1.280/lib/POE/Kernel.pm#Timer_Events_%28Delayed_Messages%29
.
Take care,
Mark.
On Tue, Dec 1, 2009 at 4:38 PM, Ryan Chan wrote:
> Hello,
>
> Consider my code below would like to execute the sleep() function in
> parallel, using POE JobQueue component:
>
>
> #=
> use strict;
> use POE qw(Component::JobQueue);
>
> # Passive queue waits for enqueue events.
> POE::Component::JobQueue->spawn(
> Alias => 'passive',
> WorkerLimit => 16,
> Worker => \&spawn_a_worker,
> Passive => {
> Prioritizer => sub { 1 }
> }
> );
>
> sub spawn_a_worker {
> my ( $postback, @job_params ) = @_;
> POE::Session->create(
> inline_states => {
> _start => \&start,
> sleep => \&sleep
> },
> args => [
> $postback,
> �...@job_params,
> ],
> );
> }
>
> POE::Session->create(
> inline_states => {
> _start => \&init,
> _stop => sub { print "END" }
> }
> );
> POE::Kernel->run();
> exit(0);
>
> sub init {
> my $kernel = $_[KERNEL];
>
> foreach ( 1 .. 10 ) {
> $kernel->post( passive => enqueue => response => $_ );
> }
> }
>
> sub start {
> my $kernel = $_[ KERNEL ];
> print "Starting\n";
> $kernel->yield("sleep");
> }
>
> sub sleep {
> print "Sleeping for 5 seconds\n";
> sleep 5;
> }
>
> #==
>
> It seems that tasks still block each other? If not, I would have all
> message executed at the end of 5 seconds.
>
Re: Execute Jobs parallel using JobQueue
The other general advice is to use fork(), with or without POE, when
you need true parallelism.
--
Rocco Caputo - rcap...@pobox.com
On Dec 1, 2009, at 08:50, Mark Morgan wrote:
Good day, Ryan,
The problem with using 'sleep' in POE code is that POE isn't true
pre-emptive multi-tasking, but rather cooperative multitasking at the
process level. Different sessions are still running in the same
process/thread, so any actions that will monopolize CPU time for a
stretch will block all POE tasks. The general advice for POE tasks is
that they should do no blocking calls in any individual event
handling, in order to allow other sessions to run when the current one
is idle.
The best way to have a session delay itself for a time is one of the
methods described in
http://search.cpan.org/~rcaputo/POE-1.280/lib/POE/Kernel.pm#Timer_Events_%28Delayed_Messages%29
.
Take care,
Mark.
On Tue, Dec 1, 2009 at 4:38 PM, Ryan Chan
wrote:
Hello,
Consider my code below would like to execute the sleep() function in
parallel, using POE JobQueue component:
#=
use strict;
use POE qw(Component::JobQueue);
# Passive queue waits for enqueue events.
POE::Component::JobQueue->spawn(
Alias => 'passive',
WorkerLimit => 16,
Worker => \&spawn_a_worker,
Passive => {
Prioritizer => sub { 1 }
}
);
sub spawn_a_worker {
my ( $postback, @job_params ) = @_;
POE::Session->create(
inline_states => {
_start => \&start,
sleep => \&sleep
},
args => [
$postback,
@job_params,
],
);
}
POE::Session->create(
inline_states => {
_start => \&init,
_stop => sub { print "END" }
}
);
POE::Kernel->run();
exit(0);
sub init {
my $kernel = $_[KERNEL];
foreach ( 1 .. 10 ) {
$kernel->post( passive => enqueue => response => $_ );
}
}
sub start {
my $kernel = $_[ KERNEL ];
print "Starting\n";
$kernel->yield("sleep");
}
sub sleep {
print "Sleeping for 5 seconds\n";
sleep 5;
}
#==
It seems that tasks still block each other? If not, I would have all
message executed at the end of 5 seconds.
Re: Execute Jobs parallel using JobQueue
On Wed, Dec 2, 2009 at 5:31 AM, Rocco Caputo wrote: > The other general advice is to use fork(), with or without POE, when you > need true parallelism. > It seems POE::Component::Pool::Thread can solve the problem? (But I was unable to install via CPAN in ubuntu) If I goes with the fork solution, any abstraction recommended? e.g. Parallel::Iterator Parallel::ForkManager Proc::Queue or others? Thanks.
Re: Execute Jobs parallel using JobQueue
On Wed, 2 Dec 2009 12:42:34 +0800 Ryan Chan wrote: > If I goes with the fork solution, any abstraction recommended? I don't want to scare you off, but I can also suggest POEx::WorkerPool. -- Nicholas Perez XMPP/Email: n...@nickandperla.net http://search.cpan.org/~nperez/ http://github.com/nperez
Re: Execute Jobs parallel using JobQueue
On Wed, Dec 2, 2009 at 2:56 PM, Nick Perez wrote: > On Wed, 2 Dec 2009 12:42:34 +0800 > Ryan Chan wrote: > >> If I goes with the fork solution, any abstraction recommended? > > > I don't want to scare you off, but I can also suggest POEx::WorkerPool. > > -- unfortunately, make failed in CPAN in ubuntu 8.10
Re: Execute Jobs parallel using JobQueue
On Thu, 3 Dec 2009 22:37:37 +0800 Ryan Chan wrote: > On Wed, Dec 2, 2009 at 2:56 PM, Nick Perez > wrote: > > On Wed, 2 Dec 2009 12:42:34 +0800 > > Ryan Chan wrote: > > > >> If I goes with the fork solution, any abstraction recommended? > > > > > > I don't want to scare you off, but I can also suggest > > POEx::WorkerPool. > > > > -- > > unfortunately, make failed in CPAN in ubuntu 8.10 > Can I get the failure output? What version of Perl? Also, why and operating system so ancient? -- Nicholas Perez XMPP/Email: n...@nickandperla.net http://search.cpan.org/~nperez/ http://github.com/nperez
