Re: How does POE speed up operation in reality?
Quoting Roy M. (setesting...@gmail.com): > > So far the POE::FTP components only allow me to connect to one ftp > server at a time, how can I get the same/similar speed as 8 concurrent > process?). > > In fact, this is a quite common multiple-producer and As per usual in these cases, there is a slight misunderstanding going on, where Rocco knows too much about POE to figure out what elementary knowledge you are missing. It's pretty simple POE, once you get your head around that it is cooperative multitasking of bundles of short events. So, POE::Component::FTP is one such bundle. Now, this bundle of events you can kick off multiple times, it will cooperate with all the other instances. As long as they are not CPU bound, and network task seldomly are, they will happily retrieve all network data "concurrently". So, just set up the component for each server, call poekernel->run and sit back. If you need to, then you can throttle back, having only 10 or 15 servers connected at the same time, adding one more for each finished transfer. Cheers, -- Merijn Broeren | We take risks, we know we take them. Therefore, when things | come out against us, we have no cause for complaint. | - Scott, last journal entry, march 1912
Re: POE not doing what I'm expecting it to......
Quoting Peter Farmer ([EMAIL PROTECTED]):
>
> Now, when I run the script, because of the
> $kernel->delay_set("scheduler",10) at the end of the "sche" sub, I'm
> expecting to see "sche" running at about 1149783936 or 1149783938 or
> as soon as possible after the 10 seconds have passed not 1149783946
> once all the other event "job_runners" are finished.
>
Actually this is what I would expect to happen. You yield 10 slow
events. They all get scheduled to happen at the same time, the current
one. Poor POE tries to do them, but they keep being uncooperative and
not yielding back. Everytime POE gets a look in it sees events it should
have done in the past. What should it do, the old events or the new
'now' event, your alarm? It will do the old stuff first.
The general solution to this is be more cooperative. Be more fine
grained. Instead of this:
foreach my $line (@lines) {
# process $line
}
write something like this:
sub process_line {
# get kernel, heap, @lines could be in the heap
if (@lines) {
unshift $line;
# process $line
$kernel->yield(process_line, @lines)
}
}
or do a few chunks at a time.
Cheers,
--
Merijn Broeren | Sometime in the middle ages, God got fed up with us
Software Geek | and put earth at sol.milky-way.univ in his kill-file.
| Pray all you want, it just gets junked.
0.32 default setting SIGCHLD to IGNORE breaks $?
Hello,
This simple program exhibits the difference in behaviour between 0.32
and 0.31 :
use POE;
my $rc = system("/bin/true");
if ($rc == -1) {
print "failed to execute: $!\n";
} else {
print "executed just fine\n";
}
Reading perldoc perlvar on $?:
If you have installed a signal handler for "SIGCHLD", the value of $?
will usually be wrong outside that handler.
This happens because the import of POE.pm does an eval of use POE::Kernel.
Its import in turn executes a POE::Kernel::->new, which then does
amongst much other stuff a $self->_data_sig_initialize(); Which calls
loop_ignore_signal. This has changed between 0.31 and 0.32 :
@@ -53,6 +53,7 @@
# Child process has stopped.
if ($signal eq 'CHLD' or $signal eq 'CLD') {
+$SIG{$signal} = "DEFAULT";
$self->_data_sig_begin_polling();
return;
}
@@ -72,6 +73,13 @@
if ($signal eq 'CHLD' or $signal eq 'CLD') {
$self->_data_sig_cease_polling();
+$SIG{$signal} = "IGNORE";
+return;
+ }
+
+ if ($signal eq 'PIPE') {
+$SIG{$signal} = "IGNORE";
+return;
}
$SIG{$signal} = "DEFAULT";
So effectivelly all system calls in POE programs that want to check the
result status of their calls cannot do so anymore directly. Or you have
to install a SIGCHLD handler yourself. Much of our code has the above
idiom, where we check immediately. Much unhappiness with 0.32 as it
returns -1 now.
Is this a desired change in behaviour?
Cheers,
--
Merijn Broeren | Sometime in the middle ages, God got fed up with us
Software Geek | and put earth at sol.milky-way.univ in his kill-file.
| Pray all you want, it just gets junked.
POE::Wheel::SocketFactory changes all sockets in the same program to non-blocking
the
# program, the child will die
# prematurely
use IO::Handle;
use File::Spec;
use POSIX qw(dup);
sub new {
my $class = shift;
return bless {};
}
sub _start {
my ($self, $heap, $kernel) = @_[OBJECT, HEAP, KERNEL];
warn "_start\n";
$kernel->sig(CHLD => 'reaper');
$self->{subprocess} = POE::Wheel::Run->new(Program =>
sub {
my $buffer;
my $input_stream = IO::Handle::->new_from_fd(dup(fileno(STDIN)),
"r");
my $output_stream = IO::Handle::->new_from_fd(dup(fileno(STDOUT)),
"w");
my $devnull = File::Spec->devnull();
open(STDIN, "$devnull");
open(STDOUT, "> $devnull");
open(STDERR, "> $devnull");
while (sysread($input_stream, $buffer, 1024 * 32)) {
last if substr($buffer, 0, 4) eq 'kill';
syswrite($output_stream, "child [$$] read: $buffer");
}
}, StdoutEvent => 'output');
warn "have a subprocess\n" if $self->{subprocess};
$heap->{counter} = 3;
$kernel->delay_set('next', 3);
}
sub output {
my ($self, $output) = @_[OBJECT, ARG0];
chomp $output;
warn "received data from subprocess: [$output]\n";
}
sub reaper {
my ($self, $kernel, $heap) = @_[OBJECT, KERNEL, HEAP];
if ($heap->{counter}) {
warn "child has died prematurely\n";
} else {
warn "child has completed when the counter ran out\n";
}
$self->{subprocess} = undef;
$kernel->sig_handled;
}
sub next {
my ($self, $kernel, $heap) = @_[OBJECT, KERNEL, HEAP];
warn "next [" . $heap->{counter}. "]\n";
if ($self->{subprocess}) {
$self->{subprocess}->put("Can you hear me $heap->{counter}");
}
if (--$heap->{counter}) {
$kernel->delay_set('next', 4)
} else {
if ($self->{subprocess}) {
warn "Trying to kill [" . $self->{subprocess}->PID . "]\n";
kill $self->{subprocess}->put("kill");
}
}
}
--
Merijn Broeren | Sometime in the middle ages, God got fed up with us
Software Geek | and put earth at sol.milky-way.univ in his kill-file.
| Pray all you want, it just gets junked.
Re: poe version for perl 5.6.1
Quoting Will Lowe ([EMAIL PROTECTED]): > It looks like POE 0.29 requires Perl 5.8 (due to the need for > Test::More). Anybody know which was the last release of POE that > would work on perl v5.6.1 is? > > The version of POE I've got has a POE::Wheel::Run that shares IO > drivers for stdout and stderr, and I want to see if a version that > uses seperate drivers will fix some problems I'm having. > You can just install Test::More from CPAN. POE works fine with perl 5.6. Cheers, -- Merijn Broeren | She doesn't want you to understand her. She knows that's Software Geek | impossible. She just wants you to understand yourself, | everything else is negotiable.
Re: POE::Wheel::Run on Win32, patch attached
Quoting Arthur Bergman ([EMAIL PROTECTED]): > > Knowing parts of the internals that the fake process environment uses I > would strongly recommending on just using CreateProcess ;-). It is also > quite wasteful to use fork() for this since unlike unix it will > actually copy all your memory over (for now [1]). > But we cannot use CreateProcess with a coderef, at least I don't see how. The issue is with a coderef, what to do when it dies, we want to kill of the thread but not fire destructors. Kill will work, but maybe leave memory around. When trying with an END block I had it fire only once. Mmmmm. Cheers, -- Merijn Broeren | She doesn't want you to understand her. She knows that's Software Geek | impossible. She just wants you to understand yourself, | everything else is negotiable.
Re: POE::Wheel::Run on Win32, patch attached
Quoting Arthur Bergman ([EMAIL PROTECTED]): > On 13 Apr 2005, at 04:53, Rocco Caputo wrote: > > >1. The exit() in POE::Wheel::Run doesn't bypass "forked" object > >destruction. With ASSERT_DEFAULT on (the default for POE's tests), a > >lot of "this thing leaked" messages show up. They're harmless, more > >or less, but object destructors are still being called in the "child" > >process. > > > >Is there any way to exit from the child "process" without triggering > >all the object destructors that were forked into it? > > > > kill 9,$$; :-) > Did you fix that then? The perlfork manpage makes it sound like there will be leaks all over the place. Cheers, -- Merijn Broeren | She doesn't want you to understand her. She knows that's Software Geek | impossible. She just wants you to understand yourself, | everything else is negotiable.
Re: POE::Wheel::Run on Win32, patch attached
Quoting Rocco Caputo ([EMAIL PROTECTED]):
> How's that going? I've been tracking the problem down on and off
> today, but no luck. Some notes:
>
> 1. The exit() in POE::Wheel::Run doesn't bypass "forked" object
> destruction. With ASSERT_DEFAULT on (the default for POE's tests), a
> lot of "this thing leaked" messages show up. They're harmless, more
> or less, but object destructors are still being called in the "child"
> process.
>
We've discussed that quite extensively and it seems the right thing to
do. Looking at the code:
unless (POE::Kernel::RUNNING_IN_HELL) {
eval { POSIX::_exit(0); };
eval { kill KILL => $$; };
eval { exec("$^X -e 0"); };
};
exit(0);
We don't want to do the first eval because _exit will kill all threads.
Fork on windows is implemented by starting a new thread with a cloned
interpreter. Kill will work, as it will kill a thread outright. But the
docs have dire warnings against its usage:
kill() kill() can be used to terminate a pseudo-process by
passing it the ID returned by fork(). This should
not be used except under dire circumstances, because
the operating system may not guarantee integrity of
the process resources when a running thread is
terminated. Note that using kill() on a
pseudo-process() may typically cause memory leaks,
because the thread that implements the pseudo-
process does not get a chance to clean up its
resources.
So that is not a good choice. The third eval is not useful on Windows,
as it will do a CreateProcess, which is slow, and then immediately
exit. Better then to do a normal exit.
The destructors are indeed called, maybe this should just be documented,
that you need to take care in your DESTROY.
> Is there any way to exit from the child "process" without triggering
> all the object destructors that were forked into it?
>
Not that I know of. Which doesn't mean much :-) Should ask Gurusamy.
> 2. The {STDIN,STDOUT,STDERR}_FILENO checks are probably important. If
> you notice just below the comment "Close any close-on-exec file
> descriptors.", files with high descriptor numbers are closed. Maybe
> the explicit checks can be removed if the loop is changed to ignore
> fileno(STDIN), fileno(STDOUT) and fileno(STDERR) rather than mimic
> UNIX's close-on-exec behavior.
>
That makes sense. The loop should close all but the fileno ones.
> 3. tests/30_loops/50_tk/wheel_run.t passes its tests when run
> stand-alone. To re-create the failure deterministically, add this
> code to test.pl just before runtests() is called:
>
> @test_files = grep /50/, @test_files;
> while (@test_files) {
> last if $test_files[-1] =~ /50.*tail/;
> pop @test_files;
> }
>
I only now realize that you are seeing this with Tk. So the normal
wheel_run.t passes? Good. For me the Tk test hangs. I will investigate.
I'm using Tk 804.027. Which one come with ActiveState.
Regards,
--
Merijn Broeren | She doesn't want you to understand her. She knows that's
Software Geek | impossible. She just wants you to understand yourself,
| everything else is negotiable.
Re: POE::Wheel::Run on Win32, patch attached
Quoting Rocco Caputo ([EMAIL PROTECTED]): > > This is some amazing work. Thank you. > Well, needs must and all that. > I applied the patches and tried them on FreeBSD and ActivePerl > 5.8.6.811. After some minor tweaks, they work! There's only one > small problem while testing on ActivePerl: > > tests\30_loops\50_tk\wheel_run..dubious > Test returned status 5 (wstat 1280, 0x500) > after all the subtests completed successfully > Weird. I don't get that ofcourse. > [...] > > Failed Test Stat Wstat Total Fail Failed List of > Failed > > --- > tests\30_loops\50_tk\wheel_run.t5 1280240 0.00% ?? > 54 tests and 22 subtests skipped. > Failed 1/126 test scripts, 99.21% okay. 0/3664 subtests failed, 100.00% > okay. > NMAKE : fatal error U1077: 'C:\WINNT\system32\cmd.exe' : return code '0xff' > Stop. > > This is really odd since the test program explicitly calls C > at its end. I've verified that it gets that far, too. > > I've committed the changes anyway. You can check them out and check > out the results yourself. The error will need to be cleared up > somehow before I release it to the CPAN. > It is very odd indeed. I'll see what happens on my home windows machine when I install ActiveState there. Cheers, -- Merijn Broeren | She doesn't want you to understand her. She knows that's Software Geek | impossible. She just wants you to understand yourself, | everything else is negotiable.
Re: POE::Wheel::Run on Win32, patch attached
Quoting Merijn Broeren ([EMAIL PROTECTED]): > Hi, > > My collegue Nick Williams and I had a stab at making POE::Wheel::Run Ahum, my colleague pointed out that I spelled that wrong :-) In addition: diff -u -r 0.3009/src/distro/mylib/Makefile-5005.pm 0.3009-ms1/src/distro/mylib/Makefile-5005.pm --- 0.3009/src/distro/mylib/Makefile-5005.pm2004-11-26 21:34:42.0 + +++ 0.3009-ms1/src/distro/mylib/Makefile-5005.pm2005-04-11 16:22:08.0 +0100 @@ -93,6 +93,11 @@ -default => 0, 'Event' => '1.00', ], + "Optional modules for Windows POE::Wheel::Run support." => [ + -default => 0, + 'Win32API::File' => '0.05', + 'Win32::Console' => '0.031', + ], ); It appears there is new life for the libwin32 package, so I'll see if I can get a patch to make SetStdHandle available somewhere more rational, like in Win32API::File. Cheers, -- Merijn Broeren | She doesn't want you to understand her. She knows that's Software Geek | impossible. She just wants you to understand yourself, | everything else is negotiable.
POE::Wheel::Run on Win32, patch attached
Hi,
My collegue Nick Williams and I had a stab at making POE::Wheel::Run
work on Win32. The attached patch makes it work with both code refs
and external programs, all regression tests pass, after adjusting the
wheel_run.t to not die anymore and use the correct commandline
quotation.
The patch solves two issues with fork+exec on windows. Firstly it
appears the stdio handles are not separated properly after the fork, so
reopening these in the child onto the Pipe::TwoWay handles actually
changes them in the parent as well. First closing them works, after that
the CODE subroutine ref branch is working properly.
The exec is at that point not working yet because the internal Perl
implementation of fork+exec on Windows remembers the underlying Windows
handles of stdio at the time of the fork to use later for the exec. To
repoint these you need the OS call 'SetStdHandle'. Even though this is
used in the perl core, it is not exposed to the Perl level. It is also
not exposed through Win32API::File, which does have FdGetOsFHandle,
necessary to get the Windows native file handles associated with the
Pipe::TwoWay.
We were stuck at this point, until we found that Win32::Console uses
_SetStdHandle internally and unwittingly exposes it. So we pull both of
these modules in conditionally and use them. If there is a better way of
getting at SetStdHandle we would like to know.
Also this match makes use of POE::Kernel::RUNNING_IN_HELL, which is
changed to be a compile time installed constant subroutine. If that is
deemed micro optimization or just plain wrong to use cross package
like that, please adjust accordingly. I've been restrained from
renaming it :-)
Comments and most definitely testing very welcome.
Cheers,
--
Merijn Broeren | She doesn't want you to understand her. She knows that's
Software Geek | impossible. She just wants you to understand yourself,
| everything else is negotiable.
diff -u -r 0.3009/src/distro/lib/POE/Kernel.pm
0.3009-ms1/src/distro/lib/POE/Kernel.pm
--- 0.3009/src/distro/lib/POE/Kernel.pm 2004-12-02 15:41:54.0 +
+++ 0.3009-ms1/src/distro/lib/POE/Kernel.pm 2005-04-08 15:26:09.0
+0100
@@ -58,11 +58,19 @@
#--
# Perform some optional setup.
-sub RUNNING_IN_HELL () { $^O eq 'MSWin32' }
BEGIN {
local $SIG{'__DIE__'} = 'DEFAULT';
+ {
+no strict 'refs';
+if ($^O eq 'MSWin32') {
+*{ __PACKAGE__ . '::RUNNING_IN_HELL' } = sub { 1 };
+} else {
+*{ __PACKAGE__ . '::RUNNING_IN_HELL' } = sub { 0 };
+}
+ }
+
# POE runs better with Time::HiRes, but it also runs without it.
{ no strict 'refs';
diff -u -r 0.3009/src/distro/lib/POE/Wheel/Run.pm
0.3009-ms1/src/distro/lib/POE/Wheel/Run.pm
--- 0.3009/src/distro/lib/POE/Wheel/Run.pm 2004-12-21 23:15:09.0
+
+++ 0.3009-ms1/src/distro/lib/POE/Wheel/Run.pm 2005-04-11 14:42:36.0
+0100
@@ -17,7 +17,6 @@
BEGIN {
die "$^O does not support fork()\n" if $^O eq 'MacOS';
- die "$^O does not fully support fork+exec\n" if $^O eq 'MSWin32';
local $SIG{'__DIE__'} = 'DEFAULT';
eval{ require IO::Pty; };
@@ -27,6 +26,16 @@
eval 'sub PTY_AVAILABLE () { 1 }';
}
+ if (POE::Kernel::RUNNING_IN_HELL) {
+ eval{ require Win32::Console; };
+ if ($@) { die "Running on Win32, need Win32::Console but failed to
load:\n$@" }
+ else{ Win32::Console->import(); };
+
+ eval{ require Win32API::File; };
+ if ($@) { die "Running on Win32, need Win32API::File but failed to
load:\n$@" }
+ else{ Win32API::File->import( qw(FdGetOsFHandle) ); };
+ }
+
# How else can I get them out?!
if (eval '&IO::Tty::Constant::TIOCSCTTY') {
*TIOCSCTTY = *IO::Tty::Constant::TIOCSCTTY;
@@ -380,6 +389,11 @@
close $stdin_write;
close $stdout_read;
close $stderr_read if defined $stderr_read;
+
+# Need to close on Win32 because std handles aren't dup'ed, no harm
elsewhere
+# Close STDERR later to not influence possible die
+close STDIN;
+close STDOUT;
# Redirect STDIN from the read end of the stdin pipe.
open( STDIN, "<&" . fileno($stdin_read) )
@@ -389,6 +403,9 @@
open( STDOUT, ">&" . fileno($stdout_write) )
or die "can't redirect stdout in child pid $$: $!";
+# Need to close on Win32 because std handles aren't dup'ed, no harm
elsewhere
+close STDERR;
+
# Redirect STDERR to the write end of the stderr pipe. If the
# stderr pipe's undef, then we use STDOUT.
open( STDERR, ">&" . fileno($stderr_write) )
@@ -399,16 +416,22 @@
select STDOUT;
