Re: Apache2::SubProcess subprocess not subprocessing properly
Tosh,
On Tuesday 09 February 2010 14:21:57 Tosh Cooey wrote:
> Hi after much trial and all error I am seeing that the browser
> connection closing is also stopping my subprocess.
>
I don't know what you are trying to achieve and I don't use
Apache2::Subprocess much. What I do to fork off processes instead is this:
my $close_fd=sub {
use POSIX ();
my %save=(2=>1); # keep STDERR
undef @sa...@_} if( @_ );
opendir my $d, "/proc/self/fd" or do {
warn "Cannot open directory /proc/self/fd: $!\n";
POSIX::_exit -1; # try this first to avoid buffer flushing
CORE::exit -1;
};
while (defined(my $fd=readdir $d)) {
next unless $fd=~/^\d+$/;
POSIX::close $fd unless exists $save{$fd};
}
};
my $spawn=sub {
use POSIX ();
my ($daemon_should_survive_apache_restart, @args)=...@_;
local $SIG{CHLD}='IGNORE';
my $pid;
# yes, even fork can fail
select undef, undef, undef, .1 while( !defined($pid=fork) );
unless( $pid ) {# child
# 2nd fork to cut parent relationship with a mod_perl apache
select undef, undef, undef, .1 while( !defined($pid=fork) );
if( $pid ) {
POSIX::_exit 0;
CORE::exit 0;
} else {
if( ref($daemon_should_survive_apache_restart) ) {
$close_fd->($daemon_should_survive_apache_restart->{fd});
POSIX::setsid if( $daemon_should_survive_apache_restart->{survive} );
} else {
$close_fd->();
POSIX::setsid if( $daemon_should_survive_apache_restart );
}
if( 'CODE' eq ref $args[0] ) {
my $f=shift @args;
# TODO: restore %ENV and exit() behavior
eval {$f->(@args)};
CORE::exit 0;
} else {
{exec @args;} # extra block to suppress a warning
POSIX::_exit -1;
CORE::exit -1;
}
}
}
waitpid $pid, 0;# avoid a zombie on some OS
};
And this is how to use it:
$spawn->({survive=>1}, sub {
my ($n, $sleep)=...@_;
for(1..$n) {
warn "sleeping $sleep\n";
sleep $sleep;
}
warn "Done\n";
}, 10, 2 );
or
$spawn->
(undef, # don't survive apache stop
qw/bash -c/,'for (( i=0; i<10; i++ )); do echo "opi" >&2; sleep 2; done')
The first argument to $spawn is an options hash with 2 possible options:
survive=>BOOLEAN# should the subprocess survive an apache stop or not
fd=>\...@list_of_open_file_descriptors_to_preserve
Normally the $spawn will close all files save for STDERR. With the fd option
one can preserve other files such as database connections or other.
With the survive option true the child process will be the leader of a new
process group.
NOTES:
- The $close_fd function is Linux-specific. I couldn't think of a really
portable (at least across UNICES) way to get all open file descriptors of the
current process.
- The waitpid at the end of $spawn is not necessary under Linux since
$SIG{CHLD}='IGNORE' already takes care of zombies. But other systems need it.
- Keep in mind that %ENV and exit() work different under mod_perl. If you pass
a subroutine this behavior remains.
Torsten
Re: Apache2::SubProcess subprocess not subprocessing properly
You aren't going to like this, and maybe what you're doing is for a
really complex situation, but it really shouldn't be so complex to just
spawn off a long running process that you don't care to ever hear back
from, honestly.
Tosh
Torsten Förtsch wrote:
Tosh,
On Tuesday 09 February 2010 14:21:57 Tosh Cooey wrote:
Hi after much trial and all error I am seeing that the browser
connection closing is also stopping my subprocess.
I don't know what you are trying to achieve and I don't use
Apache2::Subprocess much. What I do to fork off processes instead is this:
my $close_fd=sub {
use POSIX ();
my %save=(2=>1); # keep STDERR
undef @sa...@_} if( @_ );
opendir my $d, "/proc/self/fd" or do {
warn "Cannot open directory /proc/self/fd: $!\n";
POSIX::_exit -1; # try this first to avoid buffer flushing
CORE::exit -1;
};
while (defined(my $fd=readdir $d)) {
next unless $fd=~/^\d+$/;
POSIX::close $fd unless exists $save{$fd};
}
};
my $spawn=sub {
use POSIX ();
my ($daemon_should_survive_apache_restart, @args)=...@_;
local $SIG{CHLD}='IGNORE';
my $pid;
# yes, even fork can fail
select undef, undef, undef, .1 while( !defined($pid=fork) );
unless( $pid ) {# child
# 2nd fork to cut parent relationship with a mod_perl apache
select undef, undef, undef, .1 while( !defined($pid=fork) );
if( $pid ) {
POSIX::_exit 0;
CORE::exit 0;
} else {
if( ref($daemon_should_survive_apache_restart) ) {
$close_fd->($daemon_should_survive_apache_restart->{fd});
POSIX::setsid if( $daemon_should_survive_apache_restart->{survive} );
} else {
$close_fd->();
POSIX::setsid if( $daemon_should_survive_apache_restart );
}
if( 'CODE' eq ref $args[0] ) {
my $f=shift @args;
# TODO: restore %ENV and exit() behavior
eval {$f->(@args)};
CORE::exit 0;
} else {
{exec @args;} # extra block to suppress a warning
POSIX::_exit -1;
CORE::exit -1;
}
}
}
waitpid $pid, 0;# avoid a zombie on some OS
};
And this is how to use it:
$spawn->({survive=>1}, sub {
my ($n, $sleep)=...@_;
for(1..$n) {
warn "sleeping $sleep\n";
sleep $sleep;
}
warn "Done\n";
}, 10, 2 );
or
$spawn->
(undef, # don't survive apache stop
qw/bash -c/,'for (( i=0; i<10; i++ )); do echo "opi" >&2; sleep 2; done')
The first argument to $spawn is an options hash with 2 possible options:
survive=>BOOLEAN# should the subprocess survive an apache stop or not
fd=>\...@list_of_open_file_descriptors_to_preserve
Normally the $spawn will close all files save for STDERR. With the fd option
one can preserve other files such as database connections or other.
With the survive option true the child process will be the leader of a new
process group.
NOTES:
- The $close_fd function is Linux-specific. I couldn't think of a really
portable (at least across UNICES) way to get all open file descriptors of the
current process.
- The waitpid at the end of $spawn is not necessary under Linux since
$SIG{CHLD}='IGNORE' already takes care of zombies. But other systems need it.
- Keep in mind that %ENV and exit() work different under mod_perl. If you pass
a subroutine this behavior remains.
Torsten
--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Apache2::SubProcess sucks
Maybe it does or doesn't work under MP::R, the answer to that is perhaps
individual.
Anyway, the solution, at least so far until I run into other problems,
seems to be to just make a system() call and the called program uses
Proc::Daemon and things *seem* to work fine in testing, we'll see when
it hits production...
Tosh
Fred Moyer wrote:
I haven't been following this thread too closely, but is there a
reason you aren't using PerlHandlers instead of ModPerl::Registry?
MP::R is meant mostly for CGI migration, so it is probably not widely
tested with Apache2::SubProcess calls. I'd suggest trying your
external call under a PerlHandler instead and see if that works.
On Fri, Feb 19, 2010 at 1:52 PM, Tosh Cooey wrote:
Seriously, you know what, the code below doesn't even work.
And if I get rid of the redirect and replace it with some nice pretty text:
print "Content-type: text/html\n\nTesting";
You know what? The warning for 1 (from the 'for' loop) is printed and then
it all stops, probably as the browser connection "closes".
Hiding it behind more forks made it work once but then never again.
So it is my belief that Apache2::SubProcess sucks dirty things, and that
belief won't change until I see proof to the contrary.
On a related note, anyone have tips for avoiding zombies in CGI forking?
Tosh
Tosh Cooey wrote:
Hi after much trial and all error I am seeing that the browser connection
closing is also stopping my subprocess.
The main ModPerl::Registry program looks like this:
### file.pl ###
use Apache2::SubProcess ();
use JSON();
&main(shift);
sub main {
my $r = shift;
...
$r->spawn_proc_prog ('/web/html/file_fork.pl', \...@argv);
return print $cgi->redirect('index.pl');
}
###
And
### file_fork.pl ###
use JSON();
use warnings;
use POSIX 'setsid';
chdir '/'or die "Can't chdir to /: $!";
open STDIN, '/dev/null' or die "Can't read /dev/null: $!";
open STDOUT, '+>>', '/tmp/debug.txt' or die "Can't write to
/tmp/debug.txt: $!";
open STDERR, '>&STDOUT' or die "Can't dup stdout: $!";
setsid or die "Can't start a new session: $!";
# run your code here or call exec to another program
foreach my $num (1..10) {
warn $num;
sleep(1);
}
###
When file.pl is executed it calls file_fork.pl and file_fork starts to
print 1, 2, 3 and then stops around 2 or 3 depending on how long my browser
is connected.
Are there any glaring errors that anyone sees? It should work shouldn't
it, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10 after 10 seconds...
Thank-you all!
Tosh
--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
--
McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Apache2::SubProcess sucks
On Feb 20, 2010, at 7:01 AM, Tosh Cooey wrote: Anyway, the solution, at least so far until I run into other problems, seems to be to just make a system() call and the called program uses Proc::Daemon and things *seem* to work fine in testing, we'll see when it hits production... Tosh Doesn't a process that calls 'system' wait for the child process to complete? If the parent process is an Apache child, is that in keeping with what you're trying to do? cmac
Re: How to get a file listing
Hi there
Here is my code. I get the IMG displayed but also the perl error: No such
file or directory.
"
#!E:\ea12\apps\tech_st\10.1.2\perl\5.6.1\bin\MSWin32-x86\perl.exe
print "Content-type: text/html \n\n";
print "Test:";
print " /Ski/temp.jpg ";
opendir(DIR, "/Ski") or die print "Error $!";
print "Dir list: ";
while( ($filename = readdir(DIR))){
print("$filename");
}
closedir DIR; "
It seems like the webserver (apache) knows what /Ski is but perl doesn't.
It's looking for /ski in cgi-bin. my /ski folder is out of the root. I have
an alias, directory and location defined in httpd.conf for it. How do I get
perl to see it?
--
View this message in context:
http://old.nabble.com/How-to-get-a-file-listing-tp27644565p27668252.html
Sent from the mod_perl - General mailing list archive at Nabble.com.
Re: How to get a file listing
Apache knows the context, PERL does not.
Fully qualify that directory name and it should work.
On 2/20/2010 1:01 PM, ceauke wrote:
Hi there
Here is my code. I get the IMG displayed but also the perl error: No such
file or directory.
"
#!E:\ea12\apps\tech_st\10.1.2\perl\5.6.1\bin\MSWin32-x86\perl.exe
print "Content-type: text/html \n\n";
print "Test:";
print " /Ski/temp.jpg ";
opendir(DIR, "/Ski") or die print "Error $!";
print "Dir list: ";
while( ($filename = readdir(DIR))){
print("$filename");
}
closedir DIR; "
It seems like the webserver (apache) knows what /Ski is but perl doesn't.
It's looking for /ski in cgi-bin. my /ski folder is out of the root. I have
an alias, directory and location defined in httpd.conf for it. How do I get
perl to see it?
Re: How to get a file listing
No, that doesn't work either. I used opendir (DIR, "G:\FTP\Ski") ... I've tried the slashes in all directions, and with and without preceding and slashes in the end. e.g. \Ski, \Ski\, /ski, /ski/ How do I get perl to understand the context? is there some function that points to the root? (even that will be pointless as I'm on WINXP where I have multiple drives. Apache knows the context, PERL does not. Fully qualify that directory name and it should work. -- View this message in context: http://old.nabble.com/How-to-get-a-file-listing-tp27644565p27668907.html Sent from the mod_perl - General mailing list archive at Nabble.com.
Re: How to get a file listing
Sorry, got it working with your advice: opendir(DIR, "G:/FTP/Ski") or die print "Error $!"; worked in the end... Damn these slashes :-D Thanks for the help -- View this message in context: http://old.nabble.com/How-to-get-a-file-listing-tp27644565p27668922.html Sent from the mod_perl - General mailing list archive at Nabble.com.
Re: Apache2::SubProcess sucks
It does, but Proc::Daemon closes a billion file handles and sets new process IDs and forks and forks and maybe forks a couple more times for good measure... Fingers crossed! I do enjoy the fact that nobody really seems to have a simple definitive vanilla fork/spawn process down pat, it seems everyone does what I do, trying this and that stumbling about until they come up with some monstrosity like Torsen has that works under the sheer weight of tricks. Tosh macke...@animalhead.com wrote: On Feb 20, 2010, at 7:01 AM, Tosh Cooey wrote: Anyway, the solution, at least so far until I run into other problems, seems to be to just make a system() call and the called program uses Proc::Daemon and things *seem* to work fine in testing, we'll see when it hits production... Tosh Doesn't a process that calls 'system' wait for the child process to complete? If the parent process is an Apache child, is that in keeping with what you're trying to do? cmac -- McIntosh Cooey - Twelve Hundred Group LLC - http://www.1200group.com/
Re: Apache2::SubProcess sucks
On Saturday 20 February 2010 19:25:39 Tosh Cooey wrote: > I do enjoy the fact that nobody really seems to have a simple definitive > vanilla fork/spawn process down pat, it seems everyone does what I do, > trying this and that stumbling about until they come up with some > monstrosity like Torsen has that works under the sheer weight of tricks. > Well, if I'd said: "Tosh take ModPerl::Something from CPAN and call ModPerl::Something::spawn" you'd liked it (no matter what it does internally), right? That sheer weight of tricks comes down to 3 points: - you have to close the client socket in the child process or the browser will wait for more data - if your process should survive an apache restart it must create its own process group, hence the setsid(). - accidental leaking file descriptors to child processes is generally considered a security hole This is what my function does, it closes all file descriptors except for STDERR and calls setsid, nothing special. Perhaps it would be good to reopen STDIN and STDOUT as /dev/null because many programs don't like closed STDIN/STDOUT but again, no tricks. A tricky part would be to really restore the normal Perl environment if you don't want to exec() another program but simply to call a Perl function. Reverting the effect of *CORE::GLOBAL::exit=\&ModPerl::Util::exit in an already compiled function, that is tricky. Or reestablish the connection between %ENV and C-level char *environ[]. Torsten
Re: Apache2::SubProcess sucks
In data 20 febbraio 2010 alle ore 21:16:22, Torsten Förtsch ha scritto: On Saturday 20 February 2010 19:25:39 Tosh Cooey wrote: I do enjoy the fact that nobody really seems to have a simple definitive vanilla fork/spawn process down pat, it seems everyone does what I do, trying this and that stumbling about until they come up with some monstrosity like Torsen has that works under the sheer weight of tricks. Well, if I'd said: "Tosh take ModPerl::Something from CPAN and call ModPerl::Something::spawn" you'd liked it (no matter what it does internally), right? That sheer weight of tricks comes down to 3 points: [...] That's really valuable, thanks for sharing. However, a question: is this something that should be included in Apache2::SubProcess to make it better? Or is this something that could be published another, separate, CPAN distribution? Or maybe it's already out there? -- Cosimo
Re: Apache2::SubProcess sucks
On Saturday 20 February 2010 21:24:31 Cosimo Streppone wrote:
> That's really valuable, thanks for sharing.
> However, a question:
>
> is this something that should be included in Apache2::SubProcess
> to make it better? Or is this something that could be published
> another, separate, CPAN distribution?
> Or maybe it's already out there?
>
Apache2::SubProcess' purpose is more for creating processes similar to CGI-
scripts. A CGI-script is killed by apache if it does not produce output for a
certain period of time. Also, the process gets killed I think at the end of
the request.
I think it could be an Apache2::SubProcess extension. That would mean adding
the stuff to xs/Apache2/SubProcess/SubProcess_pm in the source tree. But there
are a few things that are unsolved:
- is there a portable way to get all open file descriptors of the current
process? Under Linux one can readdir(/proc/self/fd). On Darwin I once simply
closed all fds from 0 to 1000. Some systems have getdtablesize(2), sometimes
it is getrlimit. Sometimes you can open /dev/something or /proc/self/something
and read the descriptor table at a certain offset. What the module should do
on Windows I simply don't know.
If you can solve this problem with apr-functions I think the it can really be
in Apache2::SubProcess.
- a way of really restoring the perl environment would be good, so that you
really can fork off subroutines.
But the real reason why it is not shaped out as a separate module yet is that
the code is really old and quite seldom needed. The main part was written
around year 2000 for mp1. At that time I didn't know about POSIX::close and
thus invented my own version like:
sub SYS_close {6;}# works on Linux and Darwin
sub sysclose {
syscall &SYS_close, shift()+0;
}
In short, the code was ugly.
And because my solution is Linux-only.
Torsten
Re: Apache2::SubProcess sucks
re: close all open file descriptors portably
well, if one is on a POSIX system, _SC_OPEN_MAX gives the max integer.
Then just close them all.
Here's my usual recipe for this:
# close all open file descriptors
my $max_fd = POSIX::sysconf(&POSIX::_SC_OPEN_MAX);
$max_fd = ((! defined $max_fd) || $max_fd < 0) ? 64 : $max_fd;
while (--$max_fd >= 0) {
next if ($max_fd == fileno($pid_fh));
POSIX::close($max_fd);
}
# now reopen std{in,out,err} on /dev/null
open(STDIN, "+>/dev/null");
open(STDOUT, "+>&STDIN");
open(STDERR, "+>&STDIN");
Proc::Daemon does the same thing...
HTH,
-- jeff
On 2/20/10 1:10 PM, "Torsten Förtsch" wrote:
> - is there a portable way to get all open file descriptors of the current
> process? Under Linux one can readdir(/proc/self/fd). On Darwin I once simply
Re: Apache2::SubProcess sucks
2010/2/20 Torsten Förtsch [ ... ] > - is there a portable way to get all open file descriptors of the current > process? Under Linux one can readdir(/proc/self/fd). On Darwin I once > simply > closed all fds from 0 to 1000. Some systems have getdtablesize(2), > sometimes > it is getrlimit. Sometimes you can open /dev/something or > /proc/self/something > and read the descriptor table at a certain offset. What the module should > do > on Windows I simply don't know. > I think just closing stdin and stdout is likely to be enough, or maybe all file descriptors from 0 to $^F . The OS will close any others that have the close-on-exec bit set, and the close-on-exec bit is set by default by Perl's open function. Anybody setting this to something different probably understands the consequences. Scott.
