Re: Weird IPC/Apache issue
Mark Fowler wrote: > On Tue, Jan 19, 2010 at 10:21 PM, Matt Lawrence > wrote: > >> In my limited experience, running any subprocess in mod_perl is fraught with >> difficulty > > Another option is not to actually run the process from Apache2 at all, > but rather have a separate long running daemon that handles the > execution for you which you hand off work to from Apache2, thus > avoiding the need to fork at all. This is most useful as forking new > processes is quite expensive and a bunch of unplanned forks without > rate limiting could be used to DoS your site. Initially, I drew up a plan for a daemon-ish Perl module that would handle requests and filter the text through bogofilter. But the way bogofilter wants to run off of the command-line doesn't help the matter. It's not impossible, but it isn't easy (or clear, or even just mildly-challenging) either. I solved the problem by using IPC::Run, which works under Apache2/mod_perl2 as well as non-Apache environments. It's a neat little module (that I blogged about a while back), that does things in a nicely Perlish way. > Of course, you've just asked "how do I do this simple thing X" and > I've said "Do this more complex thing to your architecture Y", but > that's the nature of advice on the internet ;-) True, but it closely parallels the design I've already been toying with, so there is some reassurance there. At least I was thinking along similar lines (though I hadn't considered Gearman or other tools). Randy -- """ Randy J. Ray Sunnyvale, CA http://www.rjray.org rj...@blackperl.com Silicon Valley Scale Modelers: http://www.svsm.org
Re: Weird IPC/Apache issue
On Tue, Jan 19, 2010 at 10:21 PM, Matt Lawrence wrote: > Randy J. Ray wrote: >> >> I have a module that uses IPC::Open3 (or IPC::Open2, both exhibit this >> problem) to call an external binary (bogofilter in this case) > In my limited experience, running any subprocess in mod_perl is fraught with > difficulty Another option is not to actually run the process from Apache2 at all, but rather have a separate long running daemon that handles the execution for you which you hand off work to from Apache2, thus avoiding the need to fork at all. This is most useful as forking new processes is quite expensive and a bunch of unplanned forks without rate limiting could be used to DoS your site. Depending on the reliability and speed of what you want to do, I'd suggest looking at Gearman (if someone is interactively waiting for the result) or TheSchwartz (if they're not and you can "defer" the work to be carried out a few seconds later in a reliable way.) (YMMV, other queue systems are available) Of course, you've just asked "how do I do this simple thing X" and I've said "Do this more complex thing to your architecture Y", but that's the nature of advice on the internet ;-) Mark. http://blog.twoshortplanks.com/ - my Perl Blog.
Re: Weird IPC/Apache issue
Randy J. Ray wrote: I have a module that uses IPC::Open3 (or IPC::Open2, both exhibit this problem) to call an external binary (bogofilter in this case) and feed it some input via the child-input filehandle, then reads the result from the child-output handle. The code works fine when run in most environments. However, the main use of this module is in a web service that runs under Apache 2.2.6. And under that environment, I get the error: Cannot fdopen STDOUT: Invalid argument This only happens when the code runs under Apache. Previously, the code constructed a horribly complex command, which included a here-document for the input, and ran it with back-ticks. THAT worked, but was very slow and prone to breaking in unique and perplexing ways. I would hate to have to revert to the old version, but I cannot crack this. (If it helps, I'm using the list-form in open3 to specify the command, not a single string since that would invoke an extra bash process.) In my limited experience, running any subprocess in mod_perl is fraught with difficulty, and it rarely seems to do what you would hope would work. I've never seen this particular error before, but I'd still recommend switching to Apache2::SubProcess to do this. It's interface is broadly similar to IPC::Open3 (at least in one form). http://perl.apache.org/docs/2.0/api/Apache2/SubProcess.html Matt
Weird IPC/Apache issue
I have a module that uses IPC::Open3 (or IPC::Open2, both exhibit this problem) to call an external binary (bogofilter in this case) and feed it some input via the child-input filehandle, then reads the result from the child-output handle. The code works fine when run in most environments. However, the main use of this module is in a web service that runs under Apache 2.2.6. And under that environment, I get the error: Cannot fdopen STDOUT: Invalid argument This only happens when the code runs under Apache. Previously, the code constructed a horribly complex command, which included a here-document for the input, and ran it with back-ticks. THAT worked, but was very slow and prone to breaking in unique and perplexing ways. I would hate to have to revert to the old version, but I cannot crack this. (If it helps, I'm using the list-form in open3 to specify the command, not a single string since that would invoke an extra bash process.) Randy -- """ Randy J. Ray Sunnyvale, CA http://www.rjray.org rj...@blackperl.com Silicon Valley Scale Modelers: http://www.svsm.org
