tcpserver + perl (Re: PPerl usage)
On Wed, 10 Mar 2004, Charlie Brady wrote: http://cr.yp.to/ucspi-tcp/tcpserver.html The traditional qpsmtpd run script teams tcpserver and perl OK. Here's another possible solution: http://www.superscript.com/ucspi-tcphandle/tcpperl.html It seems to be more or less tcpserver with an embedded perl interpretor. tcpperl forks the requested number of children, each listening for TCP client connections. Before handling any requests, each child parses and runs the perl code in file. The file should end with a true expression, like a module. To handle a request, a child process executes prog as a perl subroutine, with file descriptor 0 reading from the network and file descriptor 1 writing to the network. Before handling the request, the child sets certain environment variables, a la tcpserver. --- Charlie
Re: tcpserver + perl (Re: PPerl usage)
Anyone out there willing to try some rough benchmarks on tcpperl vs. tcpserver+pperl vs. tcpserver+speedycgi vs. tcpserver+straight_perl? -jc At 12:43 PM 3/12/2004, Charlie Brady wrote: On Wed, 10 Mar 2004, Charlie Brady wrote: http://cr.yp.to/ucspi-tcp/tcpserver.html The traditional qpsmtpd run script teams tcpserver and perl OK. Here's another possible solution: http://www.superscript.com/ucspi-tcphandle/tcpperl.html It seems to be more or less tcpserver with an embedded perl interpretor. tcpperl forks the requested number of children, each listening for TCP client connections. Before handling any requests, each child parses and runs the perl code in file. The file should end with a true expression, like a module. To handle a request, a child process executes prog as a perl subroutine, with file descriptor 0 reading from the network and file descriptor 1 writing to the network. Before handling the request, the child sets certain environment variables, a la tcpserver. --- Charlie
Re: PPerl usage
On 9 Mar 2004, at 22:32, ross mueller wrote: Just FYI, if you try to load up qpsmtpd thats mounted on a NFS slice with pperl, all sorts of errors go out ;) (im sure ill get a response saying dont run it off an nfs slice) anything from: pperl: failed to read 3 bytes for an OK message: Connection reset by peer to: pperl: failed to read 3 bytes for an OK message: Bad file descriptor to: pperl: resolving full pathname to script failed: No such file or directory If you move qpsmtpd off the nfs slice, it loads fine, but i still have a bunch of issues with it. Interesting. I've recently discovered one weird thing in pperl where pperl tries too hard to become like the calling process, to the point of trying to set $$ to the caller's PID. If you comment out the code where it sets $$ everything starts working again. I have no idea if this will fix what you're doing. But it does make me think there might be a call for a pperl that doesn't try quite so hard. Or a flag that says I don't care about emulation of the caller - just give me the speedup. Unfortunately debugging pperl is one of the hardest things I've ever had to do in my career. Matt.
Re: PPerl usage
On Wed, 10 Mar 2004, Matt Sergeant wrote: Interesting. I've recently discovered one weird thing in pperl where pperl tries too hard to become like the calling process, to the point of trying to set $$ to the caller's PID. If you comment out the code where it sets $$ everything starts working again. I have no idea if this will fix what you're doing. But it does make me think there might be a call for a pperl that doesn't try quite so hard. Or a flag that says I don't care about emulation of the caller - just give me the speedup. But we do want some emulation of the caller don't we? I'm thinking of environment in particular. Unfortunately debugging pperl is one of the hardest things I've ever had to do in my career. Is it worth it? I'm curious as to whether anyone has built a forking server around qpsmtpd (essentially doing what tcpserver does, but in perl)? I'm aware of the select server version(*), but would prefer a forking server (say, using Net::Daemon). That would seem to be able to provide the same speedup as pperl, but without the headache. What am I missing? (*) Here's a patch to remove some incorrect doco: --- qpsmtpd-server 2 Nov 2003 11:36:01 - 1.2 +++ qpsmtpd-server 10 Mar 2004 17:34:45 - @@ -2,9 +2,6 @@ # Copyright (c) 2001 Ask Bjoern Hansen. See the LICENSE file for details. # The command dispatch system is taken from colobus - http://trainedmonkey.com/colobus/ # -# this is designed to be run under tcpserver (http://cr.yp.to/ucspi-tcp.html) -# or inetd if you're into that sort of thing -# # # For more information see http://develooper.com/code/qpsmtpd/ # --- Charlie
Re: PPerl usage
I'm curious as to whether anyone has built a forking server around qpsmtpd (essentially doing what tcpserver does, but in perl)? I'm aware of the select server version(*), but would prefer a forking server (say, using Net::Daemon). That would seem to be able to provide the same speedup as pperl, but without the headache. What am I missing? pperl is pre-forking. It doesn't fork on every connection. The forking on every connection is part of what causes some of the overhead. But in general, I think this is a good idea, at least to see how it works :) Net::Daemon can do preforking too. -R
Re: PPerl usage
On Wed, 10 Mar 2004, Robert Spier wrote: I'm curious as to whether anyone has built a forking server around qpsmtpd (essentially doing what tcpserver does, but in perl)? I'm aware of the select server version(*), but would prefer a forking server (say, using Net::Daemon). That would seem to be able to provide the same speedup as pperl, but without the headache. What am I missing? pperl is pre-forking. Yes, but SelectServer isn't. It doesn't fork on every connection. The forking on every connection is part of what causes some of the overhead. Do you have evidence that that is the case? Fork should be inexpensive, and it's certainly not a problem with tcpserver. But in general, I think this is a good idea, at least to see how it works :) I'll let you know if I get something working. Please be sure to let me know if you beat me to it :-) Net::Daemon can do preforking too. It can, but is latency critical enough for it to be an issue? --- Charlie
Re: PPerl usage
pperl is pre-forking. Yes, but SelectServer isn't. SelectServer doesn't fork at all. It doesn't fork on every connection. The forking on every connection is part of what causes some of the overhead. Do you have evidence that that is the case? Fork should be inexpensive, and it's certainly not a problem with tcpserver. tcpserver is designed to fork and exec lightweight C programs. forking larger programs (i.e. perl) can be an issue. And there is definitely overhead to forking. But in general, I think this is a good idea, at least to see how it works :) I'll let you know if I get something working. Please be sure to let me know if you beat me to it :-) It won't happen this month. You don't have to worry. Net::Daemon can do preforking too. It can, but is latency critical enough for it to be an issue? Yes. -R
Re: PPerl usage
On Wed, 10 Mar 2004, Robert Spier wrote: It doesn't fork on every connection. The forking on every connection is part of what causes some of the overhead. Do you have evidence that that is the case? Fork should be inexpensive, and it's certainly not a problem with tcpserver. tcpserver is designed to fork and exec lightweight C programs. I don't see lightweight C here: http://cr.yp.to/ucspi-tcp/tcpserver.html The traditional qpsmtpd run script teams tcpserver and perl OK. forking larger programs (i.e. perl) can be an issue. Much less of an issue with modern CoW fork. And there is definitely overhead to forking. Sure, but it's not necessarily a dominating issue in the design decision. The fork per se is not the problem, it's the subsequent exec and perl startup which is the real expense. Net::Daemon can do preforking too. It can, but is latency critical enough for it to be an issue? Yes. Care to explain? Do you have any figures to back up the assertion? I'm not trying to disagree with you, I just want to be sure that it's worth any effort to optimise. (Although, with Net::Daemon, it's probably a tiny change of code to have one or t'other). --- Charlie
PPerl usage
Show of hands: how many people are using PPerl with qpsmtpd? I recently has some arguments with my server when I updated one of the core lib/ files and couldn't get the stupid thing to use the new file. It turns out I forgot to restart the PPerl session (`svc -t .' wasn't sufficient). It was doubly confusing since the PPerl-created socket was in /tmp, so the .pid had long since been deleted by the code that cleans /tmp. Apart from demonstrating my ability to forget vital features of running my system, does anyone have an opinion about whether it is worthwhile running qpsmtpd under PPerl??? John
Re: PPerl usage
Apart from demonstrating my ability to forget vital features of running my system, does anyone have an opinion about whether it is worthwhile running qpsmtpd under PPerl??? We're running it (mostly) at perl.org, because it really hides the startup overhead, which can have a huge effect on the other things on the machine when a lot of email is moving. So my opinion: yes, its worth it -R
Re: PPerl usage
On 9 Mar 2004, at 21:49, [EMAIL PROTECTED] wrote: On Tue, 9 Mar 2004, Robert Spier wrote: Apart from demonstrating my ability to forget vital features of running my system, does anyone have an opinion about whether it is worthwhile running qpsmtpd under PPerl??? We're running it (mostly) at perl.org, because it really hides the startup overhead, which can have a huge effect on the other things on the machine when a lot of email is moving. I haven't been able to get PPerl to complete its tests properly - probably minor things, but I'm not going to implement it until such time as all tests work. I haven't seen reports from you :-) I was wondering if it might be possible to compile qpsmtpd (along with the plugins) with perlcc. Is there anything preventing it, or might it require a lot of changes? perlcc isn't what you're looking for :-) It doesn't do what most people expect (or work nearly as well as most people hope). Matt.
Re: PPerl usage
On Tue, 9 Mar 2004, Matt Sergeant wrote: On 9 Mar 2004, at 21:49, [EMAIL PROTECTED] wrote: I haven't been able to get PPerl to complete its tests properly - probably minor things, but I'm not going to implement it until such time as all tests work. I haven't seen reports from you :-) Actually, I still have copies of the emails we exchanged on the subject :-) I sent the details of the tests that failed, but you were unable to determine a cause. Frankly, with the Linux distribution I'm currently using, I think some of the packages failed to load properly, so something that's needed is simply missing. I'll try again when I install another distribution. I was wondering if it might be possible to compile qpsmtpd (along with the plugins) with perlcc. Is there anything preventing it, or might it require a lot of changes? perlcc isn't what you're looking for :-) It doesn't do what most people expect (or work nearly as well as most people hope). For many things that I've done, perlcc seems to work fine. However, I suspect with the way qpsmtpd is designed, it would be a fair amount of pain to compile it. -- Roger Walker HIS Pain - OUR Gain
