Re: AnyEvent mode?
On Nov 23, 2009, at 11:41, Matt Sergeant wrote: > Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written > (mostly hacked right now, but should work). > > In theory it might be faster than the Danga::Socket based one, and AnyEvent > seems to receive regular updates more than Danga::Socket these days. There are also more "other stuff" available with AnyEvent -- I think it'd be cool! - ask
Re: AnyEvent mode?
google: perl anyevent yields: http://search.cpan.org/dist/AnyEvent/ is there a better starting point ? On Mon, 2009-23-11 at 11:42 -0800, Ask Bjørn Hansen wrote: > On Nov 23, 2009, at 11:41, Matt Sergeant wrote: > > > Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written > > (mostly hacked right now, but should work). > > > > In theory it might be faster than the Danga::Socket based one, and AnyEvent > > seems to receive regular updates more than Danga::Socket these days. > > There are also more "other stuff" available with AnyEvent -- I think it'd be > cool! > > > - ask -- --gh
Re: AnyEvent mode?
On Mon, 2009-23-11 at 11:42 -0800, Ask Bjørn Hansen wrote: > There are also more "other stuff" available with AnyEvent -- I think > it'd be cool! I found this fairly easily. http://software.schmorp.de/pkg/AnyEvent.html Hmm... at least he has a good reason for upgrading to 5.10. http://search.cpan.org/dist/AnyEvent/lib/AnyEvent.pm#SEE_ALSO Perl 5.8 has numerous memleaks that sometimes hit this module and are hard to work around. -- --gh
Re: AnyEvent mode?
On Nov 23, 2009, at 12:08, Guy Hulbert wrote: > google: perl anyevent > > yields: http://search.cpan.org/dist/AnyEvent/ > > is there a better starting point ? I'm not exactly sure what your question is, but http://search.cpan.org/search?query=anyevent shows a sample of available event implementations and other modules. - ask
Re: AnyEvent mode?
On Mon, 2009-23-11 at 12:17 -0800, Ask Bjørn Hansen wrote: > On Nov 23, 2009, at 12:08, Guy Hulbert wrote: > > > google: perl anyevent > > > > yields: http://search.cpan.org/dist/AnyEvent/ > > > > is there a better starting point ? > > I'm not exactly sure what your question is, but I'm not sure either but this: http://search.cpan.org/dist/AnyEvent/lib/AnyEvent/Intro.pod is a better answer ;-). > > http://search.cpan.org/search?query=anyevent > > shows a sample of available event implementations and other modules. > > > - ask > -- --gh
Re: AnyEvent mode?
On Mon, 23 Nov 2009, Ask Bjørn Hansen wrote: On Nov 23, 2009, at 11:41, Matt Sergeant wrote: Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written (mostly hacked right now, but should work). In theory it might be faster than the Danga::Socket based one, and AnyEvent seems to receive regular updates more than Danga::Socket these days. There are also more "other stuff" available with AnyEvent -- I think it'd be cool! OK. How do I get this to you? It's basically just two new files, no patches to anything. Matt.
Re: AnyEvent mode?
On Nov 23, 2009, at 14:52, Matt Sergeant wrote: >> There are also more "other stuff" available with AnyEvent -- I think it'd be >> cool! > > OK. How do I get this to you? It's basically just two new files, no patches > to anything. Either make a github account, clone my repository and then: git clone g...@github.com:msergeant/qpsmtpd.git git add new.file another-file.pl git commit git push origin ... and then anyone can work with it from your repository. This is how most patches are processed now. or: git clone git://github.com/abh/qpsmtpd.git git add new.file another-file.pl git commit git format-patch HEAD^.. git send-email --to=qpsm...@perl.org 0001* to manage the changes in git and use git to send them to the list. ... or of course you could just send them to the list or to me or put them on http somewhere and send a link -- but since you asked. :-) - ask -- http://develooper.com/ - http://askask.com/
Re: AnyEvent mode?
OK, here's the files as they currently stand. The big note on this is that
I have done very little testing, and most importantly I have NOT updated
any of the plugins (see the async/* dir for those that need re-written to
use AnyEvent, mostly to use AnyEvent::DNS. Also the tls plugin would
need updated/rewritten, though it's probably a lot simpler with AnyEvent).
I'll be trying to do performance testing tomorrow (comparing with
qpsmtpd-async, NOT any of the other models).
Matt.#!/usr/bin/perl
use lib "./lib";
BEGIN {
delete $ENV{ENV};
delete $ENV{BASH_ENV};
$ENV{PATH} = '/bin:/usr/bin:/var/qmail/bin:/usr/local/bin';
}
# Profiling - requires Devel::Profiler 0.05
#BEGIN { $Devel::Profiler::NO_INIT = 1; }
#use Devel::Profiler;
use strict;
use vars qw($DEBUG);
use FindBin qw();
# TODO: need to make this taint friendly
use lib "$FindBin::Bin/lib";
use Qpsmtpd::AnyEvent;
use Qpsmtpd::ConfigServer;
use Qpsmtpd::Constants;
use Carp;
use POSIX qw(WNOHANG);
use Getopt::Long;
use List::Util qw(shuffle);
use Socket;
use AnyEvent::Socket;
use AnyEvent::Util;
$|++;
$SIG{'PIPE'} = "IGNORE"; # handled manually
$DEBUG = 0;
my $PORT= 2525;
my $LOCALADDR = '0.0.0.0';
my $PROCS = 1;
my $USER= (getpwuid $>)[0]; # user to suid to
$USER= "smtpd" if $USER eq "root";
my $PAUSED = 0;
my $NUMACCEPT = 20;
my $PID_FILE= '';
my $ACCEPT_RSET;
my $DETACH; # daemonize on startup
# make sure we don't spend forever doing accept()
use constant ACCEPT_MAX => 1000;
sub reset_num_accept {
$NUMACCEPT = 20;
}
sub help {
print < \$PORT,
'l|listen-address=s'=> \$LOCALADDR,
'j|procs=i' => \$PROCS,
'v|verbose+' => \$DEBUG,
'u|user=s' => \$USER,
'pid-file=s'=> \$PID_FILE,
'd|detach' => \$DETACH,
'h|help'=> \&help,
) || help();
# detaint the commandline
if ($PORT =~ /^(\d+)$/) { $PORT = $1 } else { &help }
if ($LOCALADDR =~ /^([\d\w\-.]+)$/) { $LOCALADDR = $1 } else { &help }
if ($USER =~ /^([\w\-]+)$/) { $USER = $1 } else { &help }
if ($PROCS =~ /^(\d+)$/) { $PROCS = $1 } else { &help }
use constant READY => 1;
use constant ACCEPTING => 2;
use constant RESTARTING => 999;
my $CURRENT_PROCS = 0;
my %childstatus = ();
my $SERVER;
if ($PID_FILE && -r $PID_FILE) {
open PID, "<$PID_FILE"
or die "open_pidfile $PID_FILE: $!\n";
my $running_pid = || ''; chomp $running_pid;
if ($running_pid =~ /^(\d+)/) {
if (kill 0, $running_pid) {
die "Found an already running qpsmtpd with pid $running_pid.\n";
}
}
close(PID);
}
run_as_server();
exit(0);
sub spawn_child {
my ($plugin_loader) = @_;
my $pid = fork;
if ($pid) {
$childstatus{$pid}++;
$CURRENT_PROCS++;
return $pid;
}
elsif (!defined($pid)) {
die "Unable to fork";
}
$plugin_loader->run_hooks('post-fork');
accept_loop();
}
sub accept_loop {
my $w = AE::io $SERVER, 0, sub {
print "Connect?\n";
while ($SERVER && (my $peer = accept my $fh, $SERVER)) {
fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside
world does not
my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
my $qp = Qpsmtpd::AnyEvent->new($fh, format_address($host),
$service);
$qp->process_line("Connect");
}
};
AnyEvent->condvar->wait;
exit;
}
sub sig_hup {
kill 1, keys %childstatus;
}
sub sig_chld {
my $spawn_count = 0;
while ( (my $child = waitpid(-1,WNOHANG)) > 0) {
if (!defined $childstatus{$child}) {
next;
}
last unless $child > 0;
print "SIGCHLD: child $child died\n";
delete $childstatus{$child};
$CURRENT_PROCS--;
}
$SIG{CHLD} = \&sig_chld;
}
sub HUNTSMAN {
$SIG{CHLD} = 'DEFAULT';
kill 'INT' => keys %childstatus;
if ($PID_FILE && -e $PID_FILE) {
unlink $PID_FILE or ::log(LOGERROR, "unlink: $PID_FILE: $!");
}
exit(0);
}
sub _connect_sock {
my $ipn = parse_address($LOCALADDR) or die "cannot parse '$LOCALADDR' as
host address";
my $af = address_family $ipn;
socket(my $sock, $af, SOCK_STREAM, 0) or die "socket: $!";
setsockopt($sock, SOL_SOCKET, SO_REUSEADDR, 1)
or die "tcp_server/so_reuseaddr: $!";
bind $sock, AnyEvent::Socket::pack_sockaddr($PORT, $ipn)
or die "bind: $!";
fh_nonblocking($sock, 1);
listen($sock, 128);
return $sock;
}
sub run_as_server {
AnyEvent::detect;
$SERVER = _connect_sock();
# Drop priviledges
my (undef, undef, $quid, $qgid) = getpwnam $USER or
die "unable to determine uid/gid for $USER\n";
my $groups = "$qgid $qgid";
while (my (undef, undef, $gid, $members) = getgrent) {
my @m = split(/ /, $members);
Re: AnyEvent mode?
Dear Matt, On Die 24.11.2009 22:13, Matt Sergeant wrote: OK, here's the files as they currently stand. The big note on this is that I have done very little testing, and most importantly I have NOT updated any of the plugins (see the async/* dir for those that need re-written to use AnyEvent, mostly to use AnyEvent::DNS. Also the tls plugin would need updated/rewritten, though it's probably a lot simpler with AnyEvent). I'll be trying to do performance testing tomorrow (comparing with qpsmtpd-async, NOT any of the other models). Matt. do you have done anything more in this direction? Cheers Aleks
Re: AnyEvent mode?
Hi, On Wed, Dec 1, 2010 at 9:15 PM, Aleksandar Lazic wrote: > On Die 24.11.2009 22:13, Matt Sergeant wrote: >> I'll be trying to do performance testing tomorrow (comparing with >> qpsmtpd-async, NOT any of the other models). > > do you have done anything more in this direction? I believe I saw a report later on from Matt stating that AnyEvent-based qpsmtpd was slower than Danga::Socket, but I don't remember by how much, nor can I find that email message right now. I do have a recolection that I though the performance cost of moving to AnyEvent was worthwhile if you need to create your own async plugins, like calling HTTP webservices, given that AnyEvent has a much larget module ecosystem than Danga::Socket. Bye, -- Pedro Melo http://www.simplicidade.org/ xmpp:m...@simplicidade.org mailto:m...@simplicidade.org
Re: AnyEvent mode?
On Don 02.12.2010 10:34, Pedro Melo wrote: Hi, On Wed, Dec 1, 2010 at 9:15 PM, Aleksandar Lazic wrote: On Die 24.11.2009 22:13, Matt Sergeant wrote: I'll be trying to do performance testing tomorrow (comparing with qpsmtpd-async, NOT any of the other models). do you have done anything more in this direction? I believe I saw a report later on from Matt stating that AnyEvent-based qpsmtpd was slower than Danga::Socket, but I don't remember by how much, nor can I find that email message right now. I do have a recolection that I though the performance cost of moving to AnyEvent was worthwhile if you need to create your own async plugins, like calling HTTP webservices, given that AnyEvent has a much larget module ecosystem than Danga::Socket. Hm, well I give it a try again ;-)
Re: AnyEvent mode?
On Dec 2, 2010, at 8:42, Aleksandar Lazic wrote: > Hm, well I give it a try again ;-) For what it's worth, if I was starting a qpsmtpd-like thing from scratch today I'd almost certainly make it AnyEvent based. :-) - ask
Re: AnyEvent mode?
On Don 02.12.2010 09:49, Ask Bjørn Hansen wrote: On Dec 2, 2010, at 8:42, Aleksandar Lazic wrote: Hm, well I give it a try again ;-) For what it's worth, if I was starting a qpsmtpd-like thing from scratch today I'd almost certainly make it AnyEvent based. :-) I like to develop something with AnyEvent and the qpsmtpd looks very nice and worth to get another alternative daemon ;-) Maybe we can make another benchmark AnyEvent vs Danga::Socket due to the fact that AnyEvent with EV as underlaying event lib looks very fast from the internet source ;-)
Re: AnyEvent mode?
On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote: > Maybe we can make another benchmark AnyEvent vs Danga::Socket due to > the fact that AnyEvent with EV as underlaying event lib looks very fast > from the internet source ;-) Matt was (I'm guessing) testing a load that's artificial to anyone who's not just archiving spam from a spam trap. For the rest of us by far most of the resources go to the various spam filtering stuff, so the performance will be fine. More important is the ease of use of the APIs, the general eco system etc. On those AnyEvent wins (in my opinion). - ask -- http://develooper.com/ - http://askask.com/
Re: AnyEvent mode?
On Don 02.12.2010 10:48, Ask Bjørn Hansen wrote: On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote: Maybe we can make another benchmark AnyEvent vs Danga::Socket due to the fact that AnyEvent with EV as underlaying event lib looks very fast from the internet source ;-) Matt was (I'm guessing) testing a load that's artificial to anyone who's not just archiving spam from a spam trap. For the rest of us by far most of the resources go to the various spam filtering stuff, so the performance will be fine. More important is the ease of use of the APIs, the general eco system etc. On those AnyEvent wins (in my opinion). I plan to use AnyEvent::Worker::Pool and make something like qpsmtpd-prefork with AnyEvent and multi port listening. I'am still in learn mode of qpsmtpd and AnyEvent so it will take some times before I get a usable version. Aleks
Re: AnyEvent mode?
On Thu, 2 Dec 2010, Ask Bjørn Hansen wrote: On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote: Maybe we can make another benchmark AnyEvent vs Danga::Socket due to the fact that AnyEvent with EV as underlaying event lib looks very fast from the internet source ;-) Matt was (I'm guessing) testing a load that's artificial to anyone who's not just archiving spam from a spam trap. For the rest of us by far most of the resources go to the various spam filtering stuff, so the performance will be fine. More important is the ease of use of the APIs, the general eco system etc. On those AnyEvent wins (in my opinion). Sorry yeah I think I have it working, but our work data centre is down right now so I can't get at the files :-( Will email the latest ones here when I get the chance. And yes it was slower than Danga::Socket (despite EV being "fast", Danga::Socket hits directly to epoll, with very little API in the way, so ends up being faster) but Ask's point stands - there's more dev done around AnyEvent on CPAN. Matt.
Re: AnyEvent mode?
On Don 02.12.2010 19:04, Matt Sergeant wrote: On Thu, 2 Dec 2010, Ask Bjørn Hansen wrote: On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote: Maybe we can make another benchmark AnyEvent vs Danga::Socket due to the fact that AnyEvent with EV as underlaying event lib looks very fast from the internet source ;-) Matt was (I'm guessing) testing a load that's artificial to anyone who's not just archiving spam from a spam trap. For the rest of us by far most of the resources go to the various spam filtering stuff, so the performance will be fine. More important is the ease of use of the APIs, the general eco system etc. On those AnyEvent wins (in my opinion). Sorry yeah I think I have it working, but our work data centre is down right now so I can't get at the files :-( Will email the latest ones here when I get the chance. Thanks, I'am quite interested. And yes it was slower than Danga::Socket (despite EV being "fast", Danga::Socket hits directly to epoll, with very little API in the way, so ends up being faster) but Ask's point stands - there's more dev done around AnyEvent on CPAN. Ok. I will try to get the runbench2, in eg directory of AnyEvent, working with Danga::Socket and try to compare.
Re: AnyEvent mode?
On Thu, 2 Dec 2010, Aleksandar Lazic wrote:
On Don 02.12.2010 19:04, Matt Sergeant wrote:
On Thu, 2 Dec 2010, Ask Bjørn Hansen wrote:
On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote:
Maybe we can make another benchmark AnyEvent vs Danga::Socket due to
the fact that AnyEvent with EV as underlaying event lib looks very fast
from the internet source ;-)
Matt was (I'm guessing) testing a load that's artificial to anyone who's
not just archiving spam from a spam trap. For the rest of us by far most
of the resources go to the various spam filtering stuff, so the
performance will be fine.
More important is the ease of use of the APIs, the general eco system etc.
On those AnyEvent wins (in my opinion).
Sorry yeah I think I have it working, but our work data centre is down
right now so I can't get at the files :-(
Will email the latest ones here when I get the chance.
Thanks, I'am quite interested.
OK, the relevant files are attached. Nothing else needed changing I don't
think.
AnyEvent.pm has to go in the lib/Qpsmtpd/ dir.#!/usr/bin/perl
use lib "./lib";
BEGIN {
delete $ENV{ENV};
delete $ENV{BASH_ENV};
$ENV{PATH} = '/bin:/usr/bin:/var/qmail/bin:/usr/local/bin';
}
# Profiling - requires Devel::Profiler 0.05
#BEGIN { $Devel::Profiler::NO_INIT = 1; }
#use Devel::Profiler;
use strict;
use vars qw($DEBUG);
use FindBin qw();
# TODO: need to make this taint friendly
use lib "$FindBin::Bin/lib";
use Qpsmtpd::AnyEvent;
use Qpsmtpd::ConfigServer;
use Qpsmtpd::Constants;
use Carp;
use POSIX qw(WNOHANG);
use Getopt::Long;
use List::Util qw(shuffle);
use Socket;
use AnyEvent::Socket;
use AnyEvent::Util;
$|++;
$SIG{'PIPE'} = "IGNORE"; # handled manually
$DEBUG = 0;
my $PORT= 2525;
my $LOCALADDR = '0.0.0.0';
my $PROCS = 1;
my $USER= (getpwuid $>)[0]; # user to suid to
$USER= "smtpd" if $USER eq "root";
my $PAUSED = 0;
my $NUMACCEPT = 20;
my $PID_FILE= '';
my $ACCEPT_RSET;
my $DETACH; # daemonize on startup
# make sure we don't spend forever doing accept()
use constant ACCEPT_MAX => 1000;
sub reset_num_accept {
$NUMACCEPT = 20;
}
sub help {
print < \$PORT,
'l|listen-address=s'=> \$LOCALADDR,
'j|procs=i' => \$PROCS,
'v|verbose+' => \$DEBUG,
'u|user=s' => \$USER,
'pid-file=s'=> \$PID_FILE,
'd|detach' => \$DETACH,
'h|help'=> \&help,
) || help();
# detaint the commandline
if ($PORT =~ /^(\d+)$/) { $PORT = $1 } else { &help }
if ($LOCALADDR =~ /^([\d\w\-.]+)$/) { $LOCALADDR = $1 } else { &help }
if ($USER =~ /^([\w\-]+)$/) { $USER = $1 } else { &help }
if ($PROCS =~ /^(\d+)$/) { $PROCS = $1 } else { &help }
use constant READY => 1;
use constant ACCEPTING => 2;
use constant RESTARTING => 999;
my $CURRENT_PROCS = 0;
my %childstatus = ();
my $SERVER;
if ($PID_FILE && -r $PID_FILE) {
open PID, "<$PID_FILE"
or die "open_pidfile $PID_FILE: $!\n";
my $running_pid = || ''; chomp $running_pid;
if ($running_pid =~ /^(\d+)/) {
if (kill 0, $running_pid) {
die "Found an already running qpsmtpd with pid $running_pid.\n";
}
}
close(PID);
}
run_as_server();
exit(0);
sub spawn_child {
my ($plugin_loader) = @_;
my $pid = fork;
if ($pid) {
$childstatus{$pid}++;
$CURRENT_PROCS++;
return $pid;
}
elsif (!defined($pid)) {
die "Unable to fork";
}
$plugin_loader->run_hooks('post-fork');
accept_loop();
}
sub accept_loop {
my $w = AE::io $SERVER, 0, sub {
print "Connect?\n";
while ($SERVER && (my $peer = accept my $fh, $SERVER)) {
fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside
world does not
my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
my $qp = Qpsmtpd::AnyEvent->new($fh, format_address($host),
$service);
$qp->process_line("Connect");
}
};
AnyEvent->condvar->wait;
exit;
}
sub sig_hup {
kill 1, keys %childstatus;
}
sub sig_chld {
my $spawn_count = 0;
while ( (my $child = waitpid(-1,WNOHANG)) > 0) {
if (!defined $childstatus{$child}) {
next;
}
last unless $child > 0;
print "SIGCHLD: child $child died\n";
delete $childstatus{$child};
$CURRENT_PROCS--;
}
$SIG{CHLD} = \&sig_chld;
}
sub HUNTSMAN {
$SIG{CHLD} = 'DEFAULT';
kill 'INT' => keys %childstatus;
if ($PID_FILE && -e $PID_FILE) {
unlink $PID_FILE or ::log(LOGERROR, "unlink: $PID_FILE: $!");
}
exit(0);
}
sub _connect_sock {
my $ipn = parse_address($LOCALADDR) or die "cannot parse '$LOCALADDR' as
host address";
my $af = address_family $ipn;
socket(my $sock, $af, SOCK_STREAM, 0) or die "socket: $!";
setsockopt($sock, SOL_SOCKET, SO_REUSEADDR, 1)
Re: AnyEvent mode?
Dear Matt, On Fre 03.12.2010 14:33, Matt Sergeant wrote: On Thu, 2 Dec 2010, Aleksandar Lazic wrote: Thanks, I'am quite interested. OK, the relevant files are attached. Nothing else needed changing I don't think. Thanks for the files. Do you have benchmarked it with smtpstone from postfix or some other tools?
Re: AnyEvent mode?
Aleksandar Lazic wrote: Do you have benchmarked it with smtpstone from postfix or some other tools? I just threw it on our spamtrap which does approx 50m emails/day.
Re: AnyEvent mode?
On Mon 06.12.2010 16:34, Matt Sergeant wrote: Aleksandar Lazic wrote: Do you have benchmarked it with smtpstone from postfix or some other tools? I just threw it on our spamtrap which does approx 50m emails/day. Do you really mean 50 Million? Wow that's a lot ;-)
Re: AnyEvent mode?
Aleksandar Lazic wrote: On Mon 06.12.2010 16:34, Matt Sergeant wrote: Aleksandar Lazic wrote: Do you have benchmarked it with smtpstone from postfix or some other tools? I just threw it on our spamtrap which does approx 50m emails/day. Do you really mean 50 Million? Yes. Wow that's a lot ;-) Not compared to another qpsmtpd spam trap I know of it's not. :-) Matt.
Re: AnyEvent mode?
On Tue, 7 Dec 2010, Matt Sergeant wrote: > Aleksandar Lazic wrote: > > On Mon 06.12.2010 16:34, Matt Sergeant wrote: > > > Aleksandar Lazic wrote: > > > > Do you have benchmarked it with smtpstone from postfix or some other > > > > tools? > > > > > > I just threw it on our spamtrap which does approx 50m emails/day. > > > > Do you really mean 50 Million? > > Yes. So why didn't you say 50M? :-) http://en.wikipedia.org/wiki/Mega- ;-)
