Re: qpsmtpd-async weird ParaDNS lookup problem + code fix
To be fair I don't do perl coding any more, so I'm happy for someone
to take over ParaDNS maintainence.
ParaDNS-XS is in my SVN server. Happy to share that with anyone who wants
it. It's basically ParaDNS using adns backend rather than Net::DNS. It's a
bit hacky and probably doesn't work 100%
Matt.
On Fri, 1 Aug 2014, David Favor wrote:
Matt Sergeant wrote:
Do you have ParaDNS::XS installed? If not, try installing it. It's
generally likely to be better than plain ParaDNS (and will be used
automatically if it's installed).
There is no ParaDNS::XS on cpan... At least nothing returned for...
http://search.cpan.org/search?query=paradnsmode=all
Unsure what this has to do with setting a list of servers for ParaDNS
to query. Whether pure perl or XS, likely there still has to be a set
of servers to query.
The fix I'm using right now is a slight patch to Qpsmtpd/PollServer.pm
which simply gives ParaDNS a list of servers to check...
my @nameservers = qw/127.0.0.1 8.8.8.8 8.8.4.4/;
my $obj = ParaDNS-new(
nameservers = \@nameservers,
finished = sub { $self-continue_read();
$self-run_hooks(connect); },
# NB: Setting remote_info to the same as remote_host
callback = sub { $conn-remote_info($conn-remote_host($_[0])); },
host = $ip,
);
This works like a charm.
Reading through the ParaDNS code, it's unclear to me how qpsmtpd can work
at all without a list of servers for ParaDNS to query.
Re: qpsmtpd-async weird ParaDNS lookup problem
Do you have ParaDNS::XS installed? If not, try installing it. It's generally likely to be better than plain ParaDNS (and will be used automatically if it's installed). On Thu, 17 Jul 2014, David Favor wrote: I've been running qpsmtpd-async for years on all sorts of servers. Likely I have something in DNS setup slightly wrong on a new server I'm setting up + what's wrong escapes me. The symptom is qpsmtpd-async hanging forever in the HELO sequence. Both forkserver + prefork work fine + async works so much better, I'd like to resolve this problem. Here's an example of the problem... Listen child making a Qpsmtpd::PollServer for 7. 11869 in config(plugins) 11869 config(plugins) returning (resolve_sender_host dont_require_anglebrackets rcpt_simple t...@newswire.net ch...@newswire.net d...@newswire.net supp...@newswire.net account...@newswire.net sa...@newswire.net debr...@newswire.net maildir /cluster/clients/ivan-budimir %d/users/%l/Maildir) from cache DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0) 11869 (connect) running plugin: resolve_sender_host 11869 (connect) resolve_sender_host: DEBUG: ip=127.0.0.1 host=localhost domain=localhost 11869 Plugin resolve_sender_host, hook connect returned DECLINED, 11869 in config(smtpgreeting) 11869 config(smtpgreeting) returning (Ready!) from cache Notice this line emitted from ParaDNS... DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0) Bind looks good... biz-net2# netstat -pluten | grep named tcp0 0 127.0.0.1:530.0.0.0:* LISTEN 10560684715/named tcp0 0 127.0.0.1:953 0.0.0.0:* LISTEN 10560734715/named tcp6 0 0 ::1:53 :::*LISTEN 10560704715/named tcp6 0 0 ::1:953 :::*LISTEN 10560744715/named udp0 0 127.0.0.1:530.0.0.0:* 10560674715/named udp6 0 0 ::1:53 :::* 10560694715/named And host seems to work too... biz-net2# host localhost localhost has address 127.0.0.1 localhost has IPv6 address ::1 biz-net2# host 127.0.0.1 1.0.0.127.in-addr.arpa domain name pointer localhost. If I turn on ParaDNS debugging (export PARADNS_DEBUG=100) I just see the lookup fail again... 100/2 [17636] dns lookup: Trying to resolve A: 127.0.0.1 100/2 [17636] dns lookup: NS Query: 127.0.0.1 (60806) DNS failure looking for 127.0.0.1 after 0 secs (looked for 1, got 0) biz-net2# perl -MParaDNS -e 'print $ParaDNS::VERSION\n' 2.0 Anyone have any suggestions?
Re: Qpsmtpd vs Haraka performance
Aleksandar Lazic wrote: Hi Matt, On Die 13.09.2011 11:10, Matt Sergeant wrote: I just benchmarked Qpsmtpd vs Haraka and thought some here might find the results interesting: http://baudehlo.wordpress.com/2011/09/13/node-js-is-fast/ (not meant as inflammatory, obviously I still have love for Qpsmtpd) Matt. How about to check the regex performance with a plugin like: check_badmailfrom_patterns check_badrcptto_patterns rcpt_regexp I think the main benefit of perl is his regex engine. Regular expressions are much faster in Javascript (well, in V8). They also suck (lots and lots of features missing). But if you don't need advanced features then they are going to be faster. Matt.
Qpsmtpd vs Haraka performance
I just benchmarked Qpsmtpd vs Haraka and thought some here might find
the results interesting:
http://baudehlo.wordpress.com/2011/09/13/node-js-is-fast/
(not meant as inflammatory, obviously I still have love for Qpsmtpd)
Matt.
FYI: Qpsmtpd plugins were: logging/warn, rcpt_all_ok[1], queue/do_nothing[2]
[1]: sub hook_rcpt { return OK }
[2]: sub hook_queue { return (OK, Queued) }
Re: qpsmtp-async forwarder
The forwarders (I supply both forwarder, after DATA, and full proxy) in Haraka is fully async. Maybe time to learn some _javascript_ :) Alister WestAugust 25, 2011 9:10 PM Hi qpsmtpd,I came across an entry in the mail-archive link where Chris Lewismentions he has customised qpsmtp-async for forwarding to a list ofother servers.http://www.nntp.perl.org/group/perl.qpsmtpd/2009/07/msg8919.htmlChris My qpsmtp-async forwarder has a config list of IPs (and ports).I am about to write something that does something very similar (aload-balancer for smtp). As I'm not very familiar with qpsmtp I waswondering if Chris is still active here and if he would mind sharinghis altered forwarder so I could use it as base for my project.Any other suggestions, or modifications to qpsmtp-async,plugins/..smtp-forward, etc. welcome.Cheers for the good work!~~c|_| alisterwest.com - mmm coffee!
Re: [PATCH] tweak QP's header handling for messages with no body
Yup. I should check if Haraka does the right thing with this too :) On Tue, 16 Aug 2011, Jared Johnson wrote: True, I was led astray by the comment that seemed to indicate it was all about headers. if the intention of the comment is correct, then it should probably if ( $in_header and ... ), whether or not the new regex is added. That block would also need to be moved to after the block that sets $in_header = 0 if we've reached the end, to avoid catching the blank line that separates headers and newlines. At any rate, Chris has a point -- I was actually thinking the other day that code like that original deferral would be better of sitting in a plugin, so I could muck around with it in a more straightforward manner in order to, say, log what we did to our own database, or change it to a rejection, or whatever :) Anyway, this is all kind of minutia compared to the original patch, which I feel I should remind everyone has to do with accepting legitimate mail from Lotus software without inadvertently stripping the headers :) -Jared That line of code doesn't look at the headers though, just at the final dot at the end-of-data. Jared Johnson mailto:jjohn...@efolder.net August 16, 2011 3:00 PM There's already a special case for something similar to this: # Reject messages that have either bare LF or CR. rjkaes noticed a # lot of spam that is malformed in the header. ($_ eq .\n or $_ eq .\r) and $self-respond(421, See http://smtpd.develooper.com/barelf.html;) and return $self-disconnect; you could just make it ( /\r\r\n$/ or $_ eq .\n or $_ eq .\r ) maybe? and update the link to point to a URL that explains both? -Jared Matt Sergeant mailto:m...@sergeant.org August 16, 2011 11:28 AM Yup there's a lot of this going around right now. Just to be explicit though, the header lines end in \r\r\n. Worth rejecting the bloody lot, frankly :) Chris Lewis mailto:cle...@nortel.com August 15, 2011 4:21 PM As a FYI, I've been seeing bot-emitted spam that appears to have extra \r at the end of _all_ header lines, and the qpsmtpd parser seems to be treating all of it as part of the _body_. IOW: except for the received line inserted by qpsmtpd, qpsmtpd doesn't see _any_ headers. This implementation is backrev (0.80 I think), and as it's only spam from one particular bot, we don't care about that particular wierdness enough to investigate further. But it's worth being aware of. Jared Johnson mailto:jjohn...@efolder.net August 15, 2011 3:39 PM Hi, We got a bug report from someone using IBM's Lotus suite (I think for both their MUA and MTA). Their users would often send messages where all the content was in the subject and they didn't bother sending any message content. I'm not sure if it's due to an apparently uncommon behavior for their particular MUA or their MTA, but every one of these messages was coming through with data in a form that looked like this: Subject: howdy\r\n.\r\n Rather than including the blank line that one might expect to follow headers, since it's required in the event that a message body is present: Subject: howdy\r\n\r\n.\r\n The customer reported these messages were having their subjects stripped; additional testing indicted all existing headers were being stripped. It looks like this is because the loop that processes message data in Qpsmtpd::SMTP::data_respond() and creates a Mail::Header object which is later used to write out the header in delivery, only works if a blank line exists after the header, e.g. the second form above. The following is all I could find in RFC 5322 that elaborated on this blank line, which obviously must exist if a message body is included: A message consists of header fields (collectively called the header section of the message) followed, optionally, by a body. The header section is a sequence of lines of characters with special syntax as defined in this specification. The body is simply a sequence of characters that follows the header section and is separated from the header section by an empty line (i.e., a line with nothing preceding the CRLF). I read this as implicitly allowing the exclusion of this blank line if there is no message body: the specification for the blank line is only mentioned in the description of the body, which is itself described as optional. Considering we haven't run into this bug in years of usage, I assume it's unconventional to exclude the blank line, but it looks like it is legitimate syntax. At any rate this was effecting multiple legitimate end users so we put together the attached patch, which pulls the header building into its own sub which is then called inside the loop if we
Re: [PATCH] tweak QP's header handling for messages with no body
Yup there's a lot of this going around right now. Just to be explicit though, the header lines end in \r\r\n. Worth rejecting the bloody lot, frankly :) Chris LewisAugust 15, 2011 4:21 PM As a FYI, I've been seeing bot-emitted spam that appears to have extra \r at the end of _all_ header lines, and the qpsmtpd parser seems to be treating all of it as part of the _body_. IOW: except for the received line inserted by qpsmtpd, qpsmtpd doesn't see _any_ headers. This implementation is backrev (0.80 I think), and as it's only spam from one particular bot, we don't care about that particular wierdness enough to investigate further. But it's worth being aware of. Jared JohnsonAugust 15, 2011 3:39 PM Hi,We got a bug report from someone using IBM's Lotus suite (I think for boththeir MUA and MTA). Their users would often send messages where all thecontent was in the subject and they didn't bother sending any messagecontent. I'm not sure if it's due to an apparently uncommon behavior fortheir particular MUA or their MTA, but every one of these messages wascoming through with data in a form that looked like this:"Subject: howdy\r\n.\r\n"Rather than including the blank line that one might expect to followheaders, since it's required in the event that a message body is present:"Subject: howdy\r\n\r\n.\r\n"The customer reported these messages were having their subjects stripped;additional testing indicted all existing headers were being stripped. Itlooks like this is because the loop that processes message data inQpsmtpd::SMTP::data_respond() and creates a Mail::Header object which islater used to write out the header in delivery, only works if a blank lineexists after the header, e.g. the second form above. The following is allI could find in RFC 5322 that elaborated on this blank line, whichobviously must exist if a message body is included:"A message consists of header fields (collectively called "the headersection of the message") followed, optionally, by a body. The headersection is a sequence of lines of characters with special syntax asdefined in this specification. The body is simply a sequence ofcharacters that follows the header section and is separated from theheader section by an empty line (i.e., a line with nothing preceding theCRLF)."I read this as implicitly allowing the exclusion of this blank line ifthere is no message body: the specification for the blank line is onlymentioned in the description of the body, which is itself described asoptional. Considering we haven't run into this bug in years of usage, Iassume it's unconventional to exclude the blank line, but it looks like itis legitimate syntax.At any rate this was effecting multiple legitimate end users so we puttogether the attached patch, which pulls the header building into its ownsub which is then called inside the loop if we reach the blank lineindicating the header section is complete; otherwise, it's called outsideof the loop if we have no more message data, indicating the header sectionis complete. Sorry I'm not putting this on a github fork, I still don'thave my git stuff together, I may never get around to it but I thought youguys might find this useful.-Jared
Re: [PATCH] tweak QP's header handling for messages with no body
That line of code doesn't look at the headers though, just at the final dot at the end-of-data. Jared JohnsonAugust 16, 2011 3:00 PM There's already a special case for something similar to this:# Reject messages that have either bare LF or CR. rjkaes noticed a# lot of spam that is malformed in the header.($_ eq ".\n" or $_ eq ".\r")and $self-respond(421, "Seehttp://smtpd.develooper.com/barelf.html") and return $self-disconnect;you could just make it ( /\r\r\n$/ or $_ eq ".\n" or $_ eq ".\r" ) maybe? and update the link to point to a URL that explains both?-Jared Matt SergeantAugust 16, 2011 11:28 AM Yup there's a lot of this going around right now. Just to be explicit though, the header lines end in \r\r\n. Worth rejecting the bloody lot, frankly :) Chris LewisAugust 15, 2011 4:21 PM As a FYI, I've been seeing bot-emitted spam that appears to have extra \r at the end of _all_ header lines, and the qpsmtpd parser seems to be treating all of it as part of the _body_. IOW: except for the received line inserted by qpsmtpd, qpsmtpd doesn't see _any_ headers. This implementation is backrev (0.80 I think), and as it's only spam from one particular bot, we don't care about that particular wierdness enough to investigate further. But it's worth being aware of. Jared JohnsonAugust 15, 2011 3:39 PM Hi,We got a bug report from someone using IBM's Lotus suite (I think for boththeir MUA and MTA). Their users would often send messages where all thecontent was in the subject and they didn't bother sending any messagecontent. I'm not sure if it's due to an apparently uncommon behavior fortheir particular MUA or their MTA, but every one of these messages wascoming through with data in a form that looked like this:"Subject: howdy\r\n.\r\n"Rather than including the blank line that one might expect to followheaders, since it's required in the event that a message body is present:"Subject: howdy\r\n\r\n.\r\n"The customer reported these messages were having their subjects stripped;additional testing indicted all existing headers were being stripped. Itlooks like this is because the loop that processes message data inQpsmtpd::SMTP::data_respond() and creates a Mail::Header object which islater used to write out the header in delivery, only works if a blank lineexists after the header, e.g. the second form above. The following is allI could find in RFC 5322 that elaborated on this blank line, whichobviously must exist if a message body is included:"A message consists of header fields (collectively called "the headersection of the message") followed, optionally, by a body. The headersection is a sequence of lines of characters with special syntax asdefined in this specification. The body is simply a sequence ofcharacters that follows the header section and is separated from theheader section by an empty line (i.e., a line with nothing preceding theCRLF)."I read this as implicitly allowing the exclusion of this blank line ifthere is no message body: the specification for the blank line is onlymentioned in the description of the body, which is itself described asoptional. Considering we haven't run into this bug in years of usage, Iassume it's unconventional to exclude the blank line, but it looks like itis legitimate syntax.At any rate this was effecting multiple legitimate end users so we puttogether the attached patch, which pulls the header building into its ownsub which is then called inside the loop if we reach the blank lineindicating the header section is complete; otherwise, it's called outsideof the loop if we have no more message data, indicating the header sectionis complete. Sorry I'm not putting this on a github fork, I still don'thave my git stuff together, I may never get around to it but I thought youguys might find this useful.-Jared
Re: thoughts about a new module called check_spammer_connect
On Wed, 27 Jul 2011, Jared Johnson wrote: That sounds like a pretty sweet configuration! [Note if you are running qpsmtpd-async, as we do, it's not really possible to route DNS queries differently for DNSBLs versus other DNS queries qpsmtpd does. ParaDNS doesn't handle paralleled DNS queries to different servers well.] when I first mucked around with the uribl plugin (see other threads), I switched the non-async plugin to use Net::DNS::Async, but as far as I could tell, N::D::A wasn't truly async, so I left the async plugin using ParaDNS. As it turns out, one of my associates since determined a way to use Net::DNS::Async in a truly async fashion. We're still using the prefork daemon, but when we inevitably switch to async I'll probably try to switch the async to N::D::A. We also have a consolidated feed so the multi-feed thing isn't too much of a concern, but N::D::A just seems more straightforward. If anyone is interested in switching to N::D::A now, feel free to ping me and I can get the details on how to use it in a way that doesn't break the true async plugin... I'm also curious if anyone knows of reasons why this switch would not be such a good idea ;) I have an XS version of ParaDNS too, which may be useful to people. It's in the ParaDNS subversion repository. We use it at work and it seems stable now. However I'm unlikely to maintain much on Qpsmtpd now that Haraka has taken off. Matt.
Re: thoughts about a new module called check_spammer_connect
On Wed, 27 Jul 2011, David Nicol wrote: On Wed, Jul 27, 2011 at 2:05 PM, Matt Sergeant m...@sergeant.org wrote: However I'm unlikely to maintain much on Qpsmtpd now that Haraka has taken off. Matt. how about a plugin adapter, so Haraka can use Qpsmtpd plugins or v/v? That probably implies either a node.js -- perl integration layer, a V8 -- perl integration layer, or use of a more arm's-length plugin architecture (like sendmail's Milter) around both parts. Haraka ships with both smtp_forward and smtp_proxy plugins (with tradeoffs for each), so there's no reason you wouldn't just run both, if you needed qpsmtpd plugins too. Matt.
Re: [Fwd: Re: [Fwd: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd]]
Jared Johnson wrote: I ... disagree. From my reading of plugins/tls, it looks like there is no problem at all, in the non-async code path. It resets STDIN and STDOUT to a socket created from scratch by the IO::Socket::SSL module. I haven't looked at IO::Socket::SSL to see if it has this sort of issue, but it seems unlikely to me. Ah good news then. Hopefully someone will apply my pull request for that patch. Matt.
Re: [Fwd: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd]
No takers? I do consider the bug fairly minor (it's not like a remote
root or anything)... But still?
Matt Sergeant wrote:
I'm forwarding this to the list since I didn't get a response from Ask...
The problem here is when someone sends the following packet:
STARTTLS\nSOME_COMMAND\n
The SOME_COMMAND bit gets cached internally (in PollServer/async
that's in $qp-{line}, but in the other implementations I have no idea
what happens) and then because SSL upgrading happens in external
libraries, they don't see these bits of data, so what then happens is
ssl gets upgraded, but then SOME_COMMAND gets executed after SSL has
been negotiated.
I'm pretty sure this is enough to fix it for async:
diff --git a/plugins/tls b/plugins/tls
index 37fbc9a..f850d2c 100644
--- a/plugins/tls
+++ b/plugins/tls
@@ -275,6 +275,7 @@ sub upgrade_socket {
my UpgradeClientSSL $self = shift;
unless ( $self-{_ssl_started} ) {
+$self-{_stashed_qp}-clear_data();
IO::Socket::SSL-start_SSL(
$self-{_stashed_qp}-{sock}, {
SSL_use_cert = 1,
I've submitted a pull request for that.
But for the non-async scenario I think it's a lot more complex because
the caching would be done at the C-level, so a fix more like the fix
(posted below) for postfix is required (switch to non-blocking, get
whatever data is remaining, flush it, switch back off non-blocking).
Original Message
From: Graham Todd gt...@iciti.ca
Subject: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl
qpsmtpd
Date: Wed, 01 Jun 2011 15:03:51 -0400
To: m...@fehcom.de, ga...@freebsd.org, Matt Sergeant
m...@sergeant.org
Hi I'm a user of the qpsmtpd and qmail-spamcontrol ports on FreeBSD
(ga...@freebsd.org is the qmail-spamcontrol port maintainer), thanks to
you all for making qmail even more excellent by integrating all the
patches and making it easy to build and install on FreeBSD and using perl
to make it as flexible as Apache ;-)
As I understand it STARTTLS support in qmail-spamcontrol is handled by
ucspi-ssl with TLS patches. We're currently using qpsmtpd which has a
plugin that wraps qmail with perl based support for STARTTLS (using
IO::Socket::SSL).
Does Wietse Venema's STARTTLS vulnerability impact the STARTTLS
enhancements used in ucspi-ssl, qpsmtpd or qmail-spamcontrol? The
vulnerability is described here:
http://www.securityfocus.com/bid/46767
There is a fairly simple patch that fixes the TLS vulnerability and
applies to netqmail-1.06-tls - I am wondering how the vulnerabilty might
effect TLS functions in qpsmtpd and qmail-spamcontrol. The patch for TLS
enhanced versions of vanilla qmail-1.03 is here:
http://marc.info/?l=qmail-ldapm=130564281418177w=2
There's a long list of spots to look for the vulnerability on the
Security
Focus site but neither qpsmtpd nor ucspi-ssl-tls are mentioned.
Cheers,
gtodd
[Fwd: STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd]
I'm forwarding this to the list since I didn't get a response from Ask...
The problem here is when someone sends the following packet:
STARTTLS\nSOME_COMMAND\n
The SOME_COMMAND bit gets cached internally (in PollServer/async that's
in $qp-{line}, but in the other implementations I have no idea what
happens) and then because SSL upgrading happens in external libraries,
they don't see these bits of data, so what then happens is ssl gets
upgraded, but then SOME_COMMAND gets executed after SSL has been negotiated.
I'm pretty sure this is enough to fix it for async:
diff --git a/plugins/tls b/plugins/tls
index 37fbc9a..f850d2c 100644
--- a/plugins/tls
+++ b/plugins/tls
@@ -275,6 +275,7 @@ sub upgrade_socket {
my UpgradeClientSSL $self = shift;
unless ( $self-{_ssl_started} ) {
+$self-{_stashed_qp}-clear_data();
IO::Socket::SSL-start_SSL(
$self-{_stashed_qp}-{sock}, {
SSL_use_cert = 1,
I've submitted a pull request for that.
But for the non-async scenario I think it's a lot more complex because
the caching would be done at the C-level, so a fix more like the fix
(posted below) for postfix is required (switch to non-blocking, get
whatever data is remaining, flush it, switch back off non-blocking).
Original Message
From: Graham Todd gt...@iciti.ca
Subject:STARTTLS vulnerabilty and qmail-spamcontrol ucspi-ssl qpsmtpd
Date: Wed, 01 Jun 2011 15:03:51 -0400
To: m...@fehcom.de, ga...@freebsd.org, Matt Sergeant m...@sergeant.org
Hi I'm a user of the qpsmtpd and qmail-spamcontrol ports on FreeBSD
(ga...@freebsd.org is the qmail-spamcontrol port maintainer), thanks to
you all for making qmail even more excellent by integrating all the
patches and making it easy to build and install on FreeBSD and using perl
to make it as flexible as Apache ;-)
As I understand it STARTTLS support in qmail-spamcontrol is handled by
ucspi-ssl with TLS patches. We're currently using qpsmtpd which has a
plugin that wraps qmail with perl based support for STARTTLS (using
IO::Socket::SSL).
Does Wietse Venema's STARTTLS vulnerability impact the STARTTLS
enhancements used in ucspi-ssl, qpsmtpd or qmail-spamcontrol? The
vulnerability is described here:
http://www.securityfocus.com/bid/46767
There is a fairly simple patch that fixes the TLS vulnerability and
applies to netqmail-1.06-tls - I am wondering how the vulnerabilty might
effect TLS functions in qpsmtpd and qmail-spamcontrol. The patch for TLS
enhanced versions of vanilla qmail-1.03 is here:
http://marc.info/?l=qmail-ldapm=130564281418177w=2
There's a long list of spots to look for the vulnerability on the Security
Focus site but neither qpsmtpd nor ucspi-ssl-tls are mentioned.
Cheers,
gtodd
Re: smtp proxy to external smtp server
What do you mean by signed? Do you mean like adding a banner to the text parts of the email? If so, that's a really hard problem (I mean it's doable in simple situations, but breaks very very easily). Mike Korizek wrote: On 05/17/2011 04:24 PM, Matt Sergeant wrote: It can be done, but you'll need to customise the smtp-forward plugin yourself to do it. I checked the transaction object, I could not find a handle to the message. How can I achieve the following: An email shall be signed and then put back to the queue. Before signing the email I parse it with a MIME::Parser, but how can I put the new email back to the queue? Thanks for any hint. Mike
Re: smtp proxy to external smtp server
Aleksandar Lazic wrote: Just for my curiosity, why don't you use qpsmtpd::smtp-forward =Any MTA Setup (postfix,courier,qmail, ...)? It's not sender dependent, and doesn't pass on AUTH. (but would be easily hackable to do that).
Announce: Haraka
Some of you may be interested in this... I decided I wanted to hack on node.js to see what all the fuss is about. So to do that I have basically ported Qpsmtpd to Node.js (and given it a decent name while doing so!). It's still early days - there are no plugins to speak of yet (i.e. no queue plugins at all yet), but you might be interested in just looking anyway: https://github.com/baudehlo/Haraka
Re: Announce: Haraka
Guy Hulbert wrote: https://github.com/baudehlo/Haraka Had a look. I recognize bits. Do you have any feeling for how easy it is to code versus perl Once you get used to the idiosyncrasies of Javascript, just as easy really. Took me a while to understand the object model, but everything seems to work ok now I mostly figured it out. There's still stuff I have no idea how to do (the lack of caller telling me line numbers is annoying for example), but I'm getting there. The other big thing is the libraries are simultaneously both sparse and weak. There's a CPAN equivalent (called npm) but it's still very immature in terms of what's available. Even the core libraries are missing some fundamental stuff (like I can't do TLS as they removed support for upgrading a single client connection to TLS). and how the performance might be versus qpsmtpd. It pretty much blows it away. Node can be pretty much as fast as a lisp or smalltalk implementation. All that research into dynamic languages is all going into Javascript these days, and none is going into perl (some is going into Python and Ruby, but not as much as JS). So I think it will always be quite a bit faster. For reference I can (albeit without plugins) throw about 5000 mails/sec through it fairly easily it seems, though that's with persistent connections, and on localhost. Performance was my main motivation for doing it. The other thing is that everyone doing node.js is doing event programming, so all those libraries for database access and memcached/redis access and so on are all async already. Matt.
Re: check_badrcptto, to prevent backscatter
Should we have plugins/qmail and plugins/postfix dirs? Todd Brunhoff wrote: Tim's view seems appropriate. His script is centered on qmail, and mine is centered on postfix (or more specifically, on /etc/aliases). Both scripts are probably best in their current form with appropriate disclosures. Let me know if there is any prep work you would like me to do to my script. Todd On 3/9/2011 7:23 AM, Tim Meadowcroft wrote: On Sunday 06 March 2011 06:42:50 Robert Spier wrote: Todd Brunhoff wrote: Your scripts look like they have a good deal of qmail sophistication. Some years ago I ran qmail 1.0.3, after each major system crash, I would revisit whether to use qmail, and eventually decided to switch to qpsmtp+postfix because both seem to have better support. And in fact, the reason I included /etc/alias was to replace the very useful alias mechanism in qmail. I really didn't need much, so that was sufficient for me. So it seems that among these collections of scripts there are backscatter solutions for qmail sites and qpsmtp sites. Perhaps one of the developers can fold these into a contrib folder? A lot of plugins are linked from the wiki, http://wiki.qpsmtpd.org. check_goodrcptto looks like something that might be worth having in core. Tim and Todd, are you interested in reconciling the differences between your versions? (Maybe some sort of configuration interface?) Mine is very qmail specific - I sort of feel generalising it would make it worse. Either you use qmail, in which case you might like it as it stands, (possibly in addition to other recipient checks) or you don't use qmail, in which case you can ignore it completely. I've posted the source (with disclaimers - I'm still on qpsmtpd v0.28) at http://schmerg.com/checkgoodrcptto-a-qpsmtpd-plugin-for-checking and will see about adding it to the wiki when I can set up an account there. Cheers -- Tim
Re: rolling uribl and rbl plugins into a single plugin
Jared Johnson wrote: So our organization is planning doing some big changes to the rbl plugin and it dawned on us that it seems a lot easier to just add an earlier hook to the existing uribl plugin (and rename it to rbl? or bl? or something?). But of course I still have in mind that someday I'll get the plugin completely in shape and it will be in QP, and we'll get to share code and all. Would this be an undesirable direction to take the uribl plugin in for QP proper, in the event that the uribl plugin was integrated into QP proper? It seems like it would be handy in terms of both code organization and features (the main feature I can think of is proper mask evaluation along with custom actions per mask, just like in the uribl plugin). Thoughts? I'd much prefer they both used plugin inheritance to access shared code than have them merged.
Re: rctpto rejection based on smtp verify?
Charlie Brady wrote: On Fri, 14 Jan 2011, Nicholas Lee wrote: Is there a plugin that will check rcpt addresses against a back end smtp server? I presume you intend to eventually deliver to the same smtp backend, via the smtp-forward plugin. I've long argued that the smtp-forward plugin should hook into different phases of the transaction, and do the recipient validation directly against the backend: http://www.nntp.perl.org/group/perl.qpsmtpd/2008/11/msg8288.html I think it should be optional, but yeah. There are downsides (such as keeping open the SMTP connection with the backend), but it'd be nice if you could choose. Matt.
Re: AnyEvent mode?
Aleksandar Lazic wrote: On Mon 06.12.2010 16:34, Matt Sergeant wrote: Aleksandar Lazic wrote: Do you have benchmarked it with smtpstone from postfix or some other tools? I just threw it on our spamtrap which does approx 50m emails/day. Do you really mean 50 Million? Yes. Wow that's a lot ;-) Not compared to another qpsmtpd spam trap I know of it's not. :-) Matt.
Re: AnyEvent mode?
Aleksandar Lazic wrote: Do you have benchmarked it with smtpstone from postfix or some other tools? I just threw it on our spamtrap which does approx 50m emails/day.
Re: AnyEvent mode?
On Thu, 2 Dec 2010, Aleksandar Lazic wrote:
On Don 02.12.2010 19:04, Matt Sergeant wrote:
On Thu, 2 Dec 2010, Ask Bjørn Hansen wrote:
On Dec 2, 2010, at 10:37, Aleksandar Lazic wrote:
Maybe we can make another benchmark AnyEvent vs Danga::Socket due to
the fact that AnyEvent with EV as underlaying event lib looks very fast
from the internet source ;-)
Matt was (I'm guessing) testing a load that's artificial to anyone who's
not just archiving spam from a spam trap. For the rest of us by far most
of the resources go to the various spam filtering stuff, so the
performance will be fine.
More important is the ease of use of the APIs, the general eco system etc.
On those AnyEvent wins (in my opinion).
Sorry yeah I think I have it working, but our work data centre is down
right now so I can't get at the files :-(
Will email the latest ones here when I get the chance.
Thanks, I'am quite interested.
OK, the relevant files are attached. Nothing else needed changing I don't
think.
AnyEvent.pm has to go in the lib/Qpsmtpd/ dir.#!/usr/bin/perl
use lib ./lib;
BEGIN {
delete $ENV{ENV};
delete $ENV{BASH_ENV};
$ENV{PATH} = '/bin:/usr/bin:/var/qmail/bin:/usr/local/bin';
}
# Profiling - requires Devel::Profiler 0.05
#BEGIN { $Devel::Profiler::NO_INIT = 1; }
#use Devel::Profiler;
use strict;
use vars qw($DEBUG);
use FindBin qw();
# TODO: need to make this taint friendly
use lib $FindBin::Bin/lib;
use Qpsmtpd::AnyEvent;
use Qpsmtpd::ConfigServer;
use Qpsmtpd::Constants;
use Carp;
use POSIX qw(WNOHANG);
use Getopt::Long;
use List::Util qw(shuffle);
use Socket;
use AnyEvent::Socket;
use AnyEvent::Util;
$|++;
$SIG{'PIPE'} = IGNORE; # handled manually
$DEBUG = 0;
my $PORT= 2525;
my $LOCALADDR = '0.0.0.0';
my $PROCS = 1;
my $USER= (getpwuid $)[0]; # user to suid to
$USER= smtpd if $USER eq root;
my $PAUSED = 0;
my $NUMACCEPT = 20;
my $PID_FILE= '';
my $ACCEPT_RSET;
my $DETACH; # daemonize on startup
# make sure we don't spend forever doing accept()
use constant ACCEPT_MAX = 1000;
sub reset_num_accept {
$NUMACCEPT = 20;
}
sub help {
print EOT;
Usage:
qpsmtpd [OPTIONS]
Options:
-l, --listen-address addr : listen on a specific address; default 0.0.0.0
-p, --port P : listen on a specific port; default 2525
-u, --user U : run as a particular user; defualt 'smtpd'
-j, --procs J : spawn J processes; default 1
-d, --detach : detach from controlling terminal (daemonize)
--pid-file P : print main servers PID to file P
-h, --help: this page
EOT
exit(0);
}
GetOptions(
'p|port=i' = \$PORT,
'l|listen-address=s'= \$LOCALADDR,
'j|procs=i' = \$PROCS,
'v|verbose+' = \$DEBUG,
'u|user=s' = \$USER,
'pid-file=s'= \$PID_FILE,
'd|detach' = \$DETACH,
'h|help'= \help,
) || help();
# detaint the commandline
if ($PORT =~ /^(\d+)$/) { $PORT = $1 } else { help }
if ($LOCALADDR =~ /^([\d\w\-.]+)$/) { $LOCALADDR = $1 } else { help }
if ($USER =~ /^([\w\-]+)$/) { $USER = $1 } else { help }
if ($PROCS =~ /^(\d+)$/) { $PROCS = $1 } else { help }
use constant READY = 1;
use constant ACCEPTING = 2;
use constant RESTARTING = 999;
my $CURRENT_PROCS = 0;
my %childstatus = ();
my $SERVER;
if ($PID_FILE -r $PID_FILE) {
open PID, $PID_FILE
or die open_pidfile $PID_FILE: $!\n;
my $running_pid = PID || ''; chomp $running_pid;
if ($running_pid =~ /^(\d+)/) {
if (kill 0, $running_pid) {
die Found an already running qpsmtpd with pid $running_pid.\n;
}
}
close(PID);
}
run_as_server();
exit(0);
sub spawn_child {
my ($plugin_loader) = @_;
my $pid = fork;
if ($pid) {
$childstatus{$pid}++;
$CURRENT_PROCS++;
return $pid;
}
elsif (!defined($pid)) {
die Unable to fork;
}
$plugin_loader-run_hooks('post-fork');
accept_loop();
}
sub accept_loop {
my $w = AE::io $SERVER, 0, sub {
print Connect?\n;
while ($SERVER (my $peer = accept my $fh, $SERVER)) {
fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside
world does not
my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
my $qp = Qpsmtpd::AnyEvent-new($fh, format_address($host),
$service);
$qp-process_line(Connect);
}
};
AnyEvent-condvar-wait;
exit;
}
sub sig_hup {
kill 1, keys %childstatus;
}
sub sig_chld {
my $spawn_count = 0;
while ( (my $child = waitpid(-1,WNOHANG)) 0) {
if (!defined $childstatus{$child}) {
next;
}
last unless $child 0;
print SIGCHLD: child $child died\n;
delete $childstatus{$child};
$CURRENT_PROCS--;
}
$SIG
Re: Exim alternatives?
I once wrote a very simple perl module which basically did outbound mail queueing for very simple needs... But then I discovered my needs were a bit more complex. LOL... I second Ask's question - what's wrong with using exim? David Favor wrote: I'm looking for a simple alternative to exim, sendmail, postfix for outgoing email. So when and email comes into qpsmtpd which forwards offsite, an outgoing mail server to forward these email. If someone has a suggestion about other options, please let me know. Thanks.
Re: Exim alternatives?
On Tue, 14 Sep 2010, David Favor wrote: The problem is two fold. 1) I could never get a straight answer about the correct configuration from the exim folks. 2) The config I have works for weeks to months, then develops odd (bitrot) challenges which seem to relate to DNS MX server changes in domains, which exim never senses. For example, message is queued and can't reach an MX record, then continually tries the bad MX record, never attempting to look up a new record. I'm switching back to TipJar::MTA to do some testing. What I require is a simple system, preferably perl based, so it's fairly easy to understand + configure... that simply works... all the time... no fuss, no muss. OK... try my MrQueue stuff: svn co svn://axkit.org/MrQueue (axkit.org is having a bit of trouble resolving right now, so try sergeant.org if that doesn't work - same host). Matt.
Re: Rewritten URIBL plugin
Jared Johnson wrote:
sub parse_mime {
That works, only this should be called parsed_mime because you're
asking for the parsed bit, not telling it to parse (every time).
Matt.
Re: Rewritten URIBL plugin
On Sun, 25 Jul 2010, Jared Johnson wrote: The plugin has the following advantages over the original: - Uses MIME::Parser to unpack message text so that we can look for URI's in base64-encoded data, etc., and _not_ look for URI's in noise. I think we should probably consider putting support for parsed messages into core, with the parsing done lazily if requested by the API. Thoughts? Matt.
Re: Redirecting email
Chris Lewis wrote: The version I've derived from Steve's works during hook_rcpt. Yes, but assigning to $_[0] is a horrible way to do it. We should have a supported and cleaner way to modify the email address at RCPT TO time... Ah, we do have a supported way... a hook called rcpt_pre which returns OK, $new_address (with angle brackets). Given the way my qpsmtpd works, hook_data is way way too late. Too much has already been decided by that point. I don't mean data_post - I mean right on the DATA command, which happens immediately after the RCPT TOs in a normal SMTP transaction... But then see above for the right solution. Matt.
spool files not temp?
Hi, Can anyone remember the reason that the spool files aren't proper temp files (deleted upon open)? We often end up with a hard restart of qpsmtpd and having these files left around is annoying... Matt.
Re: I am probably doing something wrong
Steve wrote: Note that qpsmtpd probably should reject the message as invalid format. If it doesn't do that in the core, at least a standard plugin should do that validation. The message is not rejected. Anyway... what plugin is doing the validation? Can you point me to the plugin doing such a validation? There isn't one. Charlie is saying there should be such a plugin. Though honestly it should probably be in core not a plugin. Matt.
Re: New plugin: smtptls-forward
On Tue, 30 Mar 2010, Charlie Brady wrote: On Thu, 25 Mar 2010, Jason Mills wrote: I wrote this plugin to help me with my local debugging. Basically a heavily modified version of smtp-foward. I'd recommend you search the archives and find some earlier comments by me about smtp-forward. It really should be rewritten to hook into more than just the queue hook, and pass sender and recipient addresses to the backend as they are provided by the connecting client, and relay responses back to the client. I think some people might not want that, as they want to reduce the load on their main SMTP server... I agree it should be an option though. Perhaps we need queue/smtp-passthru. Matt.
ParaDNS 2.0 - now does dns0x20
dns0x20 protects against things like the Kaminsky DNS attack by vastly increasing the size of the keyspace for DNS requests. It's enabled by default in ParaDNS 2.0. You can disable with an env var. Matt.
Re: Release soon
On Fri, 12 Feb 2010, Ask Bjørn Hansen wrote: Hi everyone, I'm going to make a release soon, so if you have patches that aren't merged into my branch yet that you think should be, be sure to speak up! I just merged the RPM packaging stuff Peter Holzer and Robin Bowes made (over the last 5 years; don't say anything is being rushed around here!) :-) The shortlog since the last release (so far) is below. I have the AnyEvent version of Qpsmtpd working if people are interested in getting it added to core... It's slower than the Danga::Socket version though. Matt.
Re: badmailfrom per domain
Johan Almqvist wrote:
On 21. jan. 2010, at 16.57, Christian Herndler wrote:
I get a lot of spam where the sender uses a whole /24 Subnet as mail
relay, the helo uses the pattern
mx{last-octet-of-ip}.domainname.net
so it could be blocked using the check_spamhelo plugin, but to do that
this plugin would need a small change as it only works with hostnames.
I've had a similar problem at the MAIL FROM level (right-hand side VERP, you
could say) and fixed this with this plugin:
http://github.com/tyskjohan/qpsmtpd/blob/master/plugins/check_badmailfrom_patterns
Also consider using Enemies List on the HELO. It's very effective.
ParaDNS 1.9 out
This fixes a major bug when under high load causing overly quick timeouts for some requests. Recommended upgrade. (only used by those using -async) Matt.
Re: AnyEvent mode?
OK, here's the files as they currently stand. The big note on this is that
I have done very little testing, and most importantly I have NOT updated
any of the plugins (see the async/* dir for those that need re-written to
use AnyEvent, mostly to use AnyEvent::DNS. Also the tls plugin would
need updated/rewritten, though it's probably a lot simpler with AnyEvent).
I'll be trying to do performance testing tomorrow (comparing with
qpsmtpd-async, NOT any of the other models).
Matt.#!/usr/bin/perl
use lib ./lib;
BEGIN {
delete $ENV{ENV};
delete $ENV{BASH_ENV};
$ENV{PATH} = '/bin:/usr/bin:/var/qmail/bin:/usr/local/bin';
}
# Profiling - requires Devel::Profiler 0.05
#BEGIN { $Devel::Profiler::NO_INIT = 1; }
#use Devel::Profiler;
use strict;
use vars qw($DEBUG);
use FindBin qw();
# TODO: need to make this taint friendly
use lib $FindBin::Bin/lib;
use Qpsmtpd::AnyEvent;
use Qpsmtpd::ConfigServer;
use Qpsmtpd::Constants;
use Carp;
use POSIX qw(WNOHANG);
use Getopt::Long;
use List::Util qw(shuffle);
use Socket;
use AnyEvent::Socket;
use AnyEvent::Util;
$|++;
$SIG{'PIPE'} = IGNORE; # handled manually
$DEBUG = 0;
my $PORT= 2525;
my $LOCALADDR = '0.0.0.0';
my $PROCS = 1;
my $USER= (getpwuid $)[0]; # user to suid to
$USER= smtpd if $USER eq root;
my $PAUSED = 0;
my $NUMACCEPT = 20;
my $PID_FILE= '';
my $ACCEPT_RSET;
my $DETACH; # daemonize on startup
# make sure we don't spend forever doing accept()
use constant ACCEPT_MAX = 1000;
sub reset_num_accept {
$NUMACCEPT = 20;
}
sub help {
print EOT;
Usage:
qpsmtpd [OPTIONS]
Options:
-l, --listen-address addr : listen on a specific address; default 0.0.0.0
-p, --port P : listen on a specific port; default 2525
-u, --user U : run as a particular user; defualt 'smtpd'
-j, --procs J : spawn J processes; default 1
-d, --detach : detach from controlling terminal (daemonize)
--pid-file P : print main servers PID to file P
-h, --help: this page
EOT
exit(0);
}
GetOptions(
'p|port=i' = \$PORT,
'l|listen-address=s'= \$LOCALADDR,
'j|procs=i' = \$PROCS,
'v|verbose+' = \$DEBUG,
'u|user=s' = \$USER,
'pid-file=s'= \$PID_FILE,
'd|detach' = \$DETACH,
'h|help'= \help,
) || help();
# detaint the commandline
if ($PORT =~ /^(\d+)$/) { $PORT = $1 } else { help }
if ($LOCALADDR =~ /^([\d\w\-.]+)$/) { $LOCALADDR = $1 } else { help }
if ($USER =~ /^([\w\-]+)$/) { $USER = $1 } else { help }
if ($PROCS =~ /^(\d+)$/) { $PROCS = $1 } else { help }
use constant READY = 1;
use constant ACCEPTING = 2;
use constant RESTARTING = 999;
my $CURRENT_PROCS = 0;
my %childstatus = ();
my $SERVER;
if ($PID_FILE -r $PID_FILE) {
open PID, $PID_FILE
or die open_pidfile $PID_FILE: $!\n;
my $running_pid = PID || ''; chomp $running_pid;
if ($running_pid =~ /^(\d+)/) {
if (kill 0, $running_pid) {
die Found an already running qpsmtpd with pid $running_pid.\n;
}
}
close(PID);
}
run_as_server();
exit(0);
sub spawn_child {
my ($plugin_loader) = @_;
my $pid = fork;
if ($pid) {
$childstatus{$pid}++;
$CURRENT_PROCS++;
return $pid;
}
elsif (!defined($pid)) {
die Unable to fork;
}
$plugin_loader-run_hooks('post-fork');
accept_loop();
}
sub accept_loop {
my $w = AE::io $SERVER, 0, sub {
print Connect?\n;
while ($SERVER (my $peer = accept my $fh, $SERVER)) {
fh_nonblocking $fh, 1; # POSIX requires inheritance, the outside
world does not
my ($service, $host) = AnyEvent::Socket::unpack_sockaddr $peer;
my $qp = Qpsmtpd::AnyEvent-new($fh, format_address($host),
$service);
$qp-process_line(Connect);
}
};
AnyEvent-condvar-wait;
exit;
}
sub sig_hup {
kill 1, keys %childstatus;
}
sub sig_chld {
my $spawn_count = 0;
while ( (my $child = waitpid(-1,WNOHANG)) 0) {
if (!defined $childstatus{$child}) {
next;
}
last unless $child 0;
print SIGCHLD: child $child died\n;
delete $childstatus{$child};
$CURRENT_PROCS--;
}
$SIG{CHLD} = \sig_chld;
}
sub HUNTSMAN {
$SIG{CHLD} = 'DEFAULT';
kill 'INT' = keys %childstatus;
if ($PID_FILE -e $PID_FILE) {
unlink $PID_FILE or ::log(LOGERROR, unlink: $PID_FILE: $!);
}
exit(0);
}
sub _connect_sock {
my $ipn = parse_address($LOCALADDR) or die cannot parse '$LOCALADDR' as
host address;
my $af = address_family $ipn;
socket(my $sock, $af, SOCK_STREAM, 0) or die socket: $!;
setsockopt($sock, SOL_SOCKET, SO_REUSEADDR, 1)
or die tcp_server/so_reuseaddr: $!;
bind $sock,
AnyEvent mode?
Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written (mostly hacked right now, but should work). In theory it might be faster than the Danga::Socket based one, and AnyEvent seems to receive regular updates more than Danga::Socket these days. Matt.
Re: AnyEvent mode?
On Mon, 23 Nov 2009, Ask Bjørn Hansen wrote: On Nov 23, 2009, at 11:41, Matt Sergeant wrote: Is anyone interested in an AnyEvent mode Qpsmtpd? I have the code written (mostly hacked right now, but should work). In theory it might be faster than the Danga::Socket based one, and AnyEvent seems to receive regular updates more than Danga::Socket these days. There are also more other stuff available with AnyEvent -- I think it'd be cool! OK. How do I get this to you? It's basically just two new files, no patches to anything. Matt.
Re: Language detection?
On Sat, 7 Nov 2009, Johan Almqvist wrote: Hi On 5. nov. 2009, at 16.33, Matt Sergeant wrote: Anyone got a language detection plugin? I get a lot of spam slipping through in spanish, and I'd like to just whack it on the head. There's http://spamassassin.apache.org/full/3.1.x/doc/Mail_SpamAssassin_Plugin_TextCat.html but not sure if that's what you want. Yeah there's plenty of ways to do it. Textcat being one of them. Google also has an API for language detection which is pretty good. I just wondered if anyone had anything pre-rolled for qpsmtpd. Matt.
Re: [PATCH] Log even when we aren't in a transaction
On Mon, 24 Aug 2009 09:55:52 -0500, Jared Johnson wrote: On 08/23/2009 04:03 PM, Matt Sergeant wrote: On Thu, 20 Aug 2009 10:18:39 -0500, Jared Johnson wrote: It looks like logging/file doesn't like the empty hashref returned by Qpsmtpd::transaction(). I never understood why it did that. Any reason it can't return either undef or (preferably) a new Transaction object? I don't really understand it either, and I fear that which I don't understand, so I worry about taking it out and breaking some hackage that depends no it :) Yeah agreed. I think it'd be worth checking with Ask why he made it this way.
Re: [PATCH] Log even when we aren't in a transaction
On Thu, 20 Aug 2009 10:18:39 -0500, Jared Johnson wrote: It looks like logging/file doesn't like the empty hashref returned by Qpsmtpd::transaction(). I never understood why it did that. Any reason it can't return either undef or (preferably) a new Transaction object?
Re: qpsmtpd-async authenticated relaying direction request
On Sat, 30 May 2009, Baltasar Cevc wrote: On May 28, 2009, at 4:37 PM, Matt Sergeant wrote: Seems a little fragile. There aren't many bounces that quote all headers. You'd be better off just rejecting all bounces in qpsmtpd, then you only see legit bounces where the remote end issued an immediate 5xx to your Exim's outgoing mail. Of course when I suggest that (here and on other lists) mail admins tend to freak out a bit. But I'm of the opinion that I've barely ever seen a useful full bounce in years. So my qpsmtpd runs a no_bounces plugin, which I believe I've posted here before. While I would not freak out, it is surely not RFC conformant. And, even worse, it is illegal in some legislations (I can tell for Germany): suppressing mails (thus also failure notices that made sure the sender knows a mail has not arrive) is punishable under the German Criminal Code. That's a major misunderstanding of both the RFCs and German law. However, I think everybody should descide himself/herself. Indeed. I notice 'broken' systems that will send bounces instead of rejecting mail from time to time... I'm not suggesting there aren't. But as far as I can tell, there's 3 options: 1) accept all bounces, and get flooded with invalid bounces when spammers spoof you. 2) implement BATV and the flaws it has. 3) reject all SMTP level bounces and the flaws it has. All have problems. You just pick your acceptable failures. Matt.
Re: qpsmtpd-async authenticated relaying direction request
On Wed, 27 May 2009 09:18:32 -0500, David Favor wrote: I'm currently running qpsmtpd-async. I host many domains and I'd like to protect them all against backscatter using something like this: http://psg.com/~brian/software/authbounce/configure-authbounce.txt to add a bounce key to each outgoing message of the form: X-bounce-key: $mx-$number;$sender;$timestamp;$key This requires all mail sent by every user to go through qpsmtpd + exim on my local machine. Seems a little fragile. There aren't many bounces that quote all headers. You'd be better off just rejecting all bounces in qpsmtpd, then you only see legit bounces where the remote end issued an immediate 5xx to your Exim's outgoing mail. Of course when I suggest that (here and on other lists) mail admins tend to freak out a bit. But I'm of the opinion that I've barely ever seen a useful full bounce in years. So my qpsmtpd runs a no_bounces plugin, which I believe I've posted here before. Matt.
Re: qpsmtpd-async authenticated relaying direction request
On Thu, 28 May 2009 12:04:27 -0400 (EDT), Charlie Brady wrote:
On Thu, 28 May 2009, Matt Sergeant wrote:
years. So my qpsmtpd runs a no_bounces plugin, which I believe I've
posted here before.
Google seems not to have heard of it.
Ah. OK. It basically just does this:
if ($transaction-sender-format eq
or
$transaction-sender-format =~ /MAILER-DAEMON/i
or
$transaction-sender-format =~ /postmaster\@/i
)
{
unless (grep { /(bounce-|-return-)/ } map { $_-address}
$transaction-recip
ients) {
return DENY, Mail from not accepted here;
}
}
Re: Feature request to disable CONTROL_PORT
On Wed, 27 May 2009, David Favor wrote: Having qpsmtpd listen on an additional control port creates serious complexity when running multiple copies of qpsmtpd, as each copy has to somehow figure out which control port to use, hope it's free and then connect. I usually just strip this code out of qpsmtpd or comment out the initial connection. A great feature to add is a simple command line option to turn this off. Yeah, this is only on -async anyway. I'm wondering if we should just dump the feature, since it doesn't work when having multiple async children. Matt.
Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81
On Thu, 21 May 2009, Devin Carraway wrote: On Wed, May 20, 2009 at 09:40:21PM -0400, Charlie Brady wrote: I think the -T *should* be there on the command line, but there are some bugs in qpsmtpd and/or your plugins which need to be fixed before it will work. forkserver has used -T since 29ac2860, back in 2004. Obviously prefork is newer and has seen less testing, but most of the module code and plugins have seen plenty of taint-checked use. I guess this raises a question: The return values from config() are tainted. Should we de-taint them? Matt.
Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81
On Wed, 20 May 2009, J wrote: I compared the run file with other run files (i.e. djbdns and qmail) and I think the problem is with the trailing ' \' on the 2nd line (the first exec). Indeed. That shouldn't be there. When I remove that (and installed a missing Math::BigInt package from CPAN), everything loads, but complains about an insecure dependency on line 416 in setpriority (in qpsmtpd-prefork). (And the prefork processes keep showing up in the process table, and dying.) I expect that this was tested before being released, so there is probably still something wrong with my setup :-( I think the -T shouldn't be there on the command line. Though it is probably a bug, I'm guessing we don't test with taint on. Matt.
Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81
On Wed, 20 May 2009, Charlie Brady wrote: Though it is probably a bug, I'm guessing we don't test with taint on. Perl taint mode is an underutilised gem. It is, but it's also buggy and annoying. (there's a completely ignored bug in perl with -T and hash keys which I filed months ago) Matt.
Re: [qpsmtpd] Still looking: tcpserver startup for qpsmtpd-prefork 0.81
On Wed, 20 May 2009, David Nicol wrote:
On Wed, May 20, 2009 at 10:28 PM, Matt Sergeant m...@sergeant.org wrote:
(there's a completely ignored bug in perl with -T and hash keys which I
filed months ago)
that hash keys are never tainted is documented, if that's your bug. It
allows for a quick and dirty
sub detaint($){ [ keys %{{ $_[0] = 1 }} ] - [0] }
Nope. #56842. Very annoying.
Matt.
Re: perltidy
On Fri, 3 Apr 2009 08:38:59 -0500, Jared Johnson wrote: first of all, kudos on the frequent releasing :) I've attached a suggested patch to .perltidyrc. I've been playing around with perltidy'ing all QP code and some results I don't like. This doesn't fix all the things that I don't like, but it's been sitting on my laptop for a month so I thought I'd submit it to the ML for discussion. I haven't committed it to my git since I'd just revert it if it wasn't accepted :) Here's the relevant documentation to go along with the additions: -ce, --cuddled-else You're the first perl programmer I know who likes these :-) I don't. And I'm fairly sure the rest of us probably don't either. [-nbcc], -bbc, --blanks-before-comments A blank line will be introduced before a full-line comment. This is the default. Use -nbbc or --noblanks-before-comments to prevent such blank lines from being introduced. No idea what this does? Do you have a visual example? -msc=n, --minimum-space-to-comment=n Side comments look best when lined up several spaces to the right of code. Perltidy will try to keep comments at least n spaces to the right. The default is n=4 spaces. Same as above... Matt.
Re: Who wants to play holy war over -ce ?
On Fri, 3 Apr 2009, David Nicol wrote: On Fri, Apr 3, 2009 at 12:36 PM, Matt Sergeant m...@sergeant.org wrote: -ce, --cuddled-else You're the first perl programmer I know who likes these :-) I don't. And I'm fairly sure the rest of us probably don't either. I use them; they appear in plenty of CPAN modules, production code, examples in mailing lists. Hehe, ok three of you then ;-) By the rest of us I meant the core qpsmtpd developers btw. But I was speaking out of line - I should let people speak for themselves, so I apologise for that. Matt.
Re: End of headers hook
Also note this won't work with -async properly. I'll have a look how it can be made to work asynchronously (you need to follow the respond style in the rest of the code). On Mon, 09 Feb 2009 23:43:05 -0800, Robert Spier wrote: Committed as http://github.com/rspier/qpsmtpd/commit/a0ae0453264fe8dd85c132f2e7305b5ac34bf7e8 Did you actually test that this worked? I had to tweak the code to make it make sense. Also, you had a mix of tabs and spaces and didn't follow the style of the rest of the code. I cleaned them up this time. -R Karl Y. Pradene wrote: On Mon, 09 Feb 2009 10:10:47 -0800 Robert Spier rsp...@pobox.com wrote: Functionally, just about anything you can do here you can do just as I don't see a problem with adding it per-se. It'd make my plugins/config file ordering a trifle simpler, but it doesn't really add any functionality. I agree there should be a big warning about rejects/disconnects potentially being very dangerous. Chris' analysis [mostly snipped] is compelling. Karl, Can you update the patch with - documentation - ... including a warning - changing the name? And then I'll happily apply it. As Ask said, don't worry about bad english, that's easy enough to fix later. -R And voila. Hook renamed to data_headers_end, documentation in README.plugins with a BIG warning. $self-transaction-header($header) was moved earlier to have headers in transaction object when we call the hook. Regards. -- Karl Y. Pradene
Re: qpsmtpd-async under AnyEvent
On Tue, 3 Feb 2009, Pedro Melo wrote: Hi, I have a AnyEvent-based project that requires a SMTP server. I was considering either write a AnyEvent::Impl::Danga::Socket so that I can run qpsmtpd directly. Has anybody played with something like this here? I don't think so... Though I have a backend that uses EventLib - does AnyEvent support that? Only thing is there's no async DNS libraries for the other event loops. ParaDNS seems to be the only one. The other possibility is to rewrite parts of qpsmtpd to use AnyEvent::Socket (and probably AnyEvent::Handle). Any suggestions? I'd consider writing your own backend based on the current Async one. It's not that hard, as long as you grok this stuff. Matt.
Re: Domain Alias in Qpsmtpd
On Wed, 28 Jan 2009 16:23:01 +0100, Julien wrote:
The config file (/var/qmail/control/aliasdomains) contain one
remplacment per line, each remplacment consist of 'olddomain' and
'newdomain' seperate by a ':' like :
test.domain.com:domain.com
So address jul...@test.domain.com sould be rewrite to jul...@domain.com
I'm sure there is a very elegant way to do this in perl ...
Yeah it's pretty simple:
#!perl -w
my %domain_map;
sub init {
my ($self) = @_;
for ($self-qp-config('aliasdomains')) {
my ($from, $to) = split(/:/, $_, 2);
$domain_map{$from} = $to;
}
}
sub hook_rcpt {
my ($self, $transaction) = (shift, shift);
my $rcpt_domain = $_[0]-host;
if (my $rewrite_to_domain = $domain_map{$rcpt_domain}) {
$self-log(LOGINFO, Rewriting $rcpt_domain to
$rewrite_to_domain);
$_[0]-host($rewrite_to_domain);
}
return DECLINED;
}
Re: Domain Alias in Qpsmtpd
On Wed, 28 Jan 2009 15:13:31 +0100, Julien wrote: Without any success, I still got/get the original recipient. What version of qpsmtpd? I think changing these values was only added in SVN. Matt.
Re: File Permissions bug?
On Sat, 24 Jan 2009 10:37:46 -0500 (EST), Charlie Brady wrote: Any time you spend doing useless system calls isn't available for spamassassin or clamav. Run strace against qpsmtpd (at least forkserver variant) some time and you'll see how often config files are opened. Well of course. Forkserver is unable to cache anything except for the lifetime of the fork. And I would be willing to be the fork is way more expensive than extra stats(). Matt.
File Permissions bug?
Should we fix qpsmtpd config loading to check for file permissions as described in: http://use.perl.org/~Alias/journal/38319 ? Matt.
Re: indentation, line length, trailing whitespace
On Wed, 14 Jan 2009 09:10:14 -0600, Jared Johnson wrote: Hi, Is there any interest in patches to conform current QP code to line length and whitespace standards? 1. I notice that most of the codebase uses 4-space indents, but I have seen some areas with 2-space indents and tab characters. The original codebase (Ask's code) was all 2 space indents. The newer qpsmtpd coders all prefer 4 spaces, so we sort of overruled Ask (sorry mate :)). There was a plan at some point to run everything through perltidy to set 4 spaces everywhere, but nobody ever got around to it. 2. It's been pointed out that it is preferred that lines are kept to 80 characters are less, but this is violated in a number of places. Meh, I'm not hardcore about this. Screens are bigger these days, or vim displays stuff nicely anyway. I think it's a good overall rule, but if you have a long string or something I don't have a problem violating it. Our organization conforms to similar coding guidelines WRT these issues and I would be happy to submit patches to cleanup up each of these areas if there's any desire. I would also suggest creating a 'coding guidelines' page on the wiki for this and any other style issues that are deemed useful to conform to. Think we should do this after the git migration is complete. Matt.
Re: [PATCH] Make remote_host available in hook_pre_connection
On Mon, 12 Jan 2009, Jared Johnson wrote:
I propose something like the following in the connection package
sub remote_hostname{
my $self = shift;
$self-{_remote_hostname} ||=
Net::DNS::get_name($self-{_remote_ip})
}
This seems reasonable to me on principal.
Except doesn't work with -async. Well, it works, but blocks.
Was the name 'remote_hostname' as
opposed to 'remote_host' intentional?
Probably to match apache/mod_perl - most of us qpsmtpd hackers are ex
mod_perl guys.
Matt.
Re: Christmas release?
On Mon, 22 Dec 2008 11:58:23 +0100, Kaare Rasmussen wrote: On Dec 19, 2008, at 18:56, Matt Sergeant m...@sergeant.org wrote: Shall we do a release for xmas? It's been forever... On the topic of releasing, why isn't qpsmtpd released to CPAN? Because we never really got it easily installable via the whole Makefile.PL route.
Re: general question about ClamAV plugin
On Tue, 30 Sep 2008 17:03:36 -0700 (PDT), [EMAIL PROTECTED] wrote: When using the ClamAV plugin i realize that this is not the entire ClamAV application but how do I then keep the virus definitions for the plugin up to date? do I need to actually install the ClamAV application to use freshclam to update the virus definations and then Yes. do i need to have the ClamAV plugin point to a different spot for it to see these newer definations? No, that's automatic.
Re: Latest SMTP.pm causes fatal errors
On Tue, 30 Sep 2008 08:34:05 +0200, Hanno Hecker wrote: I'm confused why the error is caused - anyone better with perl than me can help out? Same confusion here... and I cannot reproduce it. $ perl -v | grep 'This is perl' This is perl, v5.8.8 built for i486-linux-gnu-thread-multi Running -async works without a problem, with and without (start)tls. Same here. Chris can you reduce it to a couple of plugins and let us know the setup required to reproduce?
Re: DNSBL and answer id check missing
On Sat, 27 Sep 2008 13:56:58 +0200, Diego d'Ambra wrote: To me it seems that plugin DNSBL is using Net::DNS bgsend/bgread, but is not checking the id of the reply received. If true this means that an attacker can white- or blacklist any email by sending fake dns replies (only randomisation is source port). Furthermore any other application on same machine also doing dns lookup may end up using same source port and have it's replies being mixed with those plugin DNSBL is waiting for. Spamassassin is also using Net::DNS bgsend/bgread, but does verify if the dns answer id matched the request. Maybe Net::DNS requires the caller to do the validation, or did I miss something? I'm working on a way to test this, but would love to hear others opinion, before doing to much work for maybe nothing :-) I'm pretty sure you're probably right. The async version uses ParaDNS which does do the id checking. (we should probably do 0x20 checking too, but I think that's beyond my skill levels :-/ )
Re: DNSBL and answer id check missing
On Sat, 27 Sep 2008 20:09:37 -0400, Chris Lewis wrote: I've extended the async dnsbl plugin to do scoring. It occured to me a few days ago that DNSBLs with negative scores (DNSWLs) should be treated as a hit if they get a timeout or other failure. This has prompted me to comment about checking ids too. The stock one doesn't do scoring, and hence can't do DNSWL. You want my code? You might not like my logging conventions however ;-) Like I said, the async version doesn't suffer from this bug, because it uses ParaDNS which checks the id fields (but doesn't do 0x20 checking - I wish I knew enough to be able to extend it to do so).
RE: FIXED -- high CPU on lost processes with forkserver
I've applied this, but without using a global for the client as that
seems unnecessary. Please check this version works. (I've also made the
relevant change to -forkserver too).
On Fri, 26 Sep 2008 11:59:57 -0500, Ed McLain wrote:
Just an fyi for the group. I've been running this patch for several
days now and my stuck processes are completely gone and no sign of
any problems arising from the changes. Has anyone else had a chance
to look over the changes I made yet? Also, the load on my boxes has
dropped dramatically. Each of my boxes generally had 4-6 stuck
processes eating up cpu time talking down empty sockets (tls on some,
remote disconnected on others) which would give me load averages on
each box between 3 and 19. Load averages now on each box are between
0.04 and 0.19 with each box processing roughly 6-12 connections per
second.
--
Thanks,
Ed McLain
-Original Message-
From: Ed McLain
Sent: Tuesday, September 23, 2008 2:31 PM
To: qpsmtpd@perl.org
Subject: RE: high CPU on lost processes with forkserver
Does anyone have any problems with the patch to fix this bug?
Basically, when TcpServer's run method is called it is passed a copy
of the $client IO::Socket and $client-connected is called in the
respond method (for TcpServer and PreFork) to verify that the socket
is still open. I've been testing it for a while and haven't seen any
issues, also don't have any stuck processes anymore either. I'm not
a perl monger though so I just want to make sure I'm not doing
anything insane. Any and all input is greatly welcome.
--
Thanks,
Ed McLain
-Original Message-
From: Ed McLain
Sent: Monday, September 22, 2008 10:51 AM
To: Jose Luis Martinez; qpsmtpd@perl.org
Subject: RE: high CPU on lost processes with forkserver
Anything new on a fix for this bug? I seem to have quite a few
connections hitting this these days.
--
Thanks,
Ed McLain
-Original Message-
From: Jose Luis Martinez [mailto:[EMAIL PROTECTED]
Sent: Tuesday, April 29, 2008 11:42 AM
To: qpsmtpd@perl.org
Subject: Re: high CPU on lost processes with forkserver
Peter J. Holzer escribió:
On 2008-04-25 21:24:17 +0200, Jose Luis Martinez wrote:
Peter J. Holzer escribió:
You caught it!!! It did the trick!
As I wrote previously, my guess is that both the mysql library and the
tls library catch SIGPIPE but don't call the previously installed
signal handler. So only one of them gets called (whichever is
registered last) and the other one loses.
So before patching the qp core in the respond method (Matt Sergeant
commented: But I removed it because then alarm() features VERY
heavily in the performance profiling as an expensive system call.).
I chose to work around the DBD::mysql to make it behave...
my $sighandle = $SIG{'PIPE'};
my $dbh =
DBI-connect('DBI:mysql:database=xxx;host=localhost;port=3306',
'xxx','xxx') or
$self-log(LOGDEBUG, 'Could not connect ' . DBI-errstr()) ;
$SIG{'PIPE'} = $sighandle;
It seems the DBD::mysql uses the SIGPIPE to reconnect to the mysql in
case the connection is lost. Good bye feature!
Looks like Apache DBD::mysql have or have had the same problem from
this post...
Found this:
http://mail-archives.apache.org/mod_mbox/httpd-dev/199903.mbox/[EMAIL PROTECTED]
No, but there are at least two layers below that: The PerlIO layer and
the TLS layer. Either one could retry an unsuccessful write if the
actual cause of the error was lost.
I'll try to contact the author of the TLS layer so that instead of
depending on the signal, maybe he can depend on the return value of
the writes (EPIPE) to cancel out... (Seems like a more stable
solution...
that way external modules cannot influence you).
Thanks for all the help and comments.
Jose Luis Martinez
[EMAIL PROTECTED]
CAPSiDE
EnemiesList plugin
People who follow the SVN commits list will have seen that I posted an enemies_list plugin in contrib. EnemiesList is a database of regular expressions that map to dynamic-looking hosts. It's non-free, which is why I've posted it in the contrib directory. In the EL contrib directory (on their server) there's a perl module for loading the regular expressions and matching very quickly against a hostname in there (much faster than if you loaded all the regexps into perl itself). If you have all that working (the DNS::RE module) then the plugin queries the EL database for the HELO host given. This is generally a very strong spamware sign, though I've left it so that the plugin just stores the results in $transaction-notes so you can do with it as you wish. http://enemieslist.com/about/
Re: $transaction-body_filename;
On Mon, 15 Sep 2008 16:40:24 -0400, Chris Lewis wrote: According to the documentation, when you call $transaction-body_filename, you get a temporary file name that points at a file that contains the message. If you examine body_filename, it has no headers. The clamdscan plugin uses body_filename to hand off to clamdscan. Which means that ClamAV doesn't get to see the headers. Which is important to some ClamAV detections (eg: the ClamAV self-test email is _not_ caught by the clamdscan plugin). [In contrast, the spamassassin plugin carefully spits out header-as_string and then the body into spamd.] Is this working as intended? My gut instinct is to say no. But I worry a little bit that we might break something else by fixing it...
Re: $transaction-body_filename;
On Mon, 15 Sep 2008 17:39:33 -0400, Chris Lewis wrote: Matt Sergeant wrote: On Mon, 15 Sep 2008 16:40:24 -0400, Chris Lewis wrote: According to the documentation, when you call $transaction-body_filename, you get a temporary file name that points at a file that contains the message. If you examine body_filename, it has no headers. The clamdscan plugin uses body_filename to hand off to clamdscan. Which means that ClamAV doesn't get to see the headers. Which is important to some ClamAV detections (eg: the ClamAV self-test email is _not_ caught by the clamdscan plugin). [In contrast, the spamassassin plugin carefully spits out header-as_string and then the body into spamd.] Is this working as intended? My gut instinct is to say no. But I worry a little bit that we might break something else by fixing it... Would it be worth considering have a data_filename() call, that does exactly the same thing as body_filename, but includes the headers too? Then we can fix the clamdscan plugin without breaking anything else. I thought about that, but you'd have to write to a file twice then :-/
Re: $transaction-body_filename;
On Mon, 15 Sep 2008 19:30:11 -0400, Matt Sergeant wrote: Would it be worth considering have a data_filename() call, that does exactly the same thing as body_filename, but includes the headers too? Then we can fix the clamdscan plugin without breaking anything else. I thought about that, but you'd have to write to a file twice then :-/ So I think we should just fix the bug, and see if anything breaks.
Re: $transaction-body_filename;
On Mon, 15 Sep 2008 17:35:31 -0700, Ask Bjørn Hansen wrote: On Sep 15, 2008, at 13:40, Chris Lewis wrote: According to the documentation, when you call $transaction-body_filename, you get a temporary file name that points at a file that contains the message. If you examine body_filename, it has no headers. It was made that way first because the headers can change, so we needed to keep those in memory. If we put them into the file we have to document that they are there as received from the client which isn't necessarily the same as what they are now. Also any plugins (queue plugins) that use the body but need the correct headers will have to know to skip the headers (maybe a method to give a filehandle starting at the right place?). We already have that, with body_start().
Re: qpsmtpd-prefork - anyone using it and are patches accepted?
On Mon, 4 Aug 2008 15:27:46 +0200, Diego d'Ambra wrote: The latest version of qpsmtpd-prefork (revision 935) seems to miss some important patches, e.g.: - shared memory LOCK (qpsmtpd freezes) - clean shutdown of parent and any children (unable to restart qpsmtpd) - cleanup of STDIN (data from previous SMTP session still around with next session) Some of these may have been solved in other places, but I would be willing to test and check if they are still needed and patch against latest revision. Any interest? Yes of course.
Re: Correcting plugin syntax errors
On Tue, 29 Jul 2008, Jose Luis Martinez wrote: BTW: any comment on how to elaborate a testing framework. Comments from the QP-gurus would be helpful. For some in house code here that's similar to Qpsmtpd we basically have a data driven test system. You specify in a config file what plugins to load, and any other config file contents (supports CDB files too), and it generates the right config, and starts the daemon. Then you specify things like HELO, IP, full email contents, etc and it talks to the daemon using those details, and checks for the expected outcome. Obviously not all that is possible with qpsmtpd (e.g. you can't spoof the IP easily), but most of it is. And it makes developing tests much easier. I can probably share a bit of the code if it helps. Matt.
Re: [qpsmtpd] Decisions, Decisions
On Tue, 8 Jul 2008, David Nicol wrote: On Tue, Jul 8, 2008 at 3:14 PM, [EMAIL PROTECTED] wrote: Basically you just need an understanding of how async programming works - from there everything starts to become obvious. never block for IO. Work out the state so that you can drop everything and pick it up later whenever you would be waiting for IO from any source. Keep all per-connection data in an object. Yup, that and a basic understanding of how Danga::Socket works (easiest to do something like read the current queue/async/smtp-forward plugin to learn that). But do note what I've said here previously: async is for high CONCURRENCY not necessarily performance. Up to a certain level of concurrency prefork is faster. Matt.
Re: [qpsmtpd] Decisions, Decisions
On Tue, 8 Jul 2008, David Nicol wrote: On Tue, Jul 8, 2008 at 4:33 PM, Matt Sergeant [EMAIL PROTECTED] wrote: But do note what I've said here previously: async is for high CONCURRENCY not necessarily performance. Up to a certain level of concurrency prefork is faster. with SMTP one cares about throughput more than response time, and properly implemented async will always provide better throughput than an equivalent forking system on the same hardware simply because the OS doesn't have to bother with as many context switches. (lots of wiggle room in properly.) It's true, but there seems to be an annoying inefficiency in -async somewhere compared to the simplicity of prefork - I'm fairly sure it's to do with reading lines from the buffer - perl just isn't very good at that sort of thing (whereas with C you can just move a pointer along, Perl doesn't do that very well). I suppose I could probably do a bit more benchmarking to improve it at some point. Matt.
Re: [qpsmtpd] Decisions, Decisions
On Mon, 7 Jul 2008 22:30:50 + (UTC), [EMAIL PROTECTED] wrote: Looking at the wiki deployment page, it looks like I could be considering the apache or prefork versions. I'm leaning towards prefork - the page says it's faster, but there's no comparison between it and the apache version. Are there any substantial, meaningful differences between forkserver, apache, and prefork? I should be able to run either of the three, so how do I determine which would be best? Apache's downside is simply that it requires Apache. In most other aspects it's pretty good, including the fact that for free you get a working server-status page listing current connections. I think the performance of the Apache module is very slightly higher than prefork simply because all the I/O happens in C. (I definitely want stable, so pollserver looks like it's out of the question. BTW, are there any guidelines on how to write async modules to ensure they work with pollserver?) There are lots of bits and pieces that I've posted to this mailing list. That's about all though :-/ Basically you just need an understanding of how async programming works - from there everything starts to become obvious. Matt.
Re: questions re: async, perperl, new release
On Fri, 27 Jun 2008 04:17:55 -0700, Ask Bjørn Hansen wrote: Has anyone run qpsmtpd under PersistentPerl (a.k.a SpeedyCGI). There is a mention of pperl, but I've never used it. qpsmtpd-forkserver is our closest equivalent. I don't know if anyone's been running it under pperl or PersistentPerl for a while. Actually qpsmtpd-prefork is closer. I noticed a mention of a new release on 4 June. Any news on that? Is the CVS version stable enough for production? The subversion version is quite stable; indeed it's almost certainly better than the last release. There's a message there ... :-)
Interesting article on async-ness
http://oubiwann.blogspot.com/2008/06/so-you-want-your-code-to-be-asynchonous.html It's about Python, but the same stuff generally applies to working with qpsmtpd-async.
Re: require_resolvable_fromhost bug
Heh, I was going through this code a week or so ago and thought this
looked odd...
I'll apply the patch.
On Thu, 12 Jun 2008 14:53:22 +0200, Jose Luis Martinez wrote:
Hi,
We're using the require_resolvable_fromhost plugin, and have seen
that there is a bug in it. MAIL FROM commands with legitimate
domains where getting rejected as not resolvable.
The bug has been tracked down to domains that have zone files with
unresolvable MX records.
example.com. 3600 IN MX 10 fakeserver.example.com.
example.com. 3600 IN MX 20 mail.example.com.
example.com. 3600 IN MX 30 anotherfakeserver.example.com.
The plugin tries to resolve every MX record in the list (to see if it
has an A record), but gives up on the first unresolvable one,
considering the whole domain unresolvable.
I consider this a bug, as leaving fake MX records is a known antispam
technique, and after all, the domain is resolvable if you have at
least one resolvable MX record.
Patch attached:
Index: plugins/require_resolvable_fromhost
===
--- plugins/require_resolvable_fromhost (revision 1)
+++ plugins/require_resolvable_fromhost (working copy)
@@ -53,8 +53,14 @@
$res-udp_timeout(30);
my @mx = mx($res, $host);
foreach my $mx (@mx) {
-return mx_valid($self, $mx-exchange, $host);
+# if any MX is valid, then we consider the domain
+# resolvable
+return 1 if mx_valid($self, $mx-exchange, $host);
}
+ # if there are MX records, and we got here,
+ # then none of them are valid
+ return 0 if (@mx 0);
+
my $query = $res-search($host);
if ($query) {
foreach my $rrA ($query-answer) {
Jose Luis Martinez
[EMAIL PROTECTED]
CAPSiDE
Re: rcptto_vrfy
On Tue, 10 Jun 2008 13:59:43 +0200, Jan Völkers wrote:
Hello,
has anyone written a rcptto_vrfy plugin?
I set up a new mx in a hurry and now i am searching for a nice
check_rcpt plugin. Actually i am using only the smtp_forward plugin
which has two disadvantages:
- it checks _after_ queueing the whole mail.
- it rejects legitimate mails: If one sends mails to different users
in the same domain with one correct and one incorrect address also
the legitimate receiver will not get any mail as it is completely
rejected.
I would appreciate a simple plugin that checks recipients via vrfy. Yes,
my next hop answers vrfy correctly.
Does anybody know something like this or has other suggestions? My Users
are homed as virtual users and aliases in a mysql db, but i prefer
the vrfy way as it is more portable.
I don't think anyone has written one yet. We've talked a LOT about
making smtp_forward be more of a direct proxy, or at least do that
optionally, but different users seem to have different requirements for
that.
It should be really easy to write. Just use Net::SMTP to make the
connection and issue the VRFY.
Something like this:
use Net::SMTP;
sub hook_mail {
my ($self, $tran) = @_;
$tran-notes('my_smtp_server', Net::SMTP-new('wherever'));
return DECLINED;
}
sub hook_rcpt {
my ($self, $tran, $recipient) = @_;
my $smtp = $tran-notes('my_smtp_server');
if (!$smtp-verify($recipient-format)) {
return DENY, Bad recipient: $recipient;
}
return DECLINED;
}
sub hook_data {
my ($self, $tran) = @_;
my $smtp = $tran-notes('my_smtp_server');
$smtp-quit;
$tran-notes('my_smtp_server', undef);
return DECLINED;
}
Matt.
Re: [svn:qpsmtpd] r923 - in trunk: . lib/Qpsmtpd
On 3-Jun-08, at 4:04 AM, Radu Greab wrote: I think we need more consensus on this patch. Due to the objections expressed so far I reverted the change. Should I mention the overloaded meaning of remote_host in the pod? No, because it's different with different backends. Currently only async will set it to SERVFAIL or TIMEOUT in those conditions. And unfortunately there's no way for us to fix that in tcpserver mode. Matt.
Re: Clarification of hooks with ignored return values
On 3-Jun-08, at 4:36 AM, Radu Greab wrote: The documentation says that for the following hooks the return values are ignored or discarded: post-fork post-connection disconnect deny ok I had however to change my custom plugins to return DECLINE from post-fork hooks, otherwise, with no explicit return statement, only the hook of the first plugin was called, the hooks of the other plugins were not called. I think that it is a bug that the code does not ignore the return values of the hooks mentioned above and stops calling all the plugins which registered interest in the hooks. Am I right and should the code be changed to really ignore the return values from those hooks? No, the docs need changed :-) While the return values are ignored, you must still abide by the qpsmtpd processing model (e.g. return DECLINED to allow the next hook to fire, return DONE to stop there, etc). What it really means is you can't create an SMTP return code in one of those hooks, e.g. by returning DENY and hoping that a 5xx will be sent to the client. Matt.
Re: [svn:qpsmtpd] r923 - in trunk: . lib/Qpsmtpd
On 2-Jun-08, at 6:06 PM, Radu Greab wrote: Matt Sergeant wrote: I think that remote_host should not be the error code in case of errors. That contradicts the description of the method and may force other people using remote_host to do their own checks. Well on regular qmail it's set to Unknown in case of errors, but never tells you if the error was NXDOMAIN or SERVFAIL or TIMEOUT. I find the distinction useful. I find that Unknown as a value for remote_host looks more reasonable than NXDOMAIN and the other values. Why? That's less information. Unknown for NXDOMAIN always made sense. But Unknown in the case of a DNS timeout or a failure never made sense to me. What do you think about setting only remote_info to either the result of a successful lookup or the error code? And remote_host will be either the result of the successful lookup or [$remote_ip] in case of errors? Possibly... What will that result in in terms of Received headers? The Received headers won't change. Only the logs created by logterse or other logging plugins that use remote_host will change. Wow, that seems MUCH worse to me - to have the logs *not* storing the dns failure reason seems completely at odds to me with what logs should be for. I think we need more consensus on this patch. Matt.
Re: Pulling my hair out...
On Wed, 28 May 2008, Chris Lewis wrote:
I'm trying to implement a subroutine shared between plugins.
The library routine looks something like:
sub createpattern {}
It needs to be exported from your library, or called with a full package
name, how are you doing that?
The most common way is to use Exporter - see the Exporter man page for
details.
And is stored under ../qpsmtpd/lib/NTMqplib.pm
qpsmtpd-async is called from the qpsmtpd directory.
Each plugin then calls it thusly:
use NTMqplib;
sub register ... {
... = createpattern(...)
Note that the in front is a nasty perl4-ism and tends to be frowned upon
by other perl coders these days - fine if you're the only one maintaining
your script, but will look ugly if someone ever takes over...
What happens is that the first plugin executed works fine, and the next one
errors out saying it can't find createpattern().
Looks like the first plugin executed loads the library, and remembers it's
loaded. The second one thinks it's loaded, but the symbol table (or
whatever) isn't in scope, or some such.
How do you do this properly?
Sounds like you're probably just trying to put it into the current
namespace. Keep it in it's own namespace (i.e. have package NTMqplib; in
your .pm file) and use Exporter, or call it as NTMqplib::createpattern()
Matt.
Re: Connection notes and TLS
On 26-May-08, at 5:24 PM, John Peacock wrote: Those are *connection* attributes, not *transaction* attributes. If a plugin is storing them in the transaction object, that is a mistake. We can and probably should maintain the connection notes from before the TLS reset, but I maintain that we should not keep the transaction notes. OK, I'm fine with that.
Re: Connection notes and TLS
On 22-May-08, at 10:45 PM, John Peacock wrote: STARTTLS is not required to happen immediately after EHLO (not HELO, which doesn't support ESMTP extensions). And yes, you must completely discard every portion of the SMTP state that has occurred up to that point (just like with RSET). The RFC is extremely plain on this point: after STARTTLS has been sent and negotiated, the MTA must behave as if a completely new transaction has started (as indeed, it has). The transaction *must* be reset and all information contained therein must be thrown away. In practice, there isn't anything there to begin with, since all of the well-formed MTA's always sent STARTTLS as soon as practical (i.e. as soon as they see the initial EHLO banner), if they are going to send it at all. I don't think we should care so much about the RFCs. If there are bits in connection notes that might help determining if this is spam (or some other thing we're trying to detect) before STARTTLS we need to allow qpsmtpd to keep that information. Matt.
Re: Creating global DB connection
On 20-May-08, at 12:33 PM, Nighthawk wrote: Hi, I am trying to write plugin that does DB lookup and want to use the same db connection in more than 1 plugins. Somwhere I read that we can save DB connection in qp-config(). But it is not working in my case. Store it in $qp-connection-notes() in the same way you've done - config.
Re: Whitelist convention
Do it.
Re: Next steps to release?
On Mon, 12 May 2008, Hanno Hecker wrote: On Wed, 7 May 2008 19:58:46 + (UTC) Matt Sergeant [EMAIL PROTECTED] wrote: So the next release I consider to be a fairly major step - we've got async/smtp-forward and async tls working. That's most of the showstoppers against using async in production. So what's missing and what would you like to see before the next release? Remove qpsmtpd-server and it's module Qpsmtpd::SelectServer? Does anyone use it? Is it supported when we also have -async? Good thought.
Re: [svn:qpsmtpd] r903 - trunk/lib/Qpsmtpd
Note that this is probably worth doing for the other backends, though it's
only for performance really.
On Mon, 12 May 2008, [EMAIL PROTECTED] wrote:
Author: msergeant
Date: Mon May 12 10:19:31 2008
New Revision: 903
Modified:
trunk/lib/Qpsmtpd/PollServer.pm
Log:
Make sure non-responding hooks are called appropriately
Modified: trunk/lib/Qpsmtpd/PollServer.pm
==
--- trunk/lib/Qpsmtpd/PollServer.pm (original)
+++ trunk/lib/Qpsmtpd/PollServer.pm Mon May 12 10:19:31 2008
@@ -56,7 +56,7 @@
$self-load_plugins;
$self-load_logging;
-my ($rc, @msg) = $self-run_hooks(pre-connection);
+my ($rc, @msg) = $self-run_hooks_no_respond(pre-connection);
if ($rc == DENYSOFT || $rc == DENYSOFT_DISCONNECT) {
@msg = (Sorry, try again later)
unless @msg;
@@ -164,7 +164,7 @@
sub close {
my Qpsmtpd::PollServer $self = shift;
-$self-run_hooks(post-connection);
+$self-run_hooks_no_respond(post-connection);
$self-connection-reset;
$self-SUPER::close;
}
Next steps to release?
So the next release I consider to be a fairly major step - we've got async/smtp-forward and async tls working. That's most of the showstoppers against using async in production. So what's missing and what would you like to see before the next release? (obviously not limited to -async stuff) Matt.
Re: async and tls take two
On 5-May-08, at 10:45 AM, Steve Kemp wrote: Seeing changes like this makes me wonder if we should consider running all source through perltidy at some point. Perhaps as part of a make update, or make commit target. Yes, it has been discussed before. We just need the tuits to make it so. ObRelated: I would be happy to submit a patch to unify plugin argument names. I see big differences between: ./queue/exim-bsmtp:my ( $self, $txn ) = @_; ./logging/file:my ( $self, $txn ) = @_; And: ./virus/clamav:my ( $self, $transaction ) = @_; ./valid_users:my ( $self, $transaction, $sender, %param ) = @_; For example. I'd suggest s/txn/transaction/g on all plugins shipped with the core distribution, because I think the verbosity and consistency wouldn't be a bad thing. That would be a nice easy project for someone with a bit of time on their hands... (hint hint) Matt.
Re: async and tls take two
On 4-May-08, at 7:28 PM, Matt Sergeant wrote:
Yeah this makes a lot more sense... Though I wonder if we shouldn't
modify Danga::Client to just have some sort of reader entry, so
that anything else (not just TLS) can hook into the event_read
stream. I'll have a poke.
Can you check if this works:
Index: plugins/tls
===
--- plugins/tls (revision 876)
+++ plugins/tls (working copy)
@@ -150,7 +150,7 @@
return DECLINED unless $local_port == 465; # SMTPS
unless ( _convert_to_ssl($self) ) {
- return (DENY_DISCONNECT, Cannot establish SSL session);
+return (DENY_DISCONNECT, Cannot establish SSL session);
}
$self-log(LOGWARN, Connected via SMTPS);
return DECLINED;
@@ -159,6 +159,10 @@
sub _convert_to_ssl {
my ($self) = @_;
+if ($self-qp-isa('Qpsmtpd::PollServer')) {
+return _convert_to_ssl_async($self);
+}
+
eval {
my $tlssocket = IO::Socket::SSL-new_from_fd(
fileno(STDIN), '+',
@@ -178,13 +182,21 @@
$self-connection-notes('tls_enabled', 1);
};
if ($@) {
- return 0;
+return 0;
}
else {
- return 1;
+return 1;
}
}
+sub _convert_to_ssl_async {
+my ($self) = @_;
+my $upgrader = $self-connection
+-notes( 'tls_upgrader', UpgradeClientSSL-
new($self) );
+$upgrader-upgrade_socket();
+return 1;
+}
+
sub can_do_tls {
my ($self) = @_;
$self-tls_cert -r $self-tls_cert;
@@ -238,3 +250,81 @@
$self-log(LOGWARN, Exiting because 'tls_enabled' was true.);
exit;
}
+
+package UpgradeClientSSL;
+
+# borrowed heavily from Perlbal::SocketSSL
+
+use strict;
+use warnings;
+no warnings qw(deprecated);
+
+use Danga::Socket 1.44;
+use IO::Socket::SSL 0.98;
+use Errno qw( EAGAIN );
+
+use fields qw( _stashed_qp _stashed_plugin _ssl_started );
+
+sub new {
+my UpgradeClientSSL $self = shift;
+$self = fields::new($self) unless ref $self;
+$self-{_stashed_plugin} = shift;
+$self-{_stashed_qp} = $self-{_stashed_plugin}-qp;
+return $self;
+}
+
+sub upgrade_socket {
+my UpgradeClientSSL $self = shift;
+
+unless ( $self-{_ssl_started} ) {
+IO::Socket::SSL-start_SSL(
+$self-{_stashed_qp}-{sock}, {
+SSL_use_cert = 1,
+SSL_cert_file = $self-{_stashed_plugin}-tls_cert,
+SSL_key_file = $self-{_stashed_plugin}-tls_key,
+SSL_ca_file = $self-{_stashed_plugin}-tls_ca,
+SSL_cipher_list = $self-{_stashed_plugin}-
tls_ciphers,
+SSL_startHandshake = 0,
+SSL_server = 1,
+SSL_reuse_ctx = $self-{_stashed_plugin}-ssl_context,
+}
+) or die Could not upgrade socket to SSL: $!;
+$self-{_ssl_started} = 1;
+}
+
+$self-event_read($self-{_stashed_qp});
+}
+
+sub event_read {
+my UpgradeClientSSL $self = shift;
+my $qp = shift;
+
+$qp-watch_read( 0 );
+
+my $sock = $qp-{sock}-accept_SSL;
+
+if (defined $sock) {
+$qp-connection( $qp-connection-clone );
+$qp-reset_transaction;
+$qp-connection-notes('tls_socket', $sock);
+$qp-connection-notes('tls_enabled', 1);
+$qp-watch_read(1);
+return 1;
+}
+
+# nope, let's see if we can continue the process
+if ($! == EAGAIN) {
+$qp-set_reader_object($self);
+if ($SSL_ERROR == SSL_WANT_READ) {
+$qp-watch_read(1);
+} elsif ($SSL_ERROR == SSL_WANT_WRITE) {
+$qp-watch_write(1);
+} else {
+$qp-disconnect();
+}
+} else {
+$qp-disconnect();
+}
+}
+
+1;
Index: lib/Danga/Client.pm
===
--- lib/Danga/Client.pm (revision 881)
+++ lib/Danga/Client.pm (working copy)
@@ -2,7 +2,15 @@
package Danga::Client;
use base 'Danga::TimeoutSocket';
-use fields qw(line pause_count read_bytes data_bytes callback
get_chunks);
+use fields qw(
+line
+pause_count
+read_bytes
+data_bytes
+callback
+get_chunks
+reader_object
+);
use Time::HiRes ();
use bytes;
@@ -26,6 +34,7 @@
$self-{pause_count} = 0;
$self-{read_bytes} = 0;
$self-{callback} = undef;
+$self-{reader_object} = undef;
$self-{data_bytes} = '';
$self-{get_chunks} = 0;
return $self;
@@ -96,9 +105,18 @@
}
}
+sub set_reader_object {
+my Danga::Client $self = shift;
+$self-{reader_object} = shift;
+}
+
sub event_read {
my Danga::Client $self = shift;
-if ($self-{callback}) {
+if (my $obj = $self-{reader_object}) {
+$self-{reader_object} = undef;
+$obj-event_read($self);
+}
+elsif ($self-{callback}) {
$self-{alive_time} = time;
if ($self-{get_chunks}) {
my $bref = $self-read($self-{read_bytes});
Re: high CPU on lost processes with forkserver
On 25-Apr-08, at 1:15 AM, Peter J. Holzer wrote: A better option might be to have the parent process watch for long running children and terminate them. Yup, but how long is long? If the client is trying to send a 600 MB email that will take some time ... Well we could set a max session time. Or even communicate aliveness via a pipe or something.
Re: high CPU on lost processes with forkserver
On 24-Apr-08, at 5:52 PM, Brian Szymanski wrote:
Ask Bjørn Hansen wrote:
On Apr 24, 2008, at 11:02 AM, Charlie Brady wrote:
Ask said Yeah, this is a pretty bad bug in March 2007, but I
haven't seen anyone looking to fix it.
We must be in pretty good shape when billions (or whatever) of email
transactions are processed every day and nobody is bothered enough by
possibly our worst known bug to come up with a patch. :-)
- ask
We have a workaround to kill off any process that's been alive more
than
5 minutes or so.
I'm anxious to get rid of it though, fixing things the right way,
since
our mail server is struggling to keep up (only partially a result of
this). Any advice on where to start to tackle this one? And, just
to be
clear we're talking about the same bug, this exists in .3x as well,
yea?
I think the core used to do something like this:
Index: lib/Qpsmtpd.pm
===
--- lib/Qpsmtpd.pm (revision 876)
+++ lib/Qpsmtpd.pm (working copy)
@@ -390,7 +390,10 @@
if ($hooks-{$hook}) {
my @r;
for my $code (@{$hooks-{$hook}}) {
+$SIG{ALRM} = sub { die Alarm };
+my $prev = alarm(10); # should be long enough for anyone!
eval { (@r) = $code-{code}-($self, $self-
transaction, @_); };
+alarm($prev);
$@ and warn(FATAL PLUGIN ERROR: , $@) and next;
if ($r[0] == YIELD) {
die YIELD not valid from $hook hook;
@@ -419,7 +422,10 @@
#warn(Got sampler called: ${hook}_$code-{name}\n);
$self-varlog(LOGDEBUG, $hook, $code-{name});
my $tran = $self-transaction;
+$SIG{ALRM} = sub { die Alarm };
+my $prev = alarm(10); # should be long enough for anyone!
eval { (@r) = $code-{code}-($self, $tran, @$args); };
+alarm($prev);
$@ and $self-log(LOGCRIT, FATAL PLUGIN ERROR: , $@) and next;
!defined $r[0]
But I removed it because then alarm() features VERY heavily in the
performance profiling as an expensive system call.
A better option might be to have the parent process watch for long
running children and terminate them.
Matt.
Re: qpsmtpd-async: premature end of data problem
On Sat, 12 Apr 2008, Radu Greab wrote: I found a case where qpsmtpd-async detects the end of data marker incorrectly: the previous packet did not end with CRLF and the current packet starts with .CRLF. The code assumes that the previous packet ended with CRLF. Attached are a test script and a suggested patch. So the test script is doing the wrong thing. Any . at the start of a line in SMTP needs to be encoded as .. and the SMTP line processor decodes it to a single dot. So the test is wrong, and Qpsmtpd is doing the right thing in assuming the end of data has been reached. At least as far as I can tell. Matt.
Re: qpsmtpd-async: premature end of data problem
On Mon, 14 Apr 2008, Matt Sergeant wrote: On Sat, 12 Apr 2008, Radu Greab wrote: I found a case where qpsmtpd-async detects the end of data marker incorrectly: the previous packet did not end with CRLF and the current packet starts with .CRLF. The code assumes that the previous packet ended with CRLF. Attached are a test script and a suggested patch. So the test script is doing the wrong thing. Any . at the start of a line in SMTP needs to be encoded as .. and the SMTP line processor decodes it to a single dot. So the test is wrong, and Qpsmtpd is doing the right thing in assuming the end of data has been reached. At least as far as I can tell. Ugh. Ignore me. Need more coffee :-)
Re: qpsmtpd-async: premature end of data problem
On Sat, 12 Apr 2008, Radu Greab wrote: I found a case where qpsmtpd-async detects the end of data marker incorrectly: the previous packet did not end with CRLF and the current packet starts with .CRLF. The code assumes that the previous packet ended with CRLF. Attached are a test script and a suggested patch. I committed the patch 95% the same as yours. I can't help feeling there's a better way to do it - I hate applying two regexps every time a DATA packet comes in... I'll have a think on it. Matt.
Re: qpsmtpd-async: premature end of data problem
On Mon, 14 Apr 2008, Guy Hulbert wrote: On Mon, 2008-14-04 at 12:54 +, Matt Sergeant wrote: Attached are a test script and a suggested patch. I committed the patch 95% the same as yours. I can't help feeling there's a better way to do it - I hate applying two regexps every time a DATA packet comes in... I'll have a think on it. tr ? I'm thinking more on the lines of pushing back reads if they're not full lines. Probably needs a bit of support for that in lib/Danga/Client.pm Matt.
Re: qpsmtpd-async: premature end of data problem
Ask Bjørn Hansen wrote: On Apr 12, 2008, at 6:33, Radu Greab wrote: I found a case where qpsmtpd-async detects the end of data marker incorrectly: the previous packet did not end with CRLF and the current packet starts with .CRLF. The code assumes that the previous packet ended with CRLF. Hi Radu, Can you add it to the issue tracker? http://code.google.com/p/smtpd/issues/list Wouldn't the non-pollserver servers have the same problem? No, because they do $fh and block until there's a full line. Though I could have sworn I fixed this - are you sure you're testing against latest SVN? Matt.
Re: disabling plugins
Hanno Hecker wrote:
Hi,
this patch sets a way to disable any loaded plugin(s) for the current
client. I'm not going to commit this before Ask released 0.43. Matt
(and others with high traffic) should be ok with the one more line
next unless $code-{run};
per hook.
The plugins_loaded() should be in to Qpsmtpd.pm. Any other comments on
this?
I'm currently looking for a way to have plugins loaded but disabled (so
some other plugin can enable it later). Loading plugins on connect or
later at runtime would probably be possible, but too slow...
(Note to self: write docs before any commit)
I'm not sure this is the right way to do this. Surely you want something
in the transaction object that can skip plugins at run time? Yours
seems like a rather heavy hammer, which can be influenced in a very
bizarre cross connection manner.
Matt.
