Re: [spamdyke-users] Duplicate ALLOWED from log entries
On 09/04/2013 04:18 AM, Sam Clippinger wrote: Found it! The bug is being triggered because multiple messages are being delivered in a single connection. spamdyke is caching the list of valid recipient addresses so it can print the log entries but isn't clearing them afterwards. So with each additional message, the list of addresses grows and duplicate messages are printed. I'll incorporate the fix in the next version, hopefully coming soon. -- Sam Clippinger Thanks a lot for your efforts! Waiting for the next version. best regards, -- David Davidov | System Administrator | ICDSoft Ltd. ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Duplicate ALLOWED from log entries
To be honest, I've spent almost no time on it. With what little time I've put towards spamdyke recently, I've been trying to finish testing the recipient validation code for the next version. I finally had to admit the tests were never going to finish on my laptop and I've rented a VPS for testing. The scripts are running much faster now -- I'm nearly done with the 166K tests to make sure my understanding of qmail's recipient processing is correct. After that, just 71K tests to check the recipient validation code in spamdyke. Then the last 1K scripts to test the rest of spamdyke. Simple! Anyway, I'll try to take a look at this issue this week if I can. :) -- Sam Clippinger On Sep 2, 2013, at 8:58 AM, David Davidov wrote: Hello, Any progress with this issue? regards, David Davidov On 06/18/2013 06:47 PM, David Davidov wrote: Hi Sam, I am a colleague of Theodor. This is a simple way to trigger the problem: dave~$ telnet mx.example.com 25 Connected to mx.example.com Escape character is '^]'. 220 mx.example.com ESMTP mail from: test-sen...@mx.example.com 250 ok rcpt to: test-r...@mx.example.com 250 ok data 354 go ahead . 250 ok 1370523483 qp 26283 mail from: test-sen...@mx.example.com 250 ok rcpt to: test-r...@mx.example.com 250 ok data 354 go ahead . 250 ok 1370523502 qp 26625 The first message produces one log entry: Jun 18 15:58:03 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523483_qp_26283 But for the second message we have two entries: Jun 18 15:58:22 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523502_qp_26625 Jun 18 15:58:22 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523502_qp_26625 We use spamdyke ver. 4.3.1 locally compiled on Debian 6.0.7. No header filtering. Here is our spamdyke.conf --- greeting-delay-secs=0 reject-empty-rdns log-level=info idle-timeout-secs=300 ip-whitelist-file=/home/vpopmail/etc/spamdyke_whitelist.txt recipient-whitelist-file=/var/qmail/spamdyke_recipient_whitelist.txt dns-whitelist-entry=antirbl.example.com graylist-level=always graylist-dir=/var/qmail/graylist graylist-min-secs=60 tls-certificate-file=/var/qmail/control/servercert.pem tls-privatekey-file=/var/qmail/control/servercert.pem local-domains-file=/var/qmail/control/rcpthosts local-domains-file=/var/qmail/control/morercpthosts dns-timeout-secs=3 rejection-text-empty-rdns=Refused. You have no reverse DNS entry. Contact ab...@example.com for details. rejection-text-ip-in-cc-rdns=Refused. Your reverse DNS entry contains your IP address and a country code. Contact ab...@example.com for details. rejection-text-unresolvable-rdns=Refused. Your reverse DNS entry does not resolve. Contact ab...@example.com for details. --- BRs, David Davidov On 01/-10/-28163 09:59 PM, Sam Clippinger wrote: I'm very sorry it's taken so long to get back to you on this; I've been buried at work and haven't had any time to investigate. It definitely sounds like you've hit a bug. spamdyke does save the addresses of all the recipients in order to print them all out in a loop, but only when the header blacklist feature is enabled. It does this because the recipient names have already gone by before the message header is sent, so it must save the recipient addresses to print either ALLOWED or DENIED once the header is finished. But it should only do this once and I'm not seeing a way to trigger that code more than once, though it certainly looks like that's what's happening. So let me start with all the standard questions: what OS and version are you on? What version of spamdyke are you using? Could you please post your configuration file(s) (or send them to me directly)? Would you mind turning on spamdyke's full logging feature (the full-log-dir option) and capturing one of these sessions? Needless to say this behavior isn't by design and it's not happening on any of the servers I manage (and I use the header-blacklist feature on every one). After spending a little while testing and tracking through the code I can't reproduce this problem, so I suspect it's a combination of environment and a specific configuration you're using. I'd love to track this down and fix it! -- Sam Clippinger -- David Davidov | System Administrator | ICDSoft Ltd. ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Duplicate ALLOWED from log entries
Found it! The bug is being triggered because multiple messages are being delivered in a single connection. spamdyke is caching the list of valid recipient addresses so it can print the log entries but isn't clearing them afterwards. So with each additional message, the list of addresses grows and duplicate messages are printed. I'll incorporate the fix in the next version, hopefully coming soon. -- Sam Clippinger On Sep 3, 2013, at 4:30 PM, Sam Clippinger wrote: To be honest, I've spent almost no time on it. With what little time I've put towards spamdyke recently, I've been trying to finish testing the recipient validation code for the next version. I finally had to admit the tests were never going to finish on my laptop and I've rented a VPS for testing. The scripts are running much faster now -- I'm nearly done with the 166K tests to make sure my understanding of qmail's recipient processing is correct. After that, just 71K tests to check the recipient validation code in spamdyke. Then the last 1K scripts to test the rest of spamdyke. Simple! Anyway, I'll try to take a look at this issue this week if I can. :) -- Sam Clippinger On Sep 2, 2013, at 8:58 AM, David Davidov wrote: Hello, Any progress with this issue? regards, David Davidov On 06/18/2013 06:47 PM, David Davidov wrote: Hi Sam, I am a colleague of Theodor. This is a simple way to trigger the problem: dave~$ telnet mx.example.com 25 Connected to mx.example.com Escape character is '^]'. 220 mx.example.com ESMTP mail from: test-sen...@mx.example.com 250 ok rcpt to: test-r...@mx.example.com 250 ok data 354 go ahead . 250 ok 1370523483 qp 26283 mail from: test-sen...@mx.example.com 250 ok rcpt to: test-r...@mx.example.com 250 ok data 354 go ahead . 250 ok 1370523502 qp 26625 The first message produces one log entry: Jun 18 15:58:03 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523483_qp_26283 But for the second message we have two entries: Jun 18 15:58:22 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523502_qp_26625 Jun 18 15:58:22 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523502_qp_26625 We use spamdyke ver. 4.3.1 locally compiled on Debian 6.0.7. No header filtering. Here is our spamdyke.conf --- greeting-delay-secs=0 reject-empty-rdns log-level=info idle-timeout-secs=300 ip-whitelist-file=/home/vpopmail/etc/spamdyke_whitelist.txt recipient-whitelist-file=/var/qmail/spamdyke_recipient_whitelist.txt dns-whitelist-entry=antirbl.example.com graylist-level=always graylist-dir=/var/qmail/graylist graylist-min-secs=60 tls-certificate-file=/var/qmail/control/servercert.pem tls-privatekey-file=/var/qmail/control/servercert.pem local-domains-file=/var/qmail/control/rcpthosts local-domains-file=/var/qmail/control/morercpthosts dns-timeout-secs=3 rejection-text-empty-rdns=Refused. You have no reverse DNS entry. Contact ab...@example.com for details. rejection-text-ip-in-cc-rdns=Refused. Your reverse DNS entry contains your IP address and a country code. Contact ab...@example.com for details. rejection-text-unresolvable-rdns=Refused. Your reverse DNS entry does not resolve. Contact ab...@example.com for details. --- BRs, David Davidov On 01/-10/-28163 09:59 PM, Sam Clippinger wrote: I'm very sorry it's taken so long to get back to you on this; I've been buried at work and haven't had any time to investigate. It definitely sounds like you've hit a bug. spamdyke does save the addresses of all the recipients in order to print them all out in a loop, but only when the header blacklist feature is enabled. It does this because the recipient names have already gone by before the message header is sent, so it must save the recipient addresses to print either ALLOWED or DENIED once the header is finished. But it should only do this once and I'm not seeing a way to trigger that code more than once, though it certainly looks like that's what's happening. So let me start with all the standard questions: what OS and version are you on? What version of spamdyke are you using? Could you please post your configuration file(s) (or send them to me directly)? Would you mind turning on spamdyke's full logging feature (the full-log-dir option) and capturing one of these sessions? Needless to say this behavior isn't by design and it's not happening on any of the servers I manage (and I use the header-blacklist feature on every one).
Re: [spamdyke-users] Duplicate ALLOWED from log entries
Hello, Any progress with this issue? regards, David Davidov On 06/18/2013 06:47 PM, David Davidov wrote: Hi Sam, I am a colleague of Theodor. This is a simple way to trigger the problem: dave~$ telnet mx.example.com 25 Connected to mx.example.com Escape character is '^]'. 220 mx.example.com ESMTP mail from: test-sen...@mx.example.com 250 ok rcpt to: test-r...@mx.example.com 250 ok data 354 go ahead . 250 ok 1370523483 qp 26283 mail from: test-sen...@mx.example.com 250 ok rcpt to: test-r...@mx.example.com 250 ok data 354 go ahead . 250 ok 1370523502 qp 26625 The first message produces one log entry: Jun 18 15:58:03 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523483_qp_26283 But for the second message we have two entries: Jun 18 15:58:22 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523502_qp_26625 Jun 18 15:58:22 mx spamdyke[26194]: ALLOWED from: test-sen...@mx.example.com to: test-r...@mx.example.com origin_ip: 213.145.98.39 origin_rdns: ws.example.com auth: (unknown) encryption: (none) reason: 250_ok_1370523502_qp_26625 We use spamdyke ver. 4.3.1 locally compiled on Debian 6.0.7. No header filtering. Here is our spamdyke.conf --- greeting-delay-secs=0 reject-empty-rdns log-level=info idle-timeout-secs=300 ip-whitelist-file=/home/vpopmail/etc/spamdyke_whitelist.txt recipient-whitelist-file=/var/qmail/spamdyke_recipient_whitelist.txt dns-whitelist-entry=antirbl.example.com graylist-level=always graylist-dir=/var/qmail/graylist graylist-min-secs=60 tls-certificate-file=/var/qmail/control/servercert.pem tls-privatekey-file=/var/qmail/control/servercert.pem local-domains-file=/var/qmail/control/rcpthosts local-domains-file=/var/qmail/control/morercpthosts dns-timeout-secs=3 rejection-text-empty-rdns=Refused. You have no reverse DNS entry. Contact ab...@example.com for details. rejection-text-ip-in-cc-rdns=Refused. Your reverse DNS entry contains your IP address and a country code. Contact ab...@example.com for details. rejection-text-unresolvable-rdns=Refused. Your reverse DNS entry does not resolve. Contact ab...@example.com for details. --- BRs, David Davidov On 01/-10/-28163 09:59 PM, Sam Clippinger wrote: I'm very sorry it's taken so long to get back to you on this; I've been buried at work and haven't had any time to investigate. It definitely sounds like you've hit a bug. spamdyke does save the addresses of all the recipients in order to print them all out in a loop, but only when the header blacklist feature is enabled. It does this because the recipient names have already gone by before the message header is sent, so it must save the recipient addresses to print either ALLOWED or DENIED once the header is finished. But it should only do this once and I'm not seeing a way to trigger that code more than once, though it certainly looks like that's what's happening. So let me start with all the standard questions: what OS and version are you on? What version of spamdyke are you using? Could you please post your configuration file(s) (or send them to me directly)? Would you mind turning on spamdyke's full logging feature (the full-log-dir option) and capturing one of these sessions? Needless to say this behavior isn't by design and it's not happening on any of the servers I manage (and I use the header-blacklist feature on every one). After spending a little while testing and tracking through the code I can't reproduce this problem, so I suspect it's a combination of environment and a specific configuration you're using. I'd love to track this down and fix it! -- Sam Clippinger -- David Davidov | System Administrator | ICDSoft Ltd. ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Duplicate ALLOWED from log entries
On 24/05/13 02:07, Eric Shubert wrote: On 05/23/2013 03:35 PM, Lutz Petersen wrote: Hi, Which in some extreme cases where session had 9000 recipients led to multi GB log file. Imho you should configure your Spamdyke not to accept such nonsense. There is absolute no reason to accept more than a dozen recipients. Use e.g. this in your spamdyke.conf: max-recipients=15 And you'll get off those defect hosts.. I agree Lutz, and use this setting myself. I think that Teodor is referring to something different though. While qmail sends only one message per smtp session, the smtp spec allows for multiple messages to be sent in a single smtp session, however rare that might be. I expect this is what Teodor's seeing. The spamdyke docs say that max-recipients is applied to the connection, not each message, so use of this option would certainly help (more so than if it was applied to each message as I believe chkuser does). Sam, will you please confirm that this is per connection and not per message? It appears to me that spamdyke has a bug in how it's logging this type of session. I'm interested to see what Sam finds with this. Eric is correct here. We are going to add a reasonably low value max-recipients and it is going to mitigate the problem to some extent, but still at max-recipients=15 we'd have 15+14+13+12 etc. totalling 120 log entries (arithmetic progression). -- Teodor Milkov | System Administrator | ICDSoft Ltd. ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Duplicate ALLOWED from log entries
Hi, Which in some extreme cases where session had 9000 recipients led to multi GB log file. Imho you should configure your Spamdyke not to accept such nonsense. There is absolute no reason to accept more than a dozen recipients. Use e.g. this in your spamdyke.conf: max-recipients=15 And you'll get off those defect hosts.. Lutz Petersen ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
Re: [spamdyke-users] Duplicate ALLOWED from log entries
On 05/23/2013 03:35 PM, Lutz Petersen wrote: Hi, Which in some extreme cases where session had 9000 recipients led to multi GB log file. Imho you should configure your Spamdyke not to accept such nonsense. There is absolute no reason to accept more than a dozen recipients. Use e.g. this in your spamdyke.conf: max-recipients=15 And you'll get off those defect hosts.. Lutz Petersen I agree Lutz, and use this setting myself. I think that Teodor is referring to something different though. While qmail sends only one message per smtp session, the smtp spec allows for multiple messages to be sent in a single smtp session, however rare that might be. I expect this is what Teodor's seeing. The spamdyke docs say that max-recipients is applied to the connection, not each message, so use of this option would certainly help (more so than if it was applied to each message as I believe chkuser does). Sam, will you please confirm that this is per connection and not per message? It appears to me that spamdyke has a bug in how it's logging this type of session. I'm interested to see what Sam finds with this. -- -Eric 'shubes' ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
[spamdyke-users] Duplicate ALLOWED from log entries
Hello, I did a quick search in the mailing list about this issue, but didn't find anything related, so here I go: When an email with multiple RCPT TO is sent in single SMTP session, it seems all previous recipients are logged at each new RCPT TO command. See attached spamdyke.txt log for details (I've replaced original sender/recipient names for privacy reasons). Basically if there's incoming mail from one sender to 3 recipients in single smtp session I see something like: ALLOWED from: sender to recipient-1 ALLOWED from: sender to recipient-1 ALLOWED from: sender to recipient-2 ALLOWED from: sender to recipient-1 ALLOWED from: sender to recipient-2 ALLOWED from: sender to recipient-3 Which in some extreme cases where session had 9000 recipients led to multi GB log file. Glancing quickly through sources I didn't find how this works, but I'll look again later this week when I have more time. -- Teodor Milkov | System Administrator | ICDSoft Ltd. May 19 08:01:34 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:34 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:34 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:34 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:34 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:34 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:35 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:35 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:35 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... May 19 08:01:35 spamdyke[2327]: ALLOWED from: sen...@example.org to: recipien...@example.net origin_ip: ... ___ spamdyke-users mailing list spamdyke-users@spamdyke.org http://www.spamdyke.org/mailman/listinfo/spamdyke-users
