Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote: I discovered Net::DNS::Resolver::errorstring, and put some more logging into SA, and the problem is really simple: my caching-only nameserver times out when looking up NS records for a site that's not in the cache. Not entirely surprising, with a 3-second timeout in SA. And my site is infinitely small (just me), so it's going to be fairly common that one of the well-known sites is not in cache. SA realizes this, and tries to loop, in Dns.pm's is_dns_available, but the loop is coded wrong, because either a success or a failure breaks out of the loop! A timeout in lookup_ns will result in $result defined, but containing no records, and that triggers the failed horribly clause, setting $IS_DNS_AVAILABLE to zero until mimedefang eventually cycles the child process. I'm a little behind on my SA mail--I noticed this a few weeks ago. My fix since I didn't want to patch the code itself was to put dns_available yes in local.cf. That skips the whole is_dns_avialable test in DNS.pm. -- ___ Rick Beebe(203) 785-6416 Manager, Systems Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___
Re: SA 3.01 eventually stops noticing DNSBLs
Since none of the devs have posted a followup on this, I think that perhaps this is worth a bug in Bugzilla. If the analysis is valid it needs to be fixed. If it isn't, one of the devs will probably explain why it should work as it does. Loren I discovered Net::DNS::Resolver::errorstring, and put some more logging into SA, and the problem is really simple: my caching-only nameserver times out when looking up NS records for a site that's not in the cache. Not entirely surprising, with a 3-second timeout in SA. And my site is infinitely small (just me), so it's going to be fairly common that one of the well-known sites is not in cache. SA realizes this, and tries to loop, in Dns.pm's is_dns_available, but the loop is coded wrong, because either a success or a failure breaks out of the loop! A timeout in lookup_ns will result in $result defined, but containing no records, and that triggers the failed horribly clause, setting $IS_DNS_AVAILABLE to zero until mimedefang eventually cycles the child process. I *think* the bug fix is just to remove that whole else clause from is_dns_available, but as a Perl novice I'd certainly like someone to double-check that. And, you know, now that I look at it, it seems like is_dns_available uses lookup_ns to test general DNS availability, but lookup_ns has its own caching that would seem to defeat the point of the test if a site is ever hit twice!
Re: SA 3.01 eventually stops noticing DNSBLs
On Thu, Feb 24, 2005 at 07:17:16PM -0800, Loren Wilton wrote: Since none of the devs have posted a followup on this, I think that perhaps this is worth a bug in Bugzilla. If the analysis is valid it needs to be fixed. If it isn't, one of the devs will probably explain why it should work as it does. Actually a bug was already posted and a patch put into trunk. The devs are silent and stealthy. ;) -- Randomly Generated Tagline: Flanders! My socks feel dirty! Gimme some water to wash 'em! -- Homer Simpson Boy-Scoutz n the Hood pgpmwfPu94h6K.pgp Description: PGP signature
Re: SA 3.01 eventually stops noticing DNSBLs
Actually a bug was already posted and a patch put into trunk. The devs are silent and stealthy. ;) Ah! Guess I should read dev before users rather than the other way around! Loren
Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote:
[SNIP]
I tried to create a test harness to see if I can replicate this outside
of SA, but for some reason, even though I double-checked the code I
copied from Dns.pm, I'm getting weird results - it's always giving me
the root nameservers, instead of the name servers for each of the
domains. This is true with recurse = 0, recurse = 1, or recurse left
out entirely as it is in Dns.pm. I'm no Perl whiz; can anyone see my
mistake?
Code follows:
-
#!/usr/bin/perl
no strict;
no warnings;
require Net::DNS;
require Net::DNS::Resolver;
use strict;
use warnings;
my @EXISTING_DOMAINS = qw{
adelphia.net
akamai.com
apache.org
cingular.com
colorado.edu
comcast.net
doubleclick.com
ebay.com
gmx.net
google.com
intel.com
kernel.org
linux.org
mit.edu
motorola.com
msn.com
sourceforge.net
sun.com
w3.org
yahoo.com
};
my $res = Net::DNS::Resolver-new (
recurse = 0,
retry = 1,
retrans = 0,
dnsrch = 0,
defnames = 0,
tcp_timeout = 3,
udp_timeout = 3,
persistent_tcp = 1,
persistent_udp = 1
);
die unless defined $res;
for(;;) {
my @domains = @EXISTING_DOMAINS;
my $domain = splice(@domains, rand(@domains), 1);
print trying '$domain'...\n;
lookup_ns($domain);
}
sub lookup_ns {
my ($self, $dom) = @_;
Since you're not using this as a Perl Module (OOP) my
guess is that $self contains the value you expect to be in $dom and $dom
is NULL.
Try removing $self from your argument list and make it look like:
my ($dom) = @_;
and see if that works for you.
debug statements are your friend. :)
hope this helps
alan
Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote: A quick test shows that indeed, an awful lot of domains are repeatedly failing in lookup_ns, but that different domains fail at different times - the domains that repeatedly fail right now were fine last night in the SA logs. So it looks like this is something (intermittment) to do with the resolver on my system, or perhaps the caching nameserver, and nothing to do with SA. I'll keep digging and report back what I find. If anyone has any tips, of course, feel free to let me know. I spoke too soon. Turns out I'd accidentally left recurse=0 in the test harness. No wonder it was failing so often. I discovered Net::DNS::Resolver::errorstring, and put some more logging into SA, and the problem is really simple: my caching-only nameserver times out when looking up NS records for a site that's not in the cache. Not entirely surprising, with a 3-second timeout in SA. And my site is infinitely small (just me), so it's going to be fairly common that one of the well-known sites is not in cache. SA realizes this, and tries to loop, in Dns.pm's is_dns_available, but the loop is coded wrong, because either a success or a failure breaks out of the loop! A timeout in lookup_ns will result in $result defined, but containing no records, and that triggers the failed horribly clause, setting $IS_DNS_AVAILABLE to zero until mimedefang eventually cycles the child process. I *think* the bug fix is just to remove that whole else clause from is_dns_available, but as a Perl novice I'd certainly like someone to double-check that. And, you know, now that I look at it, it seems like is_dns_available uses lookup_ns to test general DNS availability, but lookup_ns has its own caching that would seem to defeat the point of the test if a site is ever hit twice! Jay
Re: SA 3.01 eventually stops noticing DNSBLs
On Tue, 22 Feb 2005, Andy Jezierski wrote:
Kelson [EMAIL PROTECTED] wrote on 02/22/2005 11:30:46 AM:
Jay Levitt wrote:
I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At
some point, SA seems to stop doing lookups on the DNSBLs; spam gets
through that is listed in multiple BLs; if I check manually with
spamassassin -t, it detects the BL entry, even if I run it moments
after
the spam was received.
[snip..]
Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on
an upgrade.
That happened to my server a while back -- We were just using
/etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was
no sa-mimedefang.cf, so it created it with the defaults -- and the
defaults disable DNSBLs.
Could this be the same problem as the discussion in the Spammer
Anti-SURBL tactic thread?
No, the problem that I saw was caused by the contents of the message.
it didn't matter how I fed it to SA, SURBL did not 'fire' on the spammer
payload URL, eventho the site was in several lists.
When I first saw the spam I was puzzled and ran it thru SA with full
debugging to see what was going on.
( spamassassin -D rulesrun=255 message.txt )
I saw lots of SURBL tests on the decoy URLs but none of the payload.
I removed the CR characters from the payload URL, retested but still
no hits, removed most of the decoy URLs from the message, then a
retest did hit. So it was entirely due to the message content
exploiting two weaknesses in SA.
PS: if you do a full debug SA run, it's best to redirect the output to
a file for reading, it produces thousands of lines of output. ;)
--
Dave Funk University of Iowa
dbfunk (at) engineering.uiowa.eduCollege of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include std_disclaimer.h
Better is not better, 'standard' is better. B{
Re: SA 3.01 eventually stops noticing DNSBLs
On Tuesday, February 22, 2005, 9:57:02 AM, Andy Jezierski wrote: Kelson [EMAIL PROTECTED] wrote on 02/22/2005 11:30:46 AM: Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on an upgrade. That happened to my server a while back -- We were just using /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was no sa-mimedefang.cf, so it created it with the defaults -- and the defaults disable DNSBLs. Could this be the same problem as the discussion in the Spammer Anti-SURBL tactic thread? Andy Possibly, but I think the lots of URI spams are still somewhat rare, so it should only be an occasional occurrence. It sounds like this thread SA 3.01 eventually stops noticing DNSBLs is more likely an installation or configuration issue for this particular system. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
On Tuesday, February 22, 2005, 10:25:36 PM, Jay Levitt wrote: Kelson wrote: Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on an upgrade. That happened to my server a while back -- We were just using /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was no sa-mimedefang.cf, so it created it with the defaults -- and the defaults disable DNSBLs. Nope, that's not it. I've been throwing debug code in bit by bit. (More accurately, I've been re-copying the dbg statements as warns, because while there's plenty of useful output, there are just too many un-categorized dbg statements to leave debug enabled... sigh.) Looks like every once in a while, the lookup_ns sanity-checks that SA does on well-known domains are returning with zero NS records. Still not sure why that happens yet, or exactly what is going on, but that does understandably lead SA to disable DNSBL processing for a while. Hmm, that sounds like something that may deserve a bugzilla. Can anyone else replicate that behavior? Is your Net::DNS completely current and happy? Have you checked all of your: /etc/resolv.conf $HOME/.resolv.conf ./.resolv.conf for the user mimedefang or SA runs as to make sure they're all correct and all the name servers on them resolve the RBLs correctly? Also when you say At some point, SA seems to stop doing lookups on the DNSBLs what is the time scale? Does At some point mean at some times of day, after several months of operation and all the time now, for a few hours at a time, for every 6th message, etc.? Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
Jeff Chan wrote (quoting Jay Levitt):
Nope, that's not it. I've been throwing debug code in bit by bit.
(More accurately, I've been re-copying the dbg statements as "warns",
because while there's plenty of useful output, there are just too many
un-categorized dbg statements to leave debug enabled... sigh.) Looks
like every once in a while, the lookup_ns sanity-checks that SA does on
well-known domains are returning with zero NS records. Still not sure
why that happens yet, or exactly what is going on, but that does
understandably lead SA to disable DNSBL processing for a while.
Hmm, that sounds like something that may deserve a bugzilla. Can
anyone else replicate that behavior?
Is your Net::DNS completely current and happy?
Yep, 0.48.
Have you checked all of your:
/etc/resolv.conf
$HOME/.resolv.conf
./.resolv.conf
for the user mimedefang or SA runs as to make sure they're all
correct and all the name servers on them resolve the RBLs
correctly?
Yep. The only *resolv.conf file on the system is /etc/resolv.conf.
Also when you say "At some point, SA seems to stop doing lookups
on the DNSBLs" what is the time scale? Does "At some point" mean
at some times of day, after several months of operation and all
the time now, for a few hours at a time, for every 6th message,
etc.?
After it's been running for a few hours, the lookup_ns check (which
does a sanity check to make sure we can resolve the nameservers of a
well-known domain) seems to fail. Or, rather, it returns, but with 0
entries in the array. This causes SA to stop doing any RBL lookups for
some period of time.
I tried to create a test harness to see if I can replicate this outside
of SA, but for some reason, even though I double-checked the code I
copied from Dns.pm, I'm getting weird results - it's always giving me
the root nameservers, instead of the name servers for each of the
domains. This is true with recurse = 0, recurse = 1, or
recurse left out entirely as it is in Dns.pm. I'm no Perl whiz; can
anyone see my mistake?
Code follows:
-
#!/usr/bin/perl
no strict;
no warnings;
require Net::DNS;
require Net::DNS::Resolver;
use strict;
use warnings;
my @EXISTING_DOMAINS = qw{
adelphia.net
akamai.com
apache.org
cingular.com
colorado.edu
comcast.net
doubleclick.com
ebay.com
gmx.net
google.com
intel.com
kernel.org
linux.org
mit.edu
motorola.com
msn.com
sourceforge.net
sun.com
w3.org
yahoo.com
};
my $res = Net::DNS::Resolver-new (
recurse = 0,
retry = 1,
retrans = 0,
dnsrch = 0,
defnames = 0,
tcp_timeout = 3,
udp_timeout = 3,
persistent_tcp = 1,
persistent_udp = 1
);
die unless defined $res;
for(;;) {
my @domains = @EXISTING_DOMAINS;
my $domain = splice(@domains, rand(@domains), 1);
print "trying '$domain'...\n";
lookup_ns($domain);
}
sub lookup_ns {
my ($self, $dom) = @_;
my $query = $res-search($dom, 'NS');
if ($query) {
foreach my $rr ($query-answer) {
print "type=", $rr-type, ", nsdname=", $rr-nsdname, "\n";
}
}
else {
print "ERROR! no query\n";
}
}
1;
Re: SA 3.01 eventually stops noticing DNSBLs
On Wednesday, February 23, 2005, 8:38:31 AM, Jay Levitt wrote: After it's been running for a few hours, the lookup_ns check (which does a sanity check to make sure we can resolve the nameservers of a well-known domain) seems to fail. Or, rather, it returns, but with 0 entries in the array. This causes SA to stop doing any RBL lookups for some period of time. I tried to create a test harness to see if I can replicate this outside of SA, but for some reason, even though I double-checked the code I copied from Dns.pm, I'm getting weird results - it's always giving me the root nameservers, instead of the name servers for each of the domains. This is true with recurse = 0, recurse = 1, or recurse left out entirely as it is in Dns.pm. I'm no Perl whiz; can anyone see my mistake? Off the top of my head, that sounds like a DNS configuration error. Do you have a recent root hints file? That got updated a couple times over the past couple years IIRC. ; This file is made available by InterNIC ; under anonymous FTP as ; file/domain/named.root ; on server FTP.INTERNIC.NET ; -OR-RS.INTERNIC.NET ; ; last update:Jan 29, 2004 ; related version of root zone: 2004012900 Somewhat WAG, but probably worth checking. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
Jeff Chan wrote: On Wednesday, February 23, 2005, 8:38:31 AM, Jay Levitt wrote: I tried to create a test harness to see if I can replicate this outside of SA, but for some reason, even though I double-checked the code I copied from Dns.pm, I'm getting weird results - it's always giving me the root nameservers, instead of the name servers for each of the domains. This is true with recurse = 0, recurse = 1, or recurse left out entirely as it is in Dns.pm. I'm no Perl whiz; can anyone see my mistake? Off the top of my head, that sounds like a DNS configuration error. Do you have a recent root hints file? That got updated a couple times over the past couple years IIRC. Nope, that's not it - I should clarify that this same code does get the right NS servers when it's running in SA, just not standalone (and I'm using the same login). So I'm doing something wrong Perl-wise; I just don't know what... Jay
Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on an upgrade. That happened to my server a while back -- We were just using /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was no sa-mimedefang.cf, so it created it with the defaults -- and the defaults disable DNSBLs. -- Kelson Vibber SpeedGate Communications www.speed.net
Re: SA 3.01 eventually stops noticing DNSBLs
Kelson [EMAIL PROTECTED] wrote on 02/22/2005 11:30:46 AM: Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on an upgrade. That happened to my server a while back -- We were just using /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was no sa-mimedefang.cf, so it created it with the defaults -- and the defaults disable DNSBLs. -- Kelson Vibber SpeedGate Communications www.speed.net Could this be the same problem as the discussion in the Spammer Anti-SURBL tactic thread? Andy
Re: SA 3.01 eventually stops noticing DNSBLs
On Saturday, February 19, 2005, 8:05:05 AM, Jay Levitt wrote: Just checked.. there is only one resolv.conf on the system, in /etc/resolv.conf, and it correctly points to my own machine, which runs a caching named (actually caching for the world, authoritative for my own domain). Note that the other two files mentioned are .resolv.conf, not resolv.conf: /etc/resolv.conf $HOME/.resolv.conf ./.resolv.conf and that you'd want to check them for the specific user that SpamAssassin (spamd, etc.) runs as. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
SA 3.01 eventually stops noticing DNSBLs
I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Jay Levitt
Re: SA 3.01 eventually stops noticing DNSBLs
On Friday, February 18, 2005, 8:35:35 PM, Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. One thing to check is whether your name resolution is truly correct. Are you running Net::DNS 0.48? Is it installed and upgraded in a consistent way (i.e. always rpms or always CPAN or always tarballs)? Using different upgrade methods can confuse things. Did you see the recent thread about the various resolve.conf's used by Net::DNS? Are they all correct for the user SpamAssassin runs as? Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
