Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote: I discovered Net::DNS::Resolver::errorstring, and put some more logging into SA, and the problem is really simple: my caching-only nameserver times out when looking up NS records for a site that's not in the cache. Not entirely surprising, with a 3-second timeout in SA. And my site is infinitely small (just me), so it's going to be fairly common that one of the well-known sites is not in cache. SA realizes this, and tries to loop, in Dns.pm's is_dns_available, but the loop is coded wrong, because either a success or a failure breaks out of the loop! A timeout in lookup_ns will result in $result defined, but containing no records, and that triggers the "failed horribly" clause, setting $IS_DNS_AVAILABLE to zero until mimedefang eventually cycles the child process. I'm a little behind on my SA mail--I noticed this a few weeks ago. My "fix" since I didn't want to patch the code itself was to put "dns_available yes" in local.cf. That skips the whole is_dns_avialable test in DNS.pm. -- ___ Rick Beebe(203) 785-6416 Manager, Systems & Network Engineering FAX: (203) 785-3481 ITS-Med Production Systems[EMAIL PROTECTED] Yale University School of Medicine Suite 124, 100 Church Street South http://its.med.yale.edu New Haven, CT 06519 ___
Re: SA 3.01 not adding correcly?
At 02:34 AM 3/7/2005, Philipp Snizek wrote: When adding Points I get a total score of 3.4. Why does SpamAssassin get 3.3? One word.. Rouding.. None of the rule scores are actually 2 digits, they're all 4 digit numbers.
Re: SA 3.01 not adding correcly?
Philipp Snizek wrote: > Content analysis details: (3.3 points, 3.0 required) > [...] > When adding Points I get a total score of 3.4. > Why does SpamAssassin get 3.3? Rounding errors. The numbers are really things like 1.07 and 0.06 and such. when printed in the summary only the tenths are printed. But internally spamassassin is using the full precision available. Bob
SA 3.01 not adding correcly?
Hi Content analysis details: (3.3 points, 3.0 required) pts rule name description -- -- 0.5 NO_RDNSSending MTA has no reverse DNS (Postfix variant) 0.9 SARE_SUB_LONG_SUBJ_170 Subject is excessively long -- more than 170 chars 0.6 SARE_SUB_LONG_SUBJ_140 Subject is excessively long -- more than 139 chars 1.5 MR_STRANGE_QUESTIONURI: MR_STRANGE_QUESTION 0.1 SARE_HTML_URI_UNSASP URI: text uri to unsubscribe link 0.0 BAYES_50 BODY: Bayesian spam probability is 40 to 60% [score: 0.5019] 0.0 UPPERCASE_25_50message body is 25-50% uppercase 0.1 SARE_FROM_SPAM_WORD3 From address suggests this may be spam 0.5 ISO_7BITS ISO charset announced as 7 bit (or bad rule ?) -0.8 AWLAWL: From: address is in the auto white-list When adding Points I get a total score of 3.4. Why does SpamAssassin get 3.3? Thank you & regards Philipp
Re: SA 3.01 eventually stops noticing DNSBLs
>Actually a bug was already posted and a patch put into trunk. >The devs are silent and stealthy. ;) Ah! Guess I should read dev before users rather than the other way around! Loren
Re: SA 3.01 eventually stops noticing DNSBLs
On Thu, Feb 24, 2005 at 07:17:16PM -0800, Loren Wilton wrote: > Since none of the devs have posted a followup on this, I think that perhaps > this is worth a bug in Bugzilla. If the analysis is valid it needs to be > fixed. If it isn't, one of the devs will probably explain why it should > work as it does. Actually a bug was already posted and a patch put into trunk. The devs are silent and stealthy. ;) -- Randomly Generated Tagline: Flanders! My socks feel dirty! Gimme some water to wash 'em! -- Homer Simpson Boy-Scoutz n the Hood pgpmwfPu94h6K.pgp Description: PGP signature
Re: SA 3.01 eventually stops noticing DNSBLs
Since none of the devs have posted a followup on this, I think that perhaps this is worth a bug in Bugzilla. If the analysis is valid it needs to be fixed. If it isn't, one of the devs will probably explain why it should work as it does. Loren > I discovered Net::DNS::Resolver::errorstring, and put some more logging > into SA, and the problem is really simple: my caching-only nameserver > times out when looking up NS records for a site that's not in the > cache. Not entirely surprising, with a 3-second timeout in SA. And my > site is infinitely small (just me), so it's going to be fairly common > that one of the well-known sites is not in cache. > > SA realizes this, and tries to loop, in Dns.pm's is_dns_available, but > the loop is coded wrong, because either a success or a failure breaks > out of the loop! A timeout in lookup_ns will result in $result defined, > but containing no records, and that triggers the "failed horribly" > clause, setting $IS_DNS_AVAILABLE to zero until mimedefang eventually > cycles the child process. > > I *think* the bug fix is just to remove that whole else clause from > is_dns_available, but as a Perl novice I'd certainly like someone to > double-check that. > > And, you know, now that I look at it, it seems like is_dns_available > uses lookup_ns to test general DNS availability, but lookup_ns has its > own caching that would seem to defeat the point of the test if a site is > ever hit twice!
Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote: A quick test shows that indeed, an awful lot of domains are repeatedly failing in lookup_ns, but that different domains fail at different times - the domains that repeatedly fail right now were fine last night in the SA logs. So it looks like this is something (intermittment) to do with the resolver on my system, or perhaps the caching nameserver, and nothing to do with SA. I'll keep digging and report back what I find. If anyone has any tips, of course, feel free to let me know. I spoke too soon. Turns out I'd accidentally left "recurse=>0" in the test harness. No wonder it was failing so often. I discovered Net::DNS::Resolver::errorstring, and put some more logging into SA, and the problem is really simple: my caching-only nameserver times out when looking up NS records for a site that's not in the cache. Not entirely surprising, with a 3-second timeout in SA. And my site is infinitely small (just me), so it's going to be fairly common that one of the well-known sites is not in cache. SA realizes this, and tries to loop, in Dns.pm's is_dns_available, but the loop is coded wrong, because either a success or a failure breaks out of the loop! A timeout in lookup_ns will result in $result defined, but containing no records, and that triggers the "failed horribly" clause, setting $IS_DNS_AVAILABLE to zero until mimedefang eventually cycles the child process. I *think* the bug fix is just to remove that whole else clause from is_dns_available, but as a Perl novice I'd certainly like someone to double-check that. And, you know, now that I look at it, it seems like is_dns_available uses lookup_ns to test general DNS availability, but lookup_ns has its own caching that would seem to defeat the point of the test if a site is ever hit twice! Jay
Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote:
[SNIP]
I tried to create a test harness to see if I can replicate this outside
of SA, but for some reason, even though I double-checked the code I
copied from Dns.pm, I'm getting weird results - it's always giving me
the root nameservers, instead of the name servers for each of the
domains. This is true with recurse => 0, recurse => 1, or recurse left
out entirely as it is in Dns.pm. I'm no Perl whiz; can anyone see my
mistake?
Code follows:
-
#!/usr/bin/perl
no strict;
no warnings;
require Net::DNS;
require Net::DNS::Resolver;
use strict;
use warnings;
my @EXISTING_DOMAINS = qw{
adelphia.net
akamai.com
apache.org
cingular.com
colorado.edu
comcast.net
doubleclick.com
ebay.com
gmx.net
google.com
intel.com
kernel.org
linux.org
mit.edu
motorola.com
msn.com
sourceforge.net
sun.com
w3.org
yahoo.com
};
my $res = Net::DNS::Resolver->new (
recurse => 0,
retry => 1,
retrans => 0,
dnsrch => 0,
defnames => 0,
tcp_timeout => 3,
udp_timeout => 3,
persistent_tcp => 1,
persistent_udp => 1
);
die unless defined $res;
for(;;) {
my @domains = @EXISTING_DOMAINS;
my $domain = splice(@domains, rand(@domains), 1);
print "trying '$domain'...\n";
lookup_ns($domain);
}
sub lookup_ns {
my ($self, $dom) = @_;
Since you're not using this as a Perl Module (OOP) my
guess is that $self contains the value you expect to be in $dom and $dom
is NULL.
Try removing $self from your argument list and make it look like:
my ($dom) = @_;
and see if that works for you.
debug statements are your friend. :)
hope this helps
alan
Re: SA 3.01 eventually stops noticing DNSBLs
Jeff Chan wrote: On Wednesday, February 23, 2005, 8:38:31 AM, Jay Levitt wrote: I tried to create a test harness to see if I can replicate this outside of SA, but for some reason, even though I double-checked the code I copied from Dns.pm, I'm getting weird results - it's always giving me the root nameservers, instead of the name servers for each of the domains. This is true with recurse => 0, recurse => 1, or recurse left out entirely as it is in Dns.pm. I'm no Perl whiz; can anyone see my mistake? Off the top of my head, that sounds like a DNS configuration error. Do you have a recent root hints file? That got updated a couple times over the past couple years IIRC. Nope, that's not it - I should clarify that this same code does get the right NS servers when it's running in SA, just not standalone (and I'm using the same login). So I'm doing something wrong Perl-wise; I just don't know what... Jay
Re: SA 3.01 eventually stops noticing DNSBLs
On Wednesday, February 23, 2005, 8:38:31 AM, Jay Levitt wrote: > After it's been running for a few hours, the lookup_ns check (which does > a sanity check to make sure we can resolve the nameservers of a > well-known domain) seems to fail. Or, rather, it returns, but with 0 > entries in the array. This causes SA to stop doing any RBL lookups for > some period of time. > I tried to create a test harness to see if I can replicate this outside > of SA, but for some reason, even though I double-checked the code I > copied from Dns.pm, I'm getting weird results - it's always giving me > the root nameservers, instead of the name servers for each of the > domains. This is true with recurse => 0, recurse => 1, or recurse left > out entirely as it is in Dns.pm. I'm no Perl whiz; can anyone see my > mistake? Off the top of my head, that sounds like a DNS configuration error. Do you have a recent root hints file? That got updated a couple times over the past couple years IIRC. > ; This file is made available by InterNIC > ; under anonymous FTP as > ; file/domain/named.root > ; on server FTP.INTERNIC.NET > ; -OR-RS.INTERNIC.NET > ; > ; last update:Jan 29, 2004 > ; related version of root zone: 2004012900 Somewhat WAG, but probably worth checking. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
Jeff Chan wrote (quoting Jay Levitt):
Nope, that's not it. I've been throwing debug code in bit by bit.
(More accurately, I've been re-copying the dbg statements as "warns",
because while there's plenty of useful output, there are just too many
un-categorized dbg statements to leave debug enabled... sigh.) Looks
like every once in a while, the lookup_ns sanity-checks that SA does on
well-known domains are returning with zero NS records. Still not sure
why that happens yet, or exactly what is going on, but that does
understandably lead SA to disable DNSBL processing for a while.
Hmm, that sounds like something that may deserve a bugzilla. Can
anyone else replicate that behavior?
Is your Net::DNS completely current and happy?
Yep, 0.48.
Have you checked all of your:
/etc/resolv.conf
$HOME/.resolv.conf
./.resolv.conf
for the user mimedefang or SA runs as to make sure they're all
correct and all the name servers on them resolve the RBLs
correctly?
Yep. The only *resolv.conf file on the system is /etc/resolv.conf.
Also when you say "At some point, SA seems to stop doing lookups
on the DNSBLs" what is the time scale? Does "At some point" mean
at some times of day, after several months of operation and all
the time now, for a few hours at a time, for every 6th message,
etc.?
After it's been running for a few hours, the lookup_ns check (which
does a sanity check to make sure we can resolve the nameservers of a
well-known domain) seems to fail. Or, rather, it returns, but with 0
entries in the array. This causes SA to stop doing any RBL lookups for
some period of time.
I tried to create a test harness to see if I can replicate this outside
of SA, but for some reason, even though I double-checked the code I
copied from Dns.pm, I'm getting weird results - it's always giving me
the root nameservers, instead of the name servers for each of the
domains. This is true with recurse => 0, recurse => 1, or
recurse left out entirely as it is in Dns.pm. I'm no Perl whiz; can
anyone see my mistake?
Code follows:
-
#!/usr/bin/perl
no strict;
no warnings;
require Net::DNS;
require Net::DNS::Resolver;
use strict;
use warnings;
my @EXISTING_DOMAINS = qw{
adelphia.net
akamai.com
apache.org
cingular.com
colorado.edu
comcast.net
doubleclick.com
ebay.com
gmx.net
google.com
intel.com
kernel.org
linux.org
mit.edu
motorola.com
msn.com
sourceforge.net
sun.com
w3.org
yahoo.com
};
my $res = Net::DNS::Resolver->new (
recurse => 0,
retry => 1,
retrans => 0,
dnsrch => 0,
defnames => 0,
tcp_timeout => 3,
udp_timeout => 3,
persistent_tcp => 1,
persistent_udp => 1
);
die unless defined $res;
for(;;) {
my @domains = @EXISTING_DOMAINS;
my $domain = splice(@domains, rand(@domains), 1);
print "trying '$domain'...\n";
lookup_ns($domain);
}
sub lookup_ns {
my ($self, $dom) = @_;
my $query = $res->search($dom, 'NS');
if ($query) {
foreach my $rr ($query->answer) {
print "type=", $rr->type, ", nsdname=", $rr->nsdname, "\n";
}
}
else {
print "ERROR! no query\n";
}
}
1;
Re: SA 3.01 eventually stops noticing DNSBLs
On Tuesday, February 22, 2005, 10:25:36 PM, Jay Levitt wrote: > Kelson wrote: >> Jay Levitt wrote: >> >>> I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. >>> At some point, SA seems to stop doing lookups on the DNSBLs; spam >>> gets through that is listed in multiple BLs; if I check manually with >>> spamassassin -t, it detects the BL entry, even if I run it moments >>> after the spam was received. >>> I don't see anything obvious in the logs. What can I do to >>> troubleshoot this? >> >> >> Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf >> on an upgrade. >> >> That happened to my server a while back -- We were just using >> /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was >> no sa-mimedefang.cf, so it created it with the defaults -- and the >> defaults disable DNSBLs. >> > Nope, that's not it. I've been throwing debug code in bit by bit. > (More accurately, I've been re-copying the dbg statements as "warns", > because while there's plenty of useful output, there are just too many > un-categorized dbg statements to leave debug enabled... sigh.) Looks > like every once in a while, the lookup_ns sanity-checks that SA does on > well-known domains are returning with zero NS records. Still not sure > why that happens yet, or exactly what is going on, but that does > understandably lead SA to disable DNSBL processing for a while. Hmm, that sounds like something that may deserve a bugzilla. Can anyone else replicate that behavior? Is your Net::DNS completely current and happy? Have you checked all of your: /etc/resolv.conf $HOME/.resolv.conf ./.resolv.conf for the user mimedefang or SA runs as to make sure they're all correct and all the name servers on them resolve the RBLs correctly? Also when you say "At some point, SA seems to stop doing lookups on the DNSBLs" what is the time scale? Does "At some point" mean at some times of day, after several months of operation and all the time now, for a few hours at a time, for every 6th message, etc.? Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
On Tuesday, February 22, 2005, 9:57:02 AM, Andy Jezierski wrote: > Kelson <[EMAIL PROTECTED]> wrote on 02/22/2005 11:30:46 AM: >> Jay Levitt wrote: >> > I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At >> > some point, SA seems to stop doing lookups on the DNSBLs; spam gets >> > through that is listed in multiple BLs; if I check manually with >> > spamassassin -t, it detects the BL entry, even if I run it moments > after >> > the spam was received. >> > I don't see anything obvious in the logs. What can I do to > troubleshoot >> > this? >> >> Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on >> an upgrade. >> >> That happened to my server a while back -- We were just using >> /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was >> no sa-mimedefang.cf, so it created it with the defaults -- and the >> defaults disable DNSBLs. > Could this be the same problem as the discussion in the "Spammer > Anti-SURBL tactic" thread? > Andy Possibly, but I think the "lots of URI" spams are still somewhat rare, so it should only be an occasional occurrence. It sounds like this thread "SA 3.01 eventually stops noticing DNSBLs" is more likely an installation or configuration issue for this particular system. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
Kelson wrote: Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on an upgrade. That happened to my server a while back -- We were just using /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was no sa-mimedefang.cf, so it created it with the defaults -- and the defaults disable DNSBLs. Nope, that's not it. I've been throwing debug code in bit by bit. (More accurately, I've been re-copying the dbg statements as "warns", because while there's plenty of useful output, there are just too many un-categorized dbg statements to leave debug enabled... sigh.) Looks like every once in a while, the lookup_ns sanity-checks that SA does on well-known domains are returning with zero NS records. Still not sure why that happens yet, or exactly what is going on, but that does understandably lead SA to disable DNSBL processing for a while. Jay
Re: SA 3.01 eventually stops noticing DNSBLs
On Tue, 22 Feb 2005, Andy Jezierski wrote:
> Kelson <[EMAIL PROTECTED]> wrote on 02/22/2005 11:30:46 AM:
>
> > Jay Levitt wrote:
> > > I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At
> > > some point, SA seems to stop doing lookups on the DNSBLs; spam gets
> > > through that is listed in multiple BLs; if I check manually with
> > > spamassassin -t, it detects the BL entry, even if I run it moments
> after
> > > the spam was received.
[snip..]
> > Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on
> > an upgrade.
> >
> > That happened to my server a while back -- We were just using
> > /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was
> > no sa-mimedefang.cf, so it created it with the defaults -- and the
> > defaults disable DNSBLs.
>
> Could this be the same problem as the discussion in the "Spammer
> Anti-SURBL tactic" thread?
No, the problem that I saw was caused by the contents of the message.
it didn't matter how I fed it to SA, SURBL did not 'fire' on the spammer
"payload" URL, eventho the site was in several lists.
When I first saw the spam I was puzzled and ran it thru SA with full
debugging to see what was going on.
( spamassassin -D rulesrun=255 < message.txt )
I saw lots of SURBL tests on the "decoy" URLs but none of the "payload".
I removed the CR characters from the "payload" URL, retested but still
no hits, removed most of the "decoy" URLs from the message, then a
retest did hit. So it was entirely due to the message content
exploiting two weaknesses in SA.
PS: if you do a full debug SA run, it's best to redirect the output to
a file for reading, it produces thousands of lines of output. ;)
--
Dave Funk University of Iowa
College of Engineering
319/335-5751 FAX: 319/384-0549 1256 Seamans Center
Sys_admin/Postmaster/cell_adminIowa City, IA 52242-1527
#include
Better is not better, 'standard' is better. B{
Re: SA 3.01 eventually stops noticing DNSBLs
Kelson <[EMAIL PROTECTED]> wrote on 02/22/2005 11:30:46 AM: > Jay Levitt wrote: > > I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At > > some point, SA seems to stop doing lookups on the DNSBLs; spam gets > > through that is listed in multiple BLs; if I check manually with > > spamassassin -t, it detects the BL entry, even if I run it moments after > > the spam was received. > > I don't see anything obvious in the logs. What can I do to troubleshoot > > this? > > Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on > an upgrade. > > That happened to my server a while back -- We were just using > /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was > no sa-mimedefang.cf, so it created it with the defaults -- and the > defaults disable DNSBLs. > > -- > Kelson Vibber > SpeedGate Communications Could this be the same problem as the discussion in the "Spammer Anti-SURBL tactic" thread? Andy
Re: SA 3.01 eventually stops noticing DNSBLs
Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Make sure MIMEDefang hasn't created a new /etc/mail/sa-mimedefang.cf on an upgrade. That happened to my server a while back -- We were just using /etc/mail/spamassassin/local.cf, and upgraded MD, and MD saw there was no sa-mimedefang.cf, so it created it with the defaults -- and the defaults disable DNSBLs. -- Kelson Vibber SpeedGate Communications
Re: SA 3.01 eventually stops noticing DNSBLs
On Saturday, February 19, 2005, 8:05:05 AM, Jay Levitt wrote: > Just checked.. there is only one resolv.conf on the system, in > /etc/resolv.conf, and it correctly points to my own machine, which runs > a caching named (actually caching for the world, authoritative for my > own domain). Note that the other two files mentioned are .resolv.conf, not resolv.conf: > >> /etc/resolv.conf > >> $HOME/.resolv.conf > >> ./.resolv.conf and that you'd want to check them for the specific user that SpamAssassin (spamd, etc.) runs as. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
On Saturday, February 19, 2005, 8:05:05 AM, Jay Levitt wrote: > It's important to note that the DNS lookups DO work for a while after > starting mimedefang; it's just at some point after days/weeks that it > stops trying (or stops succeeding). You may want to check that your name server *only* thinks it's authoritative for your own zones. If you try to primary or secondary zones you're not authoritative for, like an RBL zone for example, your name server may give odd results when it finally realizes that it's not authoritative. Not sure what else to suggest. Do you have any DNS caching or accelerating programs? Perhaps they're doing something similar. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: SA 3.01 eventually stops noticing DNSBLs
Jeff Chan wrote: On Friday, February 18, 2005, 8:35:35 PM, Jay Levitt wrote: I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. One thing to check is whether your name resolution is truly correct. Are you running Net::DNS 0.48? I was running 0.47; just upgraded to 0.48. Was there some known bug in 0.47 that could cause this? The Changes for 0.48 don't mention anything that looks relevant. Is it installed and upgraded in a consistent way (i.e. always rpms or always CPAN or always tarballs)? Using different upgrade methods can confuse things. Always CPAN. Did you see the recent thread about the various resolve.conf's used by Net::DNS? Are they all correct for the user SpamAssassin runs as? Just checked.. there is only one resolv.conf on the system, in /etc/resolv.conf, and it correctly points to my own machine, which runs a caching named (actually caching for the world, authoritative for my own domain). It's important to note that the DNS lookups DO work for a while after starting mimedefang; it's just at some point after days/weeks that it stops trying (or stops succeeding). Any tips as to where I could put debugging code? Should SA already be writing something to a log file? Jeff C.
Re: SA 3.01 eventually stops noticing DNSBLs
On Friday, February 18, 2005, 8:35:35 PM, Jay Levitt wrote: > I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At > some point, SA seems to stop doing lookups on the DNSBLs; spam gets > through that is listed in multiple BLs; if I check manually with > spamassassin -t, it detects the BL entry, even if I run it moments after > the spam was received. One thing to check is whether your name resolution is truly correct. Are you running Net::DNS 0.48? Is it installed and upgraded in a consistent way (i.e. always rpms or always CPAN or always tarballs)? Using different upgrade methods can confuse things. Did you see the recent thread about the various resolve.conf's used by Net::DNS? Are they all correct for the user SpamAssassin runs as? Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
SA 3.01 eventually stops noticing DNSBLs
I have SA 3.01 running under mimedefang 2.43 with sendmail 8.13.1. At some point, SA seems to stop doing lookups on the DNSBLs; spam gets through that is listed in multiple BLs; if I check manually with spamassassin -t, it detects the BL entry, even if I run it moments after the spam was received. I don't see anything obvious in the logs. What can I do to troubleshoot this? Jay Levitt
Re: SA 3.01 and BAYES probability too high
Loren Wilton wrote: For some days BAYES probabilty jump for almost all messages, i've BAYES_50 for almost all message including ham... What can be the reason ? Why do you think bayes 50 is wrong? BTW: Bayes_50 I would consider as ham. Bayes_50 means that Bayes doesn't know if the message is ham or spam, since it hasn't seen enough tokens yet to determine. So it would be wrong to consider a bayes_50 (or bayes_49 or Bayes_51) to be either ham OR spam. It is an "I don't know!" case. This is not a case of bayes scoring too high, but if anything it is scoring too low. "High" scores for bayes are 00 or 99, and "low" is 50. Offhand I can't think of anything that would cause Bayes to score all messages with bayes_50, other than a very corrupted bayes database. In fact, I'm a little surprised that bayes_50 is even showing up. Since that is the "oh nevermind" case I thought that (at least in 2.6x) it didn't even bother sticking that result in the message. Yes, i think my database was corrupted because the change appears suddenly... All my messages were scored between BAYES_50 and BAYES_99 How can i prevent bayes corruption ? I've two server sharring bayes files by NFS Thanks Guillaume
Re: SA 3.01 and BAYES probability too high
>> For some days BAYES probabilty jump for almost all messages, i've >> BAYES_50 for almost all message including ham... >> What can be the reason ? >> > Why do you think bayes 50 is wrong? BTW: Bayes_50 I would consider as ham. Bayes_50 means that Bayes doesn't know if the message is ham or spam, since it hasn't seen enough tokens yet to determine. So it would be wrong to consider a bayes_50 (or bayes_49 or Bayes_51) to be either ham OR spam. It is an "I don't know!" case. This is not a case of bayes scoring too high, but if anything it is scoring too low. "High" scores for bayes are 00 or 99, and "low" is 50. Offhand I can't think of anything that would cause Bayes to score all messages with bayes_50, other than a very corrupted bayes database. In fact, I'm a little surprised that bayes_50 is even showing up. Since that is the "oh nevermind" case I thought that (at least in 2.6x) it didn't even bother sticking that result in the message. Loren
Re: SA 3.01 and BAYES probability too high
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am Montag, 17. Januar 2005 15:34 schrieb Guillaume Urbejtel: > Hi All, > > For some days BAYES probabilty jump for almost all messages, i've > BAYES_50 for almost all message including ham... > What can be the reason ? > > Thanks for your help ! I have lost my magical I. Could you provide an example. Bayes depends on your messages. So: Why do you think bayes 50 is wrong? BTW: Bayes_50 I would consider as ham. Thomas - -- icq:133073900 http://www.t-arend.de -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.0 (GNU/Linux) iD8DBQFB7Bt2He2ZLU3NgHsRAgwyAJ9diXj52lKarKssHf1iCmj11zKr5ACdEc69 BUOupl6umAqpFP5ew+KV4Go= =imGW -END PGP SIGNATURE-
SA 3.01 and BAYES probability too high
Hi All, For some days BAYES probabilty jump for almost all messages, i've BAYES_50 for almost all message including ham... What can be the reason ? Thanks for your help !
Re: SA 3.01
> I suspect that its my setup the system passes any message it recieves > through and SA and passes it onto > the next mailserver. I think the way im running child processes is part > of the problem, I'm running 50 and > having to get the system to restart each child after processes 2 > messages to keep the memory usage down. Well, the obvious suggestion is to cut the max children down to about 10-12, and then maybe take max conn up to 10-20 or so. I'd expect that to improve matters. Further tweaking woudl probably be warrented after seeing how that worked. Loren
RE: SA 3.01
Tweak it down to something like -m 6 and see what happens. There was a thread about this about 6 weeks ago. Search for "SA 3.01 memory" in the archives. After tweaking it, watch the processes to see if any grow. Tweak it up from there. My spamd processes take about 45mb each. For you that would be around 2.2gb. That might be why you are running out of memory. With 2gb, 2.2gb dedicated to SA leaves about -200mb for the OS. I bet your swap is about 400mb about 20 minutes after startup... My math says you should be good around -m 20 through. Just play with it. -m 50 per second would be 4.2 million messages a day... Gary > -Original Message- > From: Gavin Pearce [mailto:[EMAIL PROTECTED] > Sent: Wednesday, December 15, 2004 2:15 PM > To: Rob MacGregor; users@spamassassin.apache.org > Subject: Re: SA 3.01 > > Here my spamd setup > > exec /usr/local/bin/spamd -H "$HOMEDIR" \ > --username=qmailq \ > --max-conn-per-child=2 \ > -A 127.0.0.1 -m 50 2>&1 >
Re: SA 3.01
Rob MacGregor wrote: On Wed, 15 Dec 2004 18:12:06 +, Gavin Pearce <[EMAIL PROTECTED]> wrote: We have presently upgraded the server that we run SA on Dual Operton 2.2Ghz 2GB DDR Memory SCSI U320 Raid5 Array Running Freebsd 5.3 Qmail Spammassassin 3.01 - Standard conf And we have had nothing but problems with it, it just chews up all its memory, till it runs out and I have to reboot to correct it. Anyone any thoughts. I've got a FreeBSD 5.3 box running SA out of the ports (along with Sendmail, MIMEDefang and ClamAV) and haven't seen any memory usage problems. The box only has 512 MB of RAM, so I'd expect to hit them sooner :) You're sure that it's SA that's the problem? What are you using to track memory usage? I suspect that its my setup the system passes any message it recieves through and SA and passes it onto the next mailserver. I think the way im running child processes is part of the problem, I'm running 50 and having to get the system to restart each child after processes 2 messages to keep the memory usage down. Here my spamd setup exec /usr/local/bin/spamd -H "$HOMEDIR" \ --username=qmailq \ --max-conn-per-child=2 \ -A 127.0.0.1 -m 50 2>&1
Re: SA 3.01
On Wed, 15 Dec 2004 18:12:06 +, Gavin Pearce <[EMAIL PROTECTED]> wrote: > We have presently upgraded the server that we run SA on > > Dual Operton 2.2Ghz > 2GB DDR Memory > SCSI U320 Raid5 Array > > Running > > Freebsd 5.3 > Qmail > Spammassassin 3.01 - Standard conf > > And we have had nothing but problems with it, it just chews up all its > memory, till it runs out and I have to reboot to correct it. Anyone any > thoughts. I've got a FreeBSD 5.3 box running SA out of the ports (along with Sendmail, MIMEDefang and ClamAV) and haven't seen any memory usage problems. The box only has 512 MB of RAM, so I'd expect to hit them sooner :) You're sure that it's SA that's the problem? What are you using to track memory usage? -- Please keep list traffic on the list. Rob MacGregor Whoever fights monsters should see to it that in the process he doesn't become a monster. Friedrich Nietzsche
SA 3.01
We have presently upgraded the server that we run SA on Dual Operton 2.2Ghz 2GB DDR Memory SCSI U320 Raid5 Array Running Freebsd 5.3 Qmail Spammassassin 3.01 - Standard conf And we have had nothing but problems with it, it just chews up all its memory, till it runs out and I have to reboot to correct it. Anyone any thoughts.
Re: SA 3.01 + DCC + Pyzor
It runs as root and has all the permissions.
I have installed several SuSE9.2 mailservers now and in every mail log i see
these error messages.
I just don't understand that there isn't a patch or workaround to solve this.
I can't believe that i am the only SuSE9.2 professional mailserver user.
Any other clues?
Grz. Johan
Op maandag 15 november 2004 22:53, schreef [EMAIL PROTECTED]:
> What user is SA running as?
> What permissions does that user have to the temporary directory where SA
> spools the message body? (usually /tmp)
>
> Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
> Hispanic Business Inc./HireDiversity.com Software Engineer
> perl -e"map{y/a-z/l-za-k/;print}shift" "Jjhi pcdiwtg Ptga wprztg,"
Heads up! SuSE YOU update broke SA 3.01
Just passing this along so you don't have to kill 2 days trying to figure out why SA suddenly stopped doing anything - I foolishly allowed SuSE auto-update (YOU) to update my Spamassassin. It (in theory) installed version 3.01 (which was already installed and working perfectly). Shortly after, I started receiving TONS of spam. SA-Learn wasn't learning, etc. I reinstalled from CPAN - and everything seems to be working again. I don't know what they broke, but they broke it throughly. Mike- -- If you can keep your head while those around you are losing theirs... You may have a great career as a network administrator ahead! -- Please note - Due to the intense volume of spam, we have installed site-wide spam filters at catherders.com. If email from you bounces, try non-HTML, non-encoded, non-attachments,
Heads up! SuSE YOU update broke SA 3.01
Just passing this along so you don't have to kill 2 days trying to figure out why SA suddenly stopped doing anything - I foolishly allowed SuSE auto-update (YOU) to update my Spamassassin. It (in theory) installed version 3.01 (which was already installed and working perfectly). Shortly after, I started receiving TONS of spam. SA-Learn wasn't learning, etc. I reinstalled from CPAN - and everything seems to be working again. I don't know what they broke, but they broke it throughly. Mike- -- If you can keep your head while those around you are losing theirs... You may have a great career as a network administrator ahead! -- Please note - Due to the intense volume of spam, we have installed site-wide spam filters at catherders.com. If email from you bounces, try non-HTML, non-encoded, non-attachments,
Re: SA 3.01 DCC and Pyzor
On Wednesday 24 November 2004 05:45 pm, Chris wrote: Don't even ask me what I did, because I could swear I had my local.cf setup like this before when it wasn't working, but its working now: add_header all DCC _DCCB_ _DCCR_ add_header all Pyzor _PYZOR_ X-Spam-DCC: dcc3mcgill cpollock 1275; Body=many Fuz1=many Fuz2=many X-Spam-Pyzor: Reported 630 times. Thanks Matt for your assistance -- Chris Registered Linux User 283774 http://counter.li.org 10:09pm up 21 days, 2:37, 2 users, load average: 0.27, 0.19, 0.11 A bird in the bush usually has a friend in there with him.
Re: SA 3.01 DCC and Pyzor
On Wednesday 24 November 2004 01:02 pm, Chris wrote: > On Monday 22 November 2004 11:59 pm, Matt Kettler wrote: > > So something like this in your local.cf would do it: > > > > clear_report > > report this is spam > > report _SUMMARY_ > > report DCC results are _DCCR_ > > report DCC brand is _DCCB_ > > report autolearn status is _AUTOLEARN_ > > report relays- trusted _RELAYSTRUSTED_ untrusted: _RELAYSUNTRUSTED_ > I've sort of got it working now - however, not the way I believe it should be. I've added this line in my local.cf: add_header all header_name "DCC:" _DCCB_ _DCCR_ \n which gives me: X-Spam-header_name: "DCC:" neonova cpollock 1127; Body=1 Fuz1=82 Fuz2=many however I can't get rid of the -header_name entry, if I remove it and have just add_header "DCC" --lint gives me an error. If I add the line: add_header all header_name PYZOR: _PYZOR_ all thats shown is: X-Spam-header_name: PYZOR: Reported 612 times. while the DCC results are left off. -- Chris Registered Linux User 283774 http://counter.li.org 5:19pm up 20 days, 21:47, 2 users, load average: 0.32, 0.22, 0.13 Laugh and the world thinks you're an idiot.
Re: SA 3.01 DCC and Pyzor
On Monday 22 November 2004 11:59 pm, Matt Kettler wrote: > So something like this in your local.cf would do it: > > clear_report > report this is spam > report _SUMMARY_ > report DCC results are _DCCR_ > report DCC brand is _DCCB_ > report autolearn status is _AUTOLEARN_ > report relays- trusted _RELAYSTRUSTED_ untrusted: _RELAYSUNTRUSTED_ > Don't know whats wrong Matt, no matter what I try I get nothing like what I used to get with 2.63: X-Spam-DCC: EATSERVER: cpollock 1166; Body=1 Fuz1=1 Fuz2=2 X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on cpollock X-Spam-Pyzor: Reported 0 times. I've changed my local.cf as below. Guess I'm totally lost or not meant to see the dcc/pyzor results. use_dcc 1 dcc_timeout 10 dcc_body_max99 dcc_fuz1_max99 dcc_fuz2_max99 report DCC Results are _DCCR_ report DCC Brand is _DCCB_ use_pyzor 1 pyzor_path /usr/bin/pyzor pyzor_max 10 report PYZOR Results are _PYZOR_ -- Chris Registered Linux User 283774 http://counter.li.org 11:57am up 20 days, 16:25, 2 users, load average: 0.21, 0.46, 0.41 Digital circuits are made from analog parts. -- Don Vonada
Re: SA 3.01 DCC and Pyzor
At 09:00 PM 11/22/2004 -0600, Chris wrote: With 2.63 and DCC active I'd see a body #, fuz1 # and fuz2 #, I no longer see that. I do have the dcc_body_max; _fuz1_max and _fuz2_max set to the suggested 99 in my local.cf. I also used to see the number of times a msg had been reported to pyzor, that too seems to be gone. Although SA is showing that DCC, Razor, and Pyzor are all being checked, has this been removed from 3.0.1? It's off by default, but you can add it by adding _DCCR_ to your report template See: http://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_SpamAssassin_Conf.html#template_tags Also see clear_report and report: http://spamassassin.apache.org/full/3.0.x/dist/doc/Mail_SpamAssassin_Conf.html#miscellaneous_options So something like this in your local.cf would do it: clear_report report this is spam report _SUMMARY_ report DCC results are _DCCR_ report DCC brand is _DCCB_ report autolearn status is _AUTOLEARN_ report relays- trusted _RELAYSTRUSTED_ untrusted: _RELAYSUNTRUSTED_ elaborate the wording to suit, or copy the one from /usr/share/spamassassin/10_misc.cf. However, I'd avoid touching 10_misc.cf, as it will be oblitterated if you upgrade.
SA 3.01 DCC and Pyzor
With 2.63 and DCC active I'd see a body #, fuz1 # and fuz2 #, I no longer see that. I do have the dcc_body_max; _fuz1_max and _fuz2_max set to the suggested 99 in my local.cf. I also used to see the number of times a msg had been reported to pyzor, that too seems to be gone. Although SA is showing that DCC, Razor, and Pyzor are all being checked, has this been removed from 3.0.1? -- Chris Registered Linux User 283774 http://counter.li.org 8:58pm up 19 days, 1:26, 1 user, load average: 0.84, 0.74, 0.64 If you want divine justice, die. -- Nick Seldon Live - From Virgin Radio UK The Rolling Stones - Sympathy for the devil
RE: SA 3.01 + DCC + Pyzor
Johan Barelds wrote:
> I *know* that dcc isn't broken.
> It only doesn't work when called from SA.
> In my case it isn't occasional:
> --
> Nov 15 16:39:49 beast dccproc[1791]: missing message body; fatal error
> --
> I happens not-stop.
> It looks to me that SA doesn't send the message body to dcc.
> Anyone any clues?
What user is SA running as?
What permissions does that user have to the temporary directory where SA spools
the message body? (usually /tmp)
Matthew.van.Eerde (at) hbinc.com 805.964.4554 x902
Hispanic Business Inc./HireDiversity.com Software Engineer
perl -e"map{y/a-z/l-za-k/;print}shift" "Jjhi pcdiwtg Ptga wprztg,"
Re: SA 3.01 + DCC + Pyzor
Op maandag 15 november 2004 16:07, schreef D.J. Fan: > I get "missing message body; fatal error" on occasion. It is > my guess this means there is no text in the body of the message. > I don't think it means DCC is broken. I *know* that dcc isn't broken. It only doesn't work when called from SA. In my case it isn't occasional: -- Nov 15 16:27:44 beast dccproc[28963]: missing message body; fatal error Nov 15 16:31:01 beast dccproc[29636]: missing message body; fatal error Nov 15 16:39:49 beast dccproc[1791]: missing message body; fatal error Nov 15 16:43:52 beast dccproc[4044]: missing message body; fatal error Nov 15 16:46:46 beast dccproc[5186]: missing message body; fatal error Nov 15 16:48:18 beast dccproc[6466]: missing message body; fatal error Nov 15 17:03:56 beast dccproc[13719]: missing message body; fatal error Nov 15 17:03:57 beast dccproc[13726]: missing message body; fatal error Nov 15 17:03:58 beast dccproc[13732]: missing message body; fatal error Nov 15 17:06:14 beast dccproc[14279]: missing message body; fatal error Nov 15 17:08:32 beast dccproc[16140]: missing message body; fatal error Nov 15 17:08:32 beast dccproc[16141]: missing message body; fatal error Nov 15 17:13:21 beast dccproc[18538]: missing message body; fatal error Nov 15 17:16:28 beast dccproc[19461]: missing message body; fatal error Nov 15 17:18:12 beast dccproc[20943]: missing message body; fatal error Nov 15 17:18:16 beast dccproc[20953]: missing message body; fatal error Nov 15 17:18:16 beast dccproc[20952]: missing message body; fatal error Nov 15 17:19:34 beast dccproc[21129]: missing message body; fatal error Nov 15 17:29:05 beast dccproc[25884]: missing message body; fatal error Nov 15 17:29:38 beast dccproc[25900]: missing message body; fatal error Nov 15 17:29:38 beast dccproc[25902]: missing message body; fatal error Nov 15 17:33:40 beast dccproc[28136]: missing message body; fatal error Nov 15 17:34:11 beast dccproc[28303]: missing message body; fatal error Nov 15 17:38:35 beast dccproc[30524]: missing message body; fatal error Nov 15 17:42:10 beast dccproc[32460]: missing message body; fatal error Nov 15 17:42:22 beast dccproc[32473]: missing message body; fatal error Nov 15 17:57:39 beast dccproc[7228]: missing message body; fatal error Nov 15 18:15:12 beast dccproc[15108]: missing message body; fatal error Nov 15 18:21:43 beast dccproc[18225]: missing message body; fatal error Nov 15 18:23:30 beast dccproc[19482]: missing message body; fatal error -- I happens not-stop. It looks to me that SA doesn't send the message body to dcc. Anyone any clues? -- Kind Regards / Met vriendelijke groet, Johan Barelds Good-IT! Tel.+31(0)70-3965230Strijplaan 320 Mob.+31(0)6-542537502285 HZ Rijswijk(ZH) [EMAIL PROTECTED] http://www.good-it.com
SA 3.01 + DCC + Pyzor
I noticed that DCC and Pyzor are broken if beeing called from SA. (Nov 14 17:00:07 beast dccproc[4795]: missing message body; fatal error) I get "missing message body; fatal error" on occasion. It is my guess this means there is no text in the body of the message. I don't think it means DCC is broken. For the Pyzor issue, this may be of help: https://sourceforge.net/mailarchive/forum.php?thread_id=5955026&forum_id=8711 _ Express yourself instantly with MSN Messenger! Download today - it's FREE! http://messenger.msn.click-url.com/go/onm00200471ave/direct/01/
Re: SA 3.01 + DCC + Pyzor
At 05:06 PM 11/14/2004 +0100, Johan Barelds wrote: At this moment i use the SuSE 9.2 distro. I noticed that DCC and Pyzor are broken if beeing called from SA. (Nov 14 17:00:07 beast dccproc[4795]: missing message body; fatal error) I also read the docu from SA3.01 and found this: -- There is an issue if you run spamd using the standard perl installation on Mac OS X and certain *BSD-flavored UNIX platforms. spamd will change effective uid to the user calling spamd for security reasons. Before calling out to any external programs (DCC and Pyzor, as of 3.0.0,) spamd will fork() and change the real uid to the same as the effective uid. Unfortunately, the default perl in at least Mac OS X, does not allow perl programs to change the real uid so for security reasons the spamd child will die. To fix this issue, either disable the DCC and Pyzor rules, or install a different version of perl which supports setuid() calls. -- Question: is this the cause that DCC and Pyzor won't run wit SA3.01 on SuSE9.2? Highly doubtful. SuSE is Linux-based. It is not a BSD-flavored Unix platform. (Anything based on the Linux kernel is inherently not based on a BSD kernel. If it were, it would cease to be Linux.)
SA 3.01 + DCC + Pyzor
Hi all, I have been upgrading to SA 3.01 this weekend and i am impressed by the amount of spam it stops! Great job! At this moment i use the SuSE 9.2 distro. I noticed that DCC and Pyzor are broken if beeing called from SA. (Nov 14 17:00:07 beast dccproc[4795]: missing message body; fatal error) I also read the docu from SA3.01 and found this: -- There is an issue if you run spamd using the standard perl installation on Mac OS X and certain *BSD-flavored UNIX platforms. spamd will change effective uid to the user calling spamd for security reasons. Before calling out to any external programs (DCC and Pyzor, as of 3.0.0,) spamd will fork() and change the real uid to the same as the effective uid. Unfortunately, the default perl in at least Mac OS X, does not allow perl programs to change the real uid so for security reasons the spamd child will die. To fix this issue, either disable the DCC and Pyzor rules, or install a different version of perl which supports setuid() calls. -- Question: is this the cause that DCC and Pyzor won't run wit SA3.01 on SuSE9.2? If this is so i can hardly believe that i have to change my version of perl in order to get it working! Isn't there some sort of patch coming up to solve this? Thanks for your replies on this matter. -- Kind Regards / Met vriendelijke groet, Johan Barelds Good-IT! Tel.+31(0)70-3965230Strijplaan 320 Mob.+31(0)6-542537502285 HZ Rijswijk(ZH) [EMAIL PROTECTED] http://www.good-it.com
Re: SA 3.01 scoring very low
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sean Doherty writes: > On Thu, 2004-11-04 at 15:04, Dave Goodrich wrote: > > > Check out trusted_network section of Mail::SpamAssassin::Conf > > > i.e no RBL tests on trusted networks. > > "If you're running with DNS checks enabled, SpamAssassin includes code > > to infer your trusted networks on the fly, so this may not be necessary. > > (Thanks to Scott Banister and Andrew Flury for the inspiration for this > > algorithm.) This inference works as follows:" > > > > This seems backwards to me. If a user does nothing, then his network > > will be considered trusted by default? We are an ISP, and SA is running > > on our toasters. I don't want any machine trusted as that leaves a door > > open for my smtp relay users (viruses, trojans, just bad folks) to spam > > local users. > > > > JMHO, but shouldn't all networks be considered untrusted unless a user > > specifies otherwise? > > I got to agree with you there - especially given that the inference > algorithm doesn't work in every environment. the idea is that an ISP *will* take the time to set that setting. ;) - --j. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.4 (GNU/Linux) Comment: Exmh CVS iD4DBQFBimZxMJF5cimLx9ARAm7VAJdjojaKXz6t++f5BwK+ocf0jT5cAKChSgLF 7Wrsz2oohTyTjYLaJktIuA== =LLyQ -END PGP SIGNATURE-
Re: SA 3.01 scoring very low
At 11:14 AM 11/4/2004, Jim Maul wrote: While i agree that trusting no one doesnt really solve the problem, I dont believe it is "just as bad" as trusting everyone. Trusting everyone stops other rules from firing and adds atleast -2.something to every message. This seems far worse than trusting no one and breaking whitelist_from_rcvd While I'll concede it may not be "just as bad" it's still much worse than you think. LOTS of rules in SA depend on trust. Not just whitelist_from_rcvd and ALL_TRUSTED. All of these rules are broken by a broken trust path, some in ways that cause FPs, others just missing out on score: HELO_DYNAMIC_* FAKE_HELO_MAIL_COM_DOM RCVD_IN_BSP_* MSGID_FROM_MTA_ID FORGED_RCVD_* AWL trust plays into "notfirsthop" as well, so all these DNSBLs get broken: RCVD_IN_NJABL_DUL RCVD_IN_SORBS_DUL RCVD_IN_XBL RCVD_IN_DSBL RCVD_IN_MAPS_DUL
Re: SA 3.01 scoring very low
Matt Kettler wrote: At 10:17 AM 11/4/2004, Sean Doherty wrote: > JMHO, but shouldn't all networks be considered untrusted unless a user > specifies otherwise? I got to agree with you there - especially given that the inference algorithm doesn't work in every environment. Unfortunately this only solves one aspect of the problem. SA NEEDS to have the correct trust path. Trusting nobody is just as bad as trusting everyone. Trusting nobody breaks whitelist_from_rcvd, for example. This is all becoming very confusing about what effect the trusted networks code has on the rest of SA. Possibly I have not read the conf pages correctly. "internal_networks ip.add.re.ss[/mask] ... (default: none) If neither trusted_networks or internal_networks is set, no addresses will be considered local; in other words, any relays past the machine where SpamAssassin is running will be considered external." And trusted? "whitelist_from_rcvd [EMAIL PROTECTED] sourceforge.net Note that this requires that internal_networks be correct. For ^^ simple cases, it will be, but for a complex network, or running with DNS checks off or with -L, you may get better results by setting that parameter." I'm confused here, if I set no trust params, then all networks are trusted by default. But if I trust no networks, then I cannot use whitelist_from_rcvd to define a trusted relay? To me that says, in order to define a trusted relay via whitelist_from_rcvd, I first must trust ALL relays, or put all the relays I have in whitelist_from_rcvd into my trusted networks as well. DAve -- Systems Administrator http://www.tls.net Get rid of Unwanted Emails...get TLS Spam Blocker!
Re: SA 3.01 scoring very low
Matt Kettler wrote: At 10:17 AM 11/4/2004, Sean Doherty wrote: > JMHO, but shouldn't all networks be considered untrusted unless a user > specifies otherwise? I got to agree with you there - especially given that the inference algorithm doesn't work in every environment. Unfortunately this only solves one aspect of the problem. SA NEEDS to have the correct trust path. Trusting nobody is just as bad as trusting everyone. Trusting nobody breaks whitelist_from_rcvd, for example. While i agree that trusting no one doesnt really solve the problem, I dont believe it is "just as bad" as trusting everyone. Trusting everyone stops other rules from firing and adds atleast -2.something to every message. This seems far worse than trusting no one and breaking whitelist_from_rcvd. -Jim
Re: SA 3.01 scoring very low
At 10:17 AM 11/4/2004, Sean Doherty wrote: > JMHO, but shouldn't all networks be considered untrusted unless a user > specifies otherwise? I got to agree with you there - especially given that the inference algorithm doesn't work in every environment. Unfortunately this only solves one aspect of the problem. SA NEEDS to have the correct trust path. Trusting nobody is just as bad as trusting everyone. Trusting nobody breaks whitelist_from_rcvd, for example.
Re: SA 3.01 scoring very low
On Thu, 2004-11-04 at 15:04, Dave Goodrich wrote: > > Check out trusted_network section of Mail::SpamAssassin::Conf > > i.e no RBL tests on trusted networks. > "If you're running with DNS checks enabled, SpamAssassin includes code > to infer your trusted networks on the fly, so this may not be necessary. > (Thanks to Scott Banister and Andrew Flury for the inspiration for this > algorithm.) This inference works as follows:" > > This seems backwards to me. If a user does nothing, then his network > will be considered trusted by default? We are an ISP, and SA is running > on our toasters. I don't want any machine trusted as that leaves a door > open for my smtp relay users (viruses, trojans, just bad folks) to spam > local users. > > JMHO, but shouldn't all networks be considered untrusted unless a user > specifies otherwise? I got to agree with you there - especially given that the inference algorithm doesn't work in every environment. - Sean
Re: SA 3.01 scoring very low
Thanks everyone, testing with several messages and comparing to 2.64 scores looks good now. Three issues, 1) My test message was munged and SA had problems parsing the headers. Used unmangled messages and SA parsed them fine. 2) Set trusted networks to 127.0.0.1, so no network is trusted. 3) set "dns_available yes", this stopped the testing of dns availability, while still allowing dns tests themselves to run. Of note, setting "skip_rbl_checks 1" does not stop SURBL tests, which is good. Just stops the rbl checks for smtp connections. DAve Matt Kettler wrote: At 09:54 AM 11/4/2004 -0500, Dave Goodrich wrote: Yes I just submitted a bug on the matter.. Currently ALL_TRUSTED fires whenever there are no untrusted relays detected.. However, it fails to check that any trusted relays exist... I opened this bug to suggest a fix for ALL_TRUSTED: http://bugzilla.spamassassin.org/show_bug.cgi?id=3949 However, the Received: path parsing bug is something I leave up to Dave to file. No need, I rechecked my test message and it had some formatting problems from being transfered off my workstation (Thunderbird) and onto the SA box. I grabbed a couple other messages right out of the Maildir and they parsed fine. I believe the issue with the headers was of my making, not a SA problem Fair enough, thanks for the follow-up. I still think it's worth fixing ALL_TRUSTED just in case. There's at least one valid open bug regarding Received: formats.. http://bugzilla.spamassassin.org/show_bug.cgi?id=3600 And many others are possible, so it's definitely worth the preventative measures. -- Systems Administrator http://www.tls.net Get rid of Unwanted Emails...get TLS Spam Blocker!
Re: SA 3.01 scoring very low
Sean Doherty wrote: On Thu, 2004-11-04 at 14:14, Dave Goodrich wrote: Sean Doherty wrote: I will look into that, I didn't set it as I want no network to be trusted. I'll reread what I can find on that. Just set trusted_network 127.0.0.1 Yes, this fixed it. Since you hit ALL_TRUSTED certain other DNS based tests are not run. Eh? Where do I find this out? Check out trusted_network section of Mail::SpamAssassin::Conf i.e no RBL tests on trusted networks. "If you're running with DNS checks enabled, SpamAssassin includes code to infer your trusted networks on the fly, so this may not be necessary. (Thanks to Scott Banister and Andrew Flury for the inspiration for this algorithm.) This inference works as follows:" This seems backwards to me. If a user does nothing, then his network will be considered trusted by default? We are an ISP, and SA is running on our toasters. I don't want any machine trusted as that leaves a door open for my smtp relay users (viruses, trojans, just bad folks) to spam local users. JMHO, but shouldn't all networks be considered untrusted unless a user specifies otherwise? DAve -- Systems Administrator http://www.tls.net Get rid of Unwanted Emails...get TLS Spam Blocker!
Re: SA 3.01 scoring very low
At 09:54 AM 11/4/2004 -0500, Dave Goodrich wrote: Yes I just submitted a bug on the matter.. Currently ALL_TRUSTED fires whenever there are no untrusted relays detected.. However, it fails to check that any trusted relays exist... I opened this bug to suggest a fix for ALL_TRUSTED: http://bugzilla.spamassassin.org/show_bug.cgi?id=3949 However, the Received: path parsing bug is something I leave up to Dave to file. No need, I rechecked my test message and it had some formatting problems from being transfered off my workstation (Thunderbird) and onto the SA box. I grabbed a couple other messages right out of the Maildir and they parsed fine. I believe the issue with the headers was of my making, not a SA problem Fair enough, thanks for the follow-up. I still think it's worth fixing ALL_TRUSTED just in case. There's at least one valid open bug regarding Received: formats.. http://bugzilla.spamassassin.org/show_bug.cgi?id=3600 And many others are possible, so it's definitely worth the preventative measures.
Re: SA 3.01 scoring very low
Matt Kettler wrote: At 02:19 PM 11/4/2004 +, Sean Doherty wrote: Matt, does this mean that even if trusted_networks is set in local.cf, SpamAssassin will fire the ALL_TRUSTED rule even if it can't parse the received headers? i.e. Since there are no parsable received headers, SA will assume that all must have been trusted? Yes I just submitted a bug on the matter.. Currently ALL_TRUSTED fires whenever there are no untrusted relays detected.. However, it fails to check that any trusted relays exist... I opened this bug to suggest a fix for ALL_TRUSTED: http://bugzilla.spamassassin.org/show_bug.cgi?id=3949 However, the Received: path parsing bug is something I leave up to Dave to file. No need, I rechecked my test message and it had some formatting problems from being transfered off my workstation (Thunderbird) and onto the SA box. I grabbed a couple other messages right out of the Maildir and they parsed fine. I believe the issue with the headers was of my making, not a SA problem. DAve -- Systems Administrator http://www.tls.net Get rid of Unwanted Emails...get TLS Spam Blocker!
Re: SA 3.01 scoring very low
At 02:19 PM 11/4/2004 +, Sean Doherty wrote: Matt, does this mean that even if trusted_networks is set in local.cf, SpamAssassin will fire the ALL_TRUSTED rule even if it can't parse the received headers? i.e. Since there are no parsable received headers, SA will assume that all must have been trusted? Yes I just submitted a bug on the matter.. Currently ALL_TRUSTED fires whenever there are no untrusted relays detected.. However, it fails to check that any trusted relays exist... I opened this bug to suggest a fix for ALL_TRUSTED: http://bugzilla.spamassassin.org/show_bug.cgi?id=3949 However, the Received: path parsing bug is something I leave up to Dave to file. Really mis-parsed Received: headers is a serious bug, the fix to ALL_TRUSTED is just damage control.
Re: SA 3.01 scoring very low
On Thu, 2004-11-04 at 14:14, Dave Goodrich wrote: > Sean Doherty wrote: > > On Wed, 2004-11-03 at 21:40, Dave Goodrich wrote: > > > >>Good afternoon, > >> > >>I just finished testing an upgrade of SA to 3.01 and my scores fell > >>through the floor. Read the docs, tried to use the Wiki, followed > >>everyone else's upgrade on the list. Not sure just what went wrong. > > > > > >>X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on avhost.tls.net > >>X-Spam-Status: No, score=0.6 required=5.0 tests=ALL_TRUSTED,DRUGS_ERECTILE, > >> FROM_NO_LOWER,INVALID_DATE,MISSING_SUBJECT,RM_hm_EmtyMsgid > >> autolearn=disabled version=3.0.1 > > > > > > You need to specify trusted_networks in local.cf, otherwise > > you're going to continue to hit the ALL_TRUSTED rule which can > > *decrease* your score by up to -3.3. If you don't specify > > trusted_networks then SpamAssassin infers what your trusted > > networks are - and the inference algorithm may not always get > > the correct result. For instance if your mail relay/server is > > on a private network and NATed thru a firewall, then the > > algorithm may infer incorrectly that the connecting mail server > > is trusted. i.e. the algorithm assumes that since you're a > > private address, then the next hop server must belong to you > > since your MX must be public. However it does not take NAT > > into account. Setting trusted_networks appropriately will solve > > this issue (I don't think SA 2.64 has the ALL_TRUSTED rule - or > > at least it scores low). > I will look into that, I didn't set it as I want no network to be > trusted. I'll reread what I can find on that. Just set trusted_network 127.0.0.1 > > > > Since you hit ALL_TRUSTED certain other DNS based tests are not > > run. > Eh? Where do I find this out? Check out trusted_network section of Mail::SpamAssassin::Conf i.e no RBL tests on trusted networks. > I don't want any networks trusted, infered or otherwise. So I left > trusted_networks and internal_networks both blank. My understanding is that if unset trusted_networks will be infered. Setting it to the loopback address and/or the host IP address will prevent this. > > Also skip_rbl_checks will do just that. > Umm I don't follow you there, are you saying skip_rbl_checks will skip > SURBL? Because if it does, I'll need to go back to 2.64. No. Just pointing out that no RBL tests will not be run. Also, Matt Kettler pointed out in this thread that reason for the ALL_TRUSTED firing may not be entirely related invalid inference of trust, but because the Received headers had unknown format in the debug output. - Sean
Re: {SPAM} SA 3.01 scoring very low
On Wed, 2004-11-03 at 21:52, Matt Kettler wrote: > At 04:40 PM 11/3/2004, Dave Goodrich wrote: > >Good afternoon, > > > >I just finished testing an upgrade of SA to 3.01 and my scores fell > >through the floor. Read the docs, tried to use the Wiki, followed everyone > >else's upgrade on the list. Not sure just what went wrong. > > > >DAve > > > >Here is a sample output of spamassassin -D < test_spam (a known spam that > >had been caught and scored as follows, > > > >debug: received-header: unknown format: > >debug: received-header: unknown format: > >debug: received-header: unknown format: > >debug: received-header: unknown format: > > > > There's the cause of your problem.. SA is having problems parsing your > received headers. > > As a result, SA is failing to properly detect a trust path, and is > triggering ALL_TRUSTED, which should never happen for outside mail. > In the short term, force ALL_TRUSTED to 0 Matt, does this mean that even if trusted_networks is set in local.cf, SpamAssassin will fire the ALL_TRUSTED rule even if it can't parse the received headers? i.e. Since there are no parsable received headers, SA will assume that all must have been trusted? Seems a bit aggressive to me... - Sean
Re: SA 3.01 scoring very low
Sean Doherty wrote: On Wed, 2004-11-03 at 21:40, Dave Goodrich wrote: Good afternoon, I just finished testing an upgrade of SA to 3.01 and my scores fell through the floor. Read the docs, tried to use the Wiki, followed everyone else's upgrade on the list. Not sure just what went wrong. X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on avhost.tls.net X-Spam-Status: No, score=0.6 required=5.0 tests=ALL_TRUSTED,DRUGS_ERECTILE, FROM_NO_LOWER,INVALID_DATE,MISSING_SUBJECT,RM_hm_EmtyMsgid autolearn=disabled version=3.0.1 You need to specify trusted_networks in local.cf, otherwise you're going to continue to hit the ALL_TRUSTED rule which can *decrease* your score by up to -3.3. If you don't specify trusted_networks then SpamAssassin infers what your trusted networks are - and the inference algorithm may not always get the correct result. For instance if your mail relay/server is on a private network and NATed thru a firewall, then the algorithm may infer incorrectly that the connecting mail server is trusted. i.e. the algorithm assumes that since you're a private address, then the next hop server must belong to you since your MX must be public. However it does not take NAT into account. Setting trusted_networks appropriately will solve this issue (I don't think SA 2.64 has the ALL_TRUSTED rule - or at least it scores low). I will look into that, I didn't set it as I want no network to be trusted. I'll reread what I can find on that. Since you hit ALL_TRUSTED certain other DNS based tests are not run. Eh? Where do I find this out? Also is dns unavailable (dns_available no)? This may explain why you're not getting SURBL hits (which you should if dns is fully operational). I marked DNS unavailable as I don't want the DNS check, I do want DNS tests run, but only SURBL. Rereading it I think it was too late in the evening, I need to set "dns_available yes" to stop the dns testing, but still allow dns tests to run. My choice for leaving trusted_networks blank was this; "If trusted_networks is not set and internal_networks is, the value of internal_networks will be used for this parameter. If you're running with DNS checks enabled, SpamAssassin includes code to infer your trusted networks on the fly, so this may not be necessary." I don't want any networks trusted, infered or otherwise. So I left trusted_networks and internal_networks both blank. > Also skip_rbl_checks will do just that. Umm I don't follow you there, are you saying skip_rbl_checks will skip SURBL? Because if it does, I'll need to go back to 2.64. "By default, SpamAssassin will run RBL checks. If your ISP already does this for you, set this to 1." Thanks, DAve -- Systems Administrator http://www.tls.net Get rid of Unwanted Emails...get TLS Spam Blocker!
Re: SA 3.01 scoring very low
On Wed, 2004-11-03 at 21:40, Dave Goodrich wrote: > Good afternoon, > > I just finished testing an upgrade of SA to 3.01 and my scores fell > through the floor. Read the docs, tried to use the Wiki, followed > everyone else's upgrade on the list. Not sure just what went wrong. > X-Spam-Checker-Version: SpamAssassin 3.0.1 (2004-10-22) on avhost.tls.net > X-Spam-Status: No, score=0.6 required=5.0 tests=ALL_TRUSTED,DRUGS_ERECTILE, > FROM_NO_LOWER,INVALID_DATE,MISSING_SUBJECT,RM_hm_EmtyMsgid > autolearn=disabled version=3.0.1 You need to specify trusted_networks in local.cf, otherwise you're going to continue to hit the ALL_TRUSTED rule which can *decrease* your score by up to -3.3. If you don't specify trusted_networks then SpamAssassin infers what your trusted networks are - and the inference algorithm may not always get the correct result. For instance if your mail relay/server is on a private network and NATed thru a firewall, then the algorithm may infer incorrectly that the connecting mail server is trusted. i.e. the algorithm assumes that since you're a private address, then the next hop server must belong to you since your MX must be public. However it does not take NAT into account. Setting trusted_networks appropriately will solve this issue (I don't think SA 2.64 has the ALL_TRUSTED rule - or at least it scores low). Since you hit ALL_TRUSTED certain other DNS based tests are not run. Also is dns unavailable (dns_available no)? This may explain why you're not getting SURBL hits (which you should if dns is fully operational). Also skip_rbl_checks will do just that. Regards, - Sean
Re: {SPAM} SA 3.01 scoring very low
At 04:40 PM 11/3/2004, Dave Goodrich wrote: Good afternoon, I just finished testing an upgrade of SA to 3.01 and my scores fell through the floor. Read the docs, tried to use the Wiki, followed everyone else's upgrade on the list. Not sure just what went wrong. DAve Here is a sample output of spamassassin -D < test_spam (a known spam that had been caught and scored as follows, debug: received-header: unknown format: debug: received-header: unknown format: debug: received-header: unknown format: debug: received-header: unknown format: There's the cause of your problem.. SA is having problems parsing your received headers. As a result, SA is failing to properly detect a trust path, and is triggering ALL_TRUSTED, which should never happen for outside mail. debug: tests=ALL_TRUSTED,DRUGS_ERECTILE,FROM_NO_LOWER,INVALID_DATE,MISSING_SUBJECT,RM_hm_EmtyMsgid debug: subtests=__DRUGS_ERECTILE3,__DRUGS_ERECTILE_C,__SARE_HTML_HAS_MSG,__UNUSABLE_MSGID In the short term, force ALL_TRUSTED to 0 score ALL_TRUSTED 0 Open a bug report pointing out that SA isn't parsing your recieved headers. After you create the bug, attach a sample message to the bug so that the devs can test and fix things.
SA 3.01 scoring very low
Good afternoon, I just finished testing an upgrade of SA to 3.01 and my scores fell through the floor. Read the docs, tried to use the Wiki, followed everyone else's upgrade on the list. Not sure just what went wrong. DAve Here is a sample output of spamassassin -D < test_spam (a known spam that had been caught and scored as follows, previous score Content analysis details: (21.9 hits, 4.0 required) 2.8 SUBJ_VIAGRASubject includes "viagra" 2.6 LOCAL_OBFU_REGALIS_SUBJ Obfuscated 'REGALIS' in subject 2.6 LOCAL_OBFU_CIALIS_SUBJ Obfuscated 'CIALIS' in subject 0.9 SUBJ_BUY 'Subject' starts with Buy, Buying 1.8 LOCAL_OBFU_CIALIS BODY: Obfuscated 'CIALIS' in body 1.8 LOCAL_OBFU_REGALIS BODY: Obfuscated 'REGALIS' in body 3.0 SPAMCOP_URI_RBLURI's domain appears in spamcop database at sc.surbl.org [a1medz.com is blacklisted in URI RBL at] [multi.surbl.org] 2.1 WS_URI_RBL URI's domain appears in ws database at ws.surbl.org [a1medz.com is blacklisted in URI RBL at] [multi.surbl.org] 2.1 OB_URI_RBL URI's domain appears in ws database at ob.surbl.org [a1medz.com is blacklisted in URI RBL at] [multi.surbl.org] 2.2 SARE_URI_MEDS URI: domain selling meds * my local.cf *** # Add your own customisations to this file. See 'man Mail::SpamAssassin::Conf' # for details of what can be tweaked. required_hits 5.0 rewrite_header Subject *SPAM* report_safe 1 skip_rbl_checks 1 use_auto_whitelist 0 bayes_auto_learn 0 use_bayes 0 use_pyzor 0 use_dcc 0 use_razor2 0 dns_available no SA 3.01 debug output * bash-2.05b# spamassassin -D < test_spam debug: SpamAssassin version 3.0.1 debug: Score set 0 chosen. debug: running in taint mode? yes debug: Running in taint mode, removing unsafe env vars, and resetting PATH debug: PATH included '/sbin', keeping. debug: PATH included '/bin', keeping. debug: PATH included '/usr/sbin', keeping. debug: PATH included '/usr/bin', keeping. debug: PATH included '/usr/games', keeping. debug: PATH included '/usr/local/sbin', keeping. debug: PATH included '/usr/local/bin', keeping. debug: Final PATH set to: /sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/usr/X11R6/bin debug: using "/etc/mail/spamassassin/init.pre" for site rules init.pre debug: config: read file /etc/mail/spamassassin/init.pre debug: using "/usr/local/share/spamassassin" for default rules dir debug: config: read file /usr/local/share/spamassassin/10_misc.cf debug: config: read file /usr/local/share/spamassassin/20_anti_ratware.cf debug: config: read file /usr/local/share/spamassassin/20_body_tests.cf debug: config: read file /usr/local/share/spamassassin/20_compensate.cf debug: config: read file /usr/local/share/spamassassin/20_dnsbl_tests.cf debug: config: read file /usr/local/share/spamassassin/20_drugs.cf debug: config: read file /usr/local/share/spamassassin/20_fake_helo_tests.cf debug: config: read file /usr/local/share/spamassassin/20_head_tests.cf debug: config: read file /usr/local/share/spamassassin/20_html_tests.cf debug: config: read file /usr/local/share/spamassassin/20_meta_tests.cf debug: config: read file /usr/local/share/spamassassin/20_phrases.cf debug: config: read file /usr/local/share/spamassassin/20_porn.cf debug: config: read file /usr/local/share/spamassassin/20_ratware.cf debug: config: read file /usr/local/share/spamassassin/20_uri_tests.cf debug: config: read file /usr/local/share/spamassassin/23_bayes.cf debug: config: read file /usr/local/share/spamassassin/25_body_tests_es.cf debug: config: read file /usr/local/share/spamassassin/25_hashcash.cf debug: config: read file /usr/local/share/spamassassin/25_spf.cf debug: config: read file /usr/local/share/spamassassin/25_uribl.cf debug: config: read file /usr/local/share/spamassassin/30_text_de.cf debug: config: read file /usr/local/share/spamassassin/30_text_fr.cf debug: config: read file /usr/local/share/spamassassin/30_text_nl.cf debug: config: read file /usr/local/share/spamassassin/30_text_pl.cf debug: config: read file /usr/local/share/spamassassin/50_scores.cf debug: config: read file /usr/local/share/spamassassin/60_whitelist.cf debug: using "/etc/mail/spamassassin" for site rules dir debug: config: read file /etc/mail/spamassassin/70_sare_adult.cf debug: config: read file /etc/mail/spamassassin/70_sare_bayes_poison_nxm.cf debug: config: read file /etc/mail/spamassassin/70_sare_genlsubj0.cf debug: config: read file /etc/mail/spamassassin/70_sare_header0.cf debug: config: read file /etc/mail/spamassassin/70_sare_html0.cf
