Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[Ramprasad A Padmanabhan]

When I build the rpm from the spec file ( on fedora core 3 ) the
spamassassin-tools rpm is not created. Was it not a part of SA.

Thanks
Ram

On Sat, 2005-08-13 at 06:44, Justin Mason wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> *** THIS IS A RELEASE CANDIDATE ONLY, NOT THE FINAL 3.1.0 RELEASE ***
> 
> SpamAssassin 3.1.0-rc1 is released!  SpamAssassin 3.1.0 is a major update.
> SpamAssassin is a mail filter which uses advanced statistical and
> heuristic tests to identify spam (also known as unsolicited bulk email).
> 
> This is a release candidate, and NOT the general availability release (yet.)
> We think it's pretty rock solid, however. ;)
> 
> Highlights of the release
> - -
> 
> - - Apache preforking algorithm adopted; number of spamd child processes is 
> now
>   scaled, according to demand.  This provides better VM behaviour when not
>   under peak load.
> 
> - - added PostgreSQL, MySQL 4.1+, and local SDBM file Bayes storage modules. 
> SQL
>   storage is now recommended for Bayes, instead of DB_File. NDBM_File support
>   has been dropped due to a major bug in that module.
> 
> - - detect legitimate SMTP AUTH submission, to avoid false positives on
>   Dynablock-style rules.
> 
> - - new plugins: DomainKeys (off by default), MIMEHeader: a new plugin to 
> perform
>   tests against header in internal MIME structure, ReplaceTags: plugin by 
> Felix
>   Bauer to support fuzzy text matching, WhiteListSubject: plugin added to
>   support user whitelists by Subject header.
> 
> - - Razor: disable Razor2 support by default per our policy, since the
>   service is not free for non-personal use.  It's trivial to reenable.
> 
> - - DCC: disable DCC for similar reasons, due to new license terms.
> 
> - - Net::DNS bug: high load caused answer packets to be mixed up and 
> delivered as
>   answers to the wrong request, causing false positives.  worked around.
> 
> - - DNSBL lookups and other DNS operations are now more efficient, by using a
>   custom single-socket event-based model instead of Net::DNS.
> 
> Downloading
> - ---
> 
> Pick it up from:
> 
>   http://people.apache.org/~jm/devel/Mail-SpamAssassin-3.1.0-rc1.tar.gz
>   http://people.apache.org/~jm/devel/Mail-SpamAssassin-3.1.0-rc1.tar.bz2
>   http://people.apache.org/~jm/devel/Mail-SpamAssassin-3.1.0-rc1.zip
> 
> md5sum:
> 
>   c41126e515eacc5480d6d44498d5b99d  Mail-SpamAssassin-3.1.0-rc1.tar.bz2
>   196a22f1a9d27792d8388fbc6f1b522f  Mail-SpamAssassin-3.1.0-rc1.tar.gz
>   1763521a992ebd45c46ca1dcab586474  Mail-SpamAssassin-3.1.0-rc1.zip
> 
> sha1sum:
> 
>   17145041222d607d1591eb5cffdff80fdd55cd6c  
> Mail-SpamAssassin-3.1.0-rc1.tar.bz2
>   904c9b67498ec456c674545c15d0c4f89950a9da  Mail-SpamAssassin-3.1.0-rc1.tar.gz
>   f6d5d50abc70a4cedde3bc50715848aba1c3a4e4  Mail-SpamAssassin-3.1.0-rc1.zip
> 
> The release files also have a .asc accompanying them.  The file serves
> as an external GPG signature for the given release file.  The signing
> key is available via the wwwkeys.pgp.net key server, as well as
> http://spamassassin.apache.org/released/GPG-SIGNING-KEY
> 
> The key information is:
> 
> pub  1024D/265FA05B 2003-06-09 SpamAssassin Signing Key <[EMAIL PROTECTED]>
>  Key fingerprint = 26C9 00A4 6DD4 0CD5 AD24  F6D7 DEE0 1987 265F A05B
> 
> Important installation notes
> - 
> 
> - - see the INSTALL and UPGRADE files in the distribution.
> 
> Summary of major changes since 3.0.x
> - 
> 
> - - Apache preforking algorithm adopted; number of spamd child processes is 
> now
>   scaled, according to demand.  This provides better VM behaviour when not
>   under peak load.
> 
> - - Inclusion of sa-update script which will allow for updates of rules and
>   scores in between code releases.
> 
> - - added PostgreSQL, MySQL 4.1+, and local SDBM file Bayes storage modules. 
> SQL
>   storage is now recommended for Bayes, instead of DB_File. NDBM_File support
>   has been dropped due to a major bug in that module.
> 
> - - detect legitimate SMTP AUTH submission, to avoid false positives on
>   Dynablock-style rules.
> 
> - - new Advance Fee Fraud (419 scam) rules.
> 
> - - removed use of the Storable module, due to several reported hangs on SMP
>   Linux machines.
> 
> - - Converted several rule/engine components into Plugins such as:
>   AccessDB, AWL, Pyzor, Razor2, DCC, Bayes AutoLearn Determination, etc.
> 
> - - new plugins: DomainKeys (off by default), MIMEHeader: a new plugin to 
> perform
>   tests against header in internal MIME structure, ReplaceTags: plugin by 
> Felix
>   Bauer to support fuzzy text matching, WhiteListSubject: plugin added to
>   support user whitelists by Subject header.
> 
> - - TextCat language guesser moved to a plugin.  (This means "ok_languages"
>   is no longer part of the core engine by default.)
> 
> - - Razor: disable Razor2 support by default per our policy, since the
>   service is not free for non-persona

RE: Faster rDNS lookups

[Rob McEwen]

>Cami asked
Exactly how is this faster than using a dns caching nameserver?

As I mentioned, (1) artificially long caching times (well beyond TTL) can be
set for both negative and positive return and (2) once cached, the lookup is
not dependent on another 3rd party server which might have been overloaded
or mis-configured.

--Rob McEwen

Re: Faster rDNS lookups

[Cami]

Rob McEwen wrote:

Cami wrote

Exactly how is this faster than using a dns caching nameserver?


As I mentioned, (1) artificially long caching times (well beyond TTL) can be
set for both negative and positive return and (2) once cached, the lookup is
not dependent on another 3rd party server which might have been overloaded
or mis-configured.


Artificially long caching the result of overloaded or mis-configured
has the same drawbacks as regular dns caching servers, possibly even
worse.

You're welcome to provide some type of tests/data to backup your
ideas/claim but i'm somewhat specticle if anything positive/usefull
will come out of it. I see no problem whatsoever with regular dns
caching nameservers, even dealing with 10's of millions of mails /
lookups per day on my systems.

Doing non-cached lookups are expensive, thats a given. Use a caching
nameserver and its no longer expensive.

Cami

FYI: ccTLD .de listed in RFC-ignorant.org

[Dirk Bonengel]

FYI:
rfc-ignorant.org has .de listed in whois.rfc-ignorant.com.

http://www.rfc-ignorant.org/tools/detail.php?domain=de&submitted=1120996396&table=whois
In a standard 3.0.x install, DNS_FROM_RFC_WHOIS gives a score of 0.492 
(net) or 0.296 (net+bayes).

Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[email builder]

> SpamAssassin 3.1.0-rc1 is released!  SpamAssassin 3.1.0 is a major update.

Very exciting!

> - - Apache preforking algorithm adopted; number of spamd child processes is
> now
>   scaled, according to demand.  This provides better VM behaviour when not
>   under peak load.

I presume this is activated by default.
 
> - - added PostgreSQL, MySQL 4.1+, and local SDBM file Bayes storage
> modules. SQL
>   storage is now recommended for Bayes, instead of DB_File. NDBM_File
> support
>   has been dropped due to a major bug in that module.

What's the difference between the MySQL support that already existed in prior
versions?  Is there anything those of us who already have our bayes data in
MySQL should do differently as of 3.1.0?

Thanks for a great product!




Start your day with Yahoo! - make it your home page 
http://www.yahoo.com/r/hs 
 

Re: FYI: ccTLD .de listed in RFC-ignorant.org

[Rob Skedgell]

On Saturday 13 Aug 2005 12:29, Dirk Bonengel wrote:
> FYI:
> rfc-ignorant.org has .de listed in whois.rfc-ignorant.com.
> 
> 
http://www.rfc-ignorant.org/tools/detail.php?domain=de&submitted=1120996396&table=whois
> In a standard 3.0.x install, DNS_FROM_RFC_WHOIS gives a score of 0.492 
> (net) or 0.296 (net+bayes). 

This shouldn't cause problems as RFCI whois returns 127.0.0.7 for entire 
TLD based domains, and 127.0.0.5 for others and SA (at least 3.0.4) 
only tests for 127.0.0.5. See the listing policy at 
.

$ grep DNS_FROM_RFC_WHOIS /usr/share/spamassassin/*
/usr/share/spamassassin/20_dnsbl_tests.cf:header DNS_FROM_RFC_WHOIS 
eval:check_rbl_sub('rfci_envfrom', '127.0.0.5')
/usr/share/spamassassin/20_dnsbl_tests.cf:describe DNS_FROM_RFC_WHOIS   
Envelope sender in whois.rfc-ignorant.org
/usr/share/spamassassin/20_dnsbl_tests.cf:tflags DNS_FROM_RFC_WHOIS 
net
/usr/share/spamassassin/30_text_de.cf:lang de describe 
DNS_FROM_RFC_WHOIS Absender in whois-Liste von www.rfc-ignorant.org
/usr/share/spamassassin/50_scores.cf:score DNS_FROM_RFC_WHOIS 0 0.492 0 
0.296

e.g.:

$ host de.whois.rfc-ignorant.org
de.whois.rfc-ignorant.org has address 127.0.0.7

$ host wot-2.com.whois.rfc-ignorant.org
wot-2.com.whois.rfc-ignorant.org has address 127.0.0.5

-- 
Rob Skedgell <[EMAIL PROTECTED]>


pgp6RdRXo1y7G.pgp
Description: PGP signature

Re: FYI: ccTLD .de listed in RFC-ignorant.org

[Dirk Bonengel]

Thanks for the clarification.

Dirk

Rob Skedgell schrieb:


On Saturday 13 Aug 2005 12:29, Dirk Bonengel wrote:
 


FYI:
rfc-ignorant.org has .de listed in whois.rfc-ignorant.com.


   


http://www.rfc-ignorant.org/tools/detail.php?domain=de&submitted=1120996396&table=whois
 

In a standard 3.0.x install, DNS_FROM_RFC_WHOIS gives a score of 0.492 
(net) or 0.296 (net+bayes). 
   



This shouldn't cause problems as RFCI whois returns 127.0.0.7 for entire 
TLD based domains, and 127.0.0.5 for others and SA (at least 3.0.4) 
only tests for 127.0.0.5. See the listing policy at 
.


$ grep DNS_FROM_RFC_WHOIS /usr/share/spamassassin/*
/usr/share/spamassassin/20_dnsbl_tests.cf:header DNS_FROM_RFC_WHOIS 
eval:check_rbl_sub('rfci_envfrom', '127.0.0.5')
/usr/share/spamassassin/20_dnsbl_tests.cf:describe DNS_FROM_RFC_WHOIS   
Envelope sender in whois.rfc-ignorant.org
/usr/share/spamassassin/20_dnsbl_tests.cf:tflags DNS_FROM_RFC_WHOIS 
net
/usr/share/spamassassin/30_text_de.cf:lang de describe 
DNS_FROM_RFC_WHOIS Absender in whois-Liste von www.rfc-ignorant.org
/usr/share/spamassassin/50_scores.cf:score DNS_FROM_RFC_WHOIS 0 0.492 0 
0.296


e.g.:

$ host de.whois.rfc-ignorant.org
de.whois.rfc-ignorant.org has address 127.0.0.7

$ host wot-2.com.whois.rfc-ignorant.org
wot-2.com.whois.rfc-ignorant.org has address 127.0.0.5

 

RE: Faster rDNS lookups

[Rob McEwen]

Please understand, I'm only proposing this as an alternative idea for
checking to see if a sending server's IP address has proper rDNS... NOT any
other type of DNS lookups.

Also, I don't have stats on this, but I know that most mail is spam and I
know that MUCH of this spam has no rDNS properly configured.

Furthermore, when rDNS is not properly configured and a subsequent lookup is
done on the same IP, I'm pretty sure that the negative (no found) lookup
isn't cached at all or at least not for long.

For example, if you do a lookup on blahblah.yourdomain.com and that domain
doesn't exist. Next, seconds later, create an A-record for
blahblah.yourdomain.com ...finally, seconds later, do another lookup for
blahblah.yourdomain.com

In my testing, that last lookup DOES find something... indicating that DNS
servers don't generally cache previously "not found" lookups... at least not
for long.

rDNS on a spammer's IP which doesn't have rDNS configured involves a similar
situation. Therefore, I don't think DNS servers cache the negative results
of rDNS checks either at all or for very long. But, ironically, these types
of checks are more expensive than most lookups. I've found that DNS server
often take longer to return a "not found" result when the IP address is from
some 3rd party network in Korea... in contrast to an "is-found" lookup from
a DNS server on your own network.

Also, if absolutely no caching of negative lookups takes place, this would
be even more beneficial for spam attacks from a single IP just prior to
tarpitting kicking in... but that particular need is not a prerequisite to
my idea being a good one overall... just "iceing on the cake".

--Rob McEwen

Anything Happened to blackholes.us ?

[Ilan Aisic]

For the last couple of days my SA wasn't able to use blackholes.us.
I tried to ping it but it seems dead.   Anyone knows the reason and
if/when it's coming back?
-- 
Ilan Aisic
Registered Linux User 8124 http://counter.li.org

Re: Faster rDNS lookups

[Cami]

Rob McEwen wrote:

Please understand, I'm only proposing this as an alternative idea for
checking to see if a sending server's IP address has proper rDNS... NOT any
other type of DNS lookups.

Also, I don't have stats on this, but I know that most mail is spam and I
know that MUCH of this spam has no rDNS properly configured.


If 9% of incoming mail is spam (at least it is on my mail systems), how
does that help? (It used to be >80% but Postfix+policyd has reduced it
to barely anything). Lets use an extreme case and say that 100% of that
9% does not have reverse dns records and are spam mails. What kind of
saving is there?


Furthermore, when rDNS is not properly configured and a subsequent lookup is
done on the same IP, I'm pretty sure that the negative (no found) lookup
isn't cached at all or at least not for long.


Thats a good thing.


In my testing, that last lookup DOES find something... indicating that DNS
servers don't generally cache previously "not found" lookups... at least not
for long.


Again, thats a good thing.


rDNS on a spammer's IP which doesn't have rDNS configured involves a similar
situation. Therefore, I don't think DNS servers cache the negative results
of rDNS checks either at all or for very long. But, ironically, these types
of checks are more expensive than most lookups. I've found that DNS server
often take longer to return a "not found" result when the IP address is from
some 3rd party network in Korea... in contrast to an "is-found" lookup from
a DNS server on your own network.


I do not see that in my tests.

Random hosts with reverse DNS:

[/]# time host 70.106.32.110
110.32.106.70.in-addr.arpa domain name pointer 
pool-70-106-32-110.hag.east.verizon.net.

real0m0.005s
user0m0.001s
sys 0m0.003s
[/]# time host 213.93.30.26
26.30.93.213.in-addr.arpa domain name pointer e30026.upc-e.chello.nl.
real0m0.005s
user0m0.001s
sys 0m0.006s


Random hosts without reverse DNS:


[/]# time host 211.195.99.146
Host 146.99.195.211.in-addr.arpa not found: 3(NXDOMAIN)
real0m0.005s
user0m0.000s
sys 0m0.002s
[/]# time host 61.149.3.145
Host 145.3.149.61.in-addr.arpa not found: 3(NXDOMAIN)
real0m0.004s
user0m0.000s
sys 0m0.004s

Typically hosts without any rdns are faster to lookup than hosts with
rdns records. From some poking around, it appears that negative rdns
lookups are cached for 10 minutes, up to a maximum of up to 3 hours:

Negative Caching of DNS Queries -> http://www.faqs.org/rfcs/rfc2308.html

Cami



Cami

Re: FYI: ccTLD .de listed in RFC-ignorant.org

[Ralf Hildebrandt]

* Dirk Bonengel <[EMAIL PROTECTED]>:
> FYI:
> rfc-ignorant.org has .de listed in whois.rfc-ignorant.com.

Yes, since Spet. 2004

-- 
Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED]
Charite - Universitätsmedizin BerlinTel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-BerlinFax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF send no mail to [EMAIL PROTECTED]

Re: Faster rDNS lookups

[Rob McEwen (PowerView Systems)]

Cami said:
>It used to be >80% but Postfix+policyd has reduced it
>to barely anything

The fact that you use graylisting as a means of eliminating spam as a 1st line 
of defense is makes your "9% of incoming mail is spam" as VERY anecdotal for 
the purposes of this discussion. Please, don't confuse issues here. MANY people 
prefer to not use graylisting (for a variety of reasons)... and their 
proportions of incoming spam (prior to being filtered) really is more like your 
previous "used to be >80%". Therefore, for the rest of us non-graylisters... 
this really could be beneficial.

> it appears that negative rdns
> lookups are cached for 10 minutes
I think that this depends of a variety of real world factors which might be 
very different from published standards.

Cami... why don't we just count you as a "no vote" for my idea and let others 
weigh in on it and, certainly, I'm sure they will take into account all of the 
good valid point point you brought up.

Rob McEwen
PowerView Systems

Re: Faster rDNS lookups

[Cami]

Rob McEwen (PowerView Systems) wrote:

Cami said:


it appears that negative rdns lookups are cached for 10 minutes


I think that this depends of a variety of real world factors
which might be very different from published standards.


Published standards (bind/named) is 10 minutes.


Cami... why don't we just count you as a "no vote" for my idea
and let others weigh in on it and, certainly, I'm sure they
will take into account all of the good valid point point you
brought up.


Ok..

Cami

Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[Michael Parker]

email builder wrote:

>>- - added PostgreSQL, MySQL 4.1+, and local SDBM file Bayes storage
>>modules. SQL
>>  storage is now recommended for Bayes, instead of DB_File. NDBM_File
>>support
>>  has been dropped due to a major bug in that module.
>>
>>
>
>What's the difference between the MySQL support that already existed in prior
>versions?  Is there anything those of us who already have our bayes data in
>MySQL should do differently as of 3.1.0?
>
>  
>

The previous SQL support (Mail::SpamAssassin::BayesStore::SQL) was very
generic, usable by multiple database drivers.  With 3.1.0 we broke out
the support and now include 2 very specific SQL backends (MySQL 4.1+ and
PostgreSQL) in addition to the more generic backend.  These specific
backends make use of non-standard SQL features to get a speed boost.

That said, if you were previously using SQL support with a MySQL
database then you should be able to simply switch to using
Mail::SpamAssassin::BayesStore::MySQL and get an instant speedup,
assuming you already have MySQL 4.1+ installed.  We do suggest that you
switch your tables to InnoDB type tables (not currently the default) to
get better data integrity (with the support of transactions).

If you were using PostgreSQL with the previous support, you should
switch (we're talking about a 7x - 27x improvement) ASAP, which might
involve a complete wipe and rebuild of your database.  Although, I would
try an sa-learn --backup and sa-learn --restore before I completely gave
up on the data.

If you are interested in how well the various backends perform, compared
to the others, see http://wiki.apache.org/spamassassin/BayesBenchmarkResults
It is very hard to compare to previous versions, due to changes in other
factors such as rules and message parsing code, but the improvments in
3.1 represent anywhere from a 2x - 27x improvements in previous performance.

Hmmmmaybe some of the above should be captured in the documentation,
patches welcome.

Michael




signature.asc
Description: OpenPGP digital signature

Re: Faster rDNS lookups

[List Mail User]

>...
>
>Rob McEwen wrote:
>> I've got an idea for faster rDNS lookups.
>> 
>> Before I present the solution. Here is the problem...
>> 
>> ...basically, rDNS checks are "expensive". They sometimes take a few seconds
>> when done in real time and depend on the timeliness and of other people's
>> DNS servers. This 1-5 (or more) seconds may not seem like a big deal... but
>> if this can be more instantaneous, it increases the volume of messages a
>> server-side spam filter can handle. (every little bit counts)
>> 
>> Anyone doubt me? Well... my point is valid for the same reason that it is
>> typically MUCH faster to check again a DNSBL (like SURBL) than it is to
>> convert domains found within the e-mail to IP addresses and then check these
>> again SpamHaus's RBL. Both of these are good techniques... but DNSBL
>> checking where each domain doesn't have to be resolved to an IP address is
>> much faster. For the very same reasons... again... rDNS checks are a bit
>> "expensive".
>
>Exactly how is this faster than using a dns caching nameserver?
>
>Cami
>
Cami's point is very close - DNS lookups are slow, but not
"expensive" (your machine can do other operations while waiting).
Other problems with the idea and valid (i.e. non-spam) domains are:

a) virtual hosting with many to one DNS maps, but a single IP->name rDNS
   and the hosting company moves domains around to "load balance".  This
   will cause rDNS to "flap" whenever a virtual domain is moved.

b) multi-homed machines and clusters - rDNS is valid, but needs a many
   to one mapping to be correct (i.e. your database would have to allow
   for that, so the replies could be larger than the existing case, and
   thus actually be "more expensive" - i.e. more processing required).

And for spam domains, IP-jumping is common (i.e. they get in the
SBL, so the host changes its IP);  Therefore your hypothetical server would
have to query every entry at least as often as the TTL to keep up to date.
Simply, the rDNS for spam (not most ham) domains and name servers (SA looks
those up also) does change often, so for these cases, your assumption is
incorrect for the important case (for well run, legitimate domains, what
you say is indeed correct - rDNS seldom changes).

The shortest possible reason is "slow != expensive".  At best, you
would shorten delivery times by a few seconds, but even on a slow machine
I doubt the CPU savings would be more than a couple of milliseconds (OK,
if you have a lousy NIC card and a slow CPU, maybe 10 milliseconds).  *AND*
the load on the hypothetical "pre-mapped rDNS" server would be extreme.

Paul Shupak
[EMAIL PROTECTED]

Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[Michael Parker]

Michael Parker wrote:

> Hmmmmaybe some of the above should be captured in the documentation,
>
>patches welcome.
>
>  
>
Nevermind, this is what I just added to UPGRADE, hopefully that make
things clearer:

+- As of 3.1.0, in addition to the generic BayesSQL support (via
+  Mail::SpamAssassin::BayesStore::SQL) usable by multiple database
+  drivers there is now specific support for MySQL 4.1+ and
+  PostgreSQL.  This support is based on non-standard features present
+  in both database servers that allow for various performance boosts.
+
+  If you were using the previous BayesSQL support with MySQL, and
+  already have MySQL 4.1+ installed you can begin using the new module
+  immediately by replacing the bayes_store_module line in your
+  configuration with:  Mail::SpamAssassin::BayesStore::MySQL
+
+  We do however recommend that you switch your MySQL tables over to
+  InnoDB for better data integrity and multi user support.  You can
+  most often do this via a simple ALTER TABLE command, refer to the
+  MySQL documentation for more information.
+
+  If you were previously using PostgreSQL for your bayes database then
+  we STRONGLY recommend switching to the PostgreSQL specific backend:
+  Mail::SpamAssassin::BayesStore::PgSQL
+  To switch to this backend you should first run sa-learn --backup to
+  make a backup of your existing data and then drop and recreate the
+  database following the instructions in sql/README.bayes.  Then you
+  can restore the database with sa-learn --restore.  If you have
+  multiple users then you will have to run --backup and --restore for
+  each user to fully restore the database.


Michael



signature.asc
Description: OpenPGP digital signature

Re: FYI: ccTLD .de listed in RFC-ignorant.org

[List Mail User]

>...
>FYI:
>rfc-ignorant.org has .de listed in whois.rfc-ignorant.com.
>
>http://www.rfc-ignorant.org/tools/detail.php?domain=de&submitted=1120996396&table=whois
>In a standard 3.0.x install, DNS_FROM_RFC_WHOIS gives a score of 0.492 
>(net) or 0.296 (net+bayes).
>
Dirk,

By default the check for the rule is against the value 127.0.0.5,
so there is no penalty (TLDs which are invalid are marked with 127.0.0.7).
However, on my own servers, you get penalized .1 point (local rule).
Note, that ".de" was *almost* fixed about a month ago, but was completely
broken again last week.

Paul Shupak
[EMAIL PROTECTED]

P.S.  The person who does most of the checking for rfci is himself German
and his email all goes through ".de" domains (and he does occasionally
respond on this list).

RE: Faster rDNS lookups

[List Mail User]

>...
>
>Rob,
>
>...
>
>For example, if you do a lookup on blahblah.yourdomain.com and that domain
>...
>doesn't exist. Next, seconds later, create an A-record for
>blahblah.yourdomain.com ...finally, seconds later, do another lookup for
>blahblah.yourdomain.com
>
>In my testing, that last lookup DOES find something... indicating that DNS
>servers don't generally cache previously "not found" lookups... at least not
>for long.

Be vary careful with this one.  If indeed you are testing
"yourdomain", it is possible that the server checked is authoritative
for the domain and then *no* caching was done;  Are you certain that
you checked using a different server that was not authoritative;
Otherwise the test was invalid and needs to be redone.
>
>...
>
>--Rob McEwen
>
Paul Shupak
[EMAIL PROTECTED]

Re: Faster rDNS lookups

[Rob McEwen (PowerView Systems)]

>And for spam domains, IP-jumping is common...
>...for well run, legitimate domains, what
>you say is indeed correct
Overall, I think you actually make the case FOR my idea of artifically long 
cacheing of rDNS checks. And, I think my earlier messages covered the various 
scenarios.

>the load on the hypothetical "pre-mapped rDNS" server would be extreme.
Probably the best point so far against my idea

>Are you certain that you checked using a 
>different server that was not authoritative;
>Otherwise the test was invalid and needs to be redone.
Should I put a "dunce cap on" now... turns out that what you describe is 
exactly that happened. But, since then, I've gotten some mixed results...

Rob McEwen
PowerView Systems
[EMAIL PROTECTED]
(478) 475-9032

Re: Faster rDNS lookups

[List Mail User]

>>And for spam domains, IP-jumping is common...
>>...for well run, legitimate domains, what
>>you say is indeed correct
>Overall, I think you actually make the case FOR my idea of artifically long 
>cacheing of rDNS checks. And, I think my earlier messages covered the various 
>scenarios.
>
>>the load on the hypothetical "pre-mapped rDNS" server would be extreme.
>Probably the best point so far against my idea
>
>>Are you certain that you checked using a 
>>different server that was not authoritative;
>>Otherwise the test was invalid and needs to be redone.
>Should I put a "dunce cap on" now... turns out that what you describe is 
>exactly that happened. But, since then, I've gotten some mixed results...
>
>Rob McEwen
>PowerView Systems
>[EMAIL PROTECTED]
>(478) 475-9032
>
Rob,

Certainly not (no dunce cap).  Public airing of ideas generally
has merit - quite possibly this idea can be refined to something similar
that will provide a benefit (I admit, I have not given it a lot of thought).

Paul Shupak
[EMAIL PROTECTED]

Re: FYI: ccTLD .de listed in RFC-ignorant.org

[Ralf Hildebrandt]

* List Mail User <[EMAIL PROTECTED]>:

>   By default the check for the rule is against the value
> 127.0.0.5, so there is no penalty (TLDs which are invalid are marked
> with 127.0.0.7). However, on my own servers, you get penalized .1 point
> (local rule). Note, that ".de" was *almost* fixed about a month ago,
> but was completely broken again last week.

Was it? Got some details?

-- 
Ralf Hildebrandt (i.A. des IT-Zentrums) [EMAIL PROTECTED]
Charite - Universitätsmedizin BerlinTel.  +49 (0)30-450 570-155
Gemeinsame Einrichtung von FU- und HU-BerlinFax.  +49 (0)30-450 570-962
IT-Zentrum Standort CBF send no mail to [EMAIL PROTECTED]

Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[The Doctor]

On Fri, Aug 12, 2005 at 06:14:43PM -0700, Justin Mason wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
> 
> *** THIS IS A RELEASE CANDIDATE ONLY, NOT THE FINAL 3.1.0 RELEASE ***
> 
> SpamAssassin 3.1.0-rc1 is released!  SpamAssassin 3.1.0 is a major update.
> SpamAssassin is a mail filter which uses advanced statistical and
> heuristic tests to identify spam (also known as unsolicited bulk email).
> 
> This is a release candidate, and NOT the general availability release (yet.)
> We think it's pretty rock solid, however. ;)
> 
> Highlights of the release
> - -
> 
> - - Apache preforking algorithm adopted; number of spamd child processes is 
> now
>   scaled, according to demand.  This provides better VM behaviour when not
>   under peak load.
> 
> - - added PostgreSQL, MySQL 4.1+, and local SDBM file Bayes storage modules. 
> SQL
>   storage is now recommended for Bayes, instead of DB_File. NDBM_File support
>   has been dropped due to a major bug in that module.
> 
> - - detect legitimate SMTP AUTH submission, to avoid false positives on
>   Dynablock-style rules.
> 
> - - new plugins: DomainKeys (off by default), MIMEHeader: a new plugin to 
> perform
>   tests against header in internal MIME structure, ReplaceTags: plugin by 
> Felix
>   Bauer to support fuzzy text matching, WhiteListSubject: plugin added to
>   support user whitelists by Subject header.
> 
> - - Razor: disable Razor2 support by default per our policy, since the
>   service is not free for non-personal use.  It's trivial to reenable.
> 
> - - DCC: disable DCC for similar reasons, due to new license terms.
> 
> - - Net::DNS bug: high load caused answer packets to be mixed up and 
> delivered as
>   answers to the wrong request, causing false positives.  worked around.
> 
> - - DNSBL lookups and other DNS operations are now more efficient, by using a
>   custom single-socket event-based model instead of Net::DNS.
> 
> Downloading
> - ---
> 
> Pick it up from:
> 
>   http://people.apache.org/~jm/devel/Mail-SpamAssassin-3.1.0-rc1.tar.gz
>   http://people.apache.org/~jm/devel/Mail-SpamAssassin-3.1.0-rc1.tar.bz2
>   http://people.apache.org/~jm/devel/Mail-SpamAssassin-3.1.0-rc1.zip
> 
> md5sum:
> 
>   c41126e515eacc5480d6d44498d5b99d  Mail-SpamAssassin-3.1.0-rc1.tar.bz2
>   196a22f1a9d27792d8388fbc6f1b522f  Mail-SpamAssassin-3.1.0-rc1.tar.gz
>   1763521a992ebd45c46ca1dcab586474  Mail-SpamAssassin-3.1.0-rc1.zip
> 
> sha1sum:
> 
>   17145041222d607d1591eb5cffdff80fdd55cd6c  
> Mail-SpamAssassin-3.1.0-rc1.tar.bz2
>   904c9b67498ec456c674545c15d0c4f89950a9da  Mail-SpamAssassin-3.1.0-rc1.tar.gz
>   f6d5d50abc70a4cedde3bc50715848aba1c3a4e4  Mail-SpamAssassin-3.1.0-rc1.zip
> 
> The release files also have a .asc accompanying them.  The file serves
> as an external GPG signature for the given release file.  The signing
> key is available via the wwwkeys.pgp.net key server, as well as
> http://spamassassin.apache.org/released/GPG-SIGNING-KEY
> 
> The key information is:
> 
> pub  1024D/265FA05B 2003-06-09 SpamAssassin Signing Key <[EMAIL PROTECTED]>
>  Key fingerprint = 26C9 00A4 6DD4 0CD5 AD24  F6D7 DEE0 1987 265F A05B
> 
> Important installation notes
> - 
> 
> - - see the INSTALL and UPGRADE files in the distribution.
> 
> Summary of major changes since 3.0.x
> - 
> 
> - - Apache preforking algorithm adopted; number of spamd child processes is 
> now
>   scaled, according to demand.  This provides better VM behaviour when not
>   under peak load.
> 
> - - Inclusion of sa-update script which will allow for updates of rules and
>   scores in between code releases.
> 
> - - added PostgreSQL, MySQL 4.1+, and local SDBM file Bayes storage modules. 
> SQL
>   storage is now recommended for Bayes, instead of DB_File. NDBM_File support
>   has been dropped due to a major bug in that module.
> 
> - - detect legitimate SMTP AUTH submission, to avoid false positives on
>   Dynablock-style rules.
> 
> - - new Advance Fee Fraud (419 scam) rules.
> 
> - - removed use of the Storable module, due to several reported hangs on SMP
>   Linux machines.
> 
> - - Converted several rule/engine components into Plugins such as:
>   AccessDB, AWL, Pyzor, Razor2, DCC, Bayes AutoLearn Determination, etc.
> 
> - - new plugins: DomainKeys (off by default), MIMEHeader: a new plugin to 
> perform
>   tests against header in internal MIME structure, ReplaceTags: plugin by 
> Felix
>   Bauer to support fuzzy text matching, WhiteListSubject: plugin added to
>   support user whitelists by Subject header.
> 
> - - TextCat language guesser moved to a plugin.  (This means "ok_languages"
>   is no longer part of the core engine by default.)
> 
> - - Razor: disable Razor2 support by default per our policy, since the
>   service is not free for non-personal use.  It's trivial to reenable.
> 
> - - DCC: disable DCC for similar reasons, due to new license terms.
> 
> - - Net::DNS bug:

Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[Jim Knuth]

Hallo und Guten Abend The,

Heute (am 13.08.2005 - 22:19 Uhr)
   schriebst Du: 

> On Fri, Aug 12, 2005 at 06:14:43PM -0700, Justin Mason wrote:
>> -BEGIN PGP SIGNED MESSAGE-
>> Hash: SHA1
>> 
>> *** THIS IS A RELEASE CANDIDATE ONLY, NOT THE FINAL 3.1.0 RELEASE ***
>> 
>>


> So far this BSD Box running MailScanner and milter-spamc is not seeing
> any of the problems reported earlier.

I`ve updated tonight. No problems here.  (with Debian Sarge)


-- 
Viele Grüße, Kind regards,
 Jim Knuth
 [EMAIL PROTECTED]
 ICQ #277289867
 PGP: 54C9 1A46 D3B2 95B6 454D 74FA AC73 773E 1F78 066F
--
Zufalls-Zitat
--
Wir leben alle unter demselben Himmel, aber wir haben nicht 
alle denselben Horizont. (Konrad Adenauer, dt. 
Bundeskanzler 1876-1967)
--
Der Text hat nichts mit dem Empfänger der Mail zu tun
--

Virus free. Checked by NOD32 Version 1.1193 Update 12.08.2005

Re: Faster rDNS lookups

[mouss]

Rob McEwen wrote:

I've got an idea for faster rDNS lookups.

Before I present the solution. Here is the problem...

...basically, rDNS checks are "expensive". They sometimes take a few seconds
when done in real time and depend on the timeliness and of other people's
DNS servers. This 1-5 (or more) seconds may not seem like a big deal... but
if this can be more instantaneous, it increases the volume of messages a
server-side spam filter can handle. (every little bit counts)

Anyone doubt me? Well... my point is valid for the same reason that it is
typically MUCH faster to check again a DNSBL (like SURBL) than it is to
convert domains found within the e-mail to IP addresses and then check these
again SpamHaus's RBL. Both of these are good techniques... but DNSBL
checking where each domain doesn't have to be resolved to an IP address is
much faster. For the very same reasons... again... rDNS checks are a bit
"expensive".

But consider that a given IP's rDNS doesn't change that much (on average).
Therefore, it would be a huge time and resource saver to specially program a
DNS server to receive rDNS queries in RBL-like format, to do the rDNS lookup
on the IP address passed in the RBL-like query, and then to return results
much like an RBL would... "Null" for having an rDNS, and "127.0.0.x" if the
IP does NOT have rDNS. This way, the mail server is "tricked" into thinking
that this is just another RBL.

You might ask, "where is the time savings if the DNS server has to fetch the
rDNS all the same?"

Here is the clincher... because, as I mentioned, rDNS settings don't change
often, therefore, PERHAPS the DNS server could be configured or
programmed to cache BOTH positive and negative results for 24 hours (or
whatever the config file requests... maybe even DAYS). This way, although
the 1st rDNS query would still be expensive, subsequent queries would be
lightening fast. Also, since rDNS should (ideally) never be enough of a
factor by itself to mark a message as spam, then, therefore, a rare mishap
on someone's legit mail server wouldn't block mail if all else is well. (And
those few who have no problem blocking soly on rDNS... they'd not mind such
extremely rare FPs anyways) Therefore, the negative caching would have
virtually no effect on temporary mistakes made by other mail admins... and
this is in contrast to the damage that might be done if a regular DNSBL's
records were cached for too long and a FP comes along that was quickly and
already "fixed" but then still "stuck" as blacklisted on a server due to
such caching.

Also, before anyone mentions TTL values providing caching anyways...
consider that you typically get not long or zero caching of negative
results.

This feature I describe would be VERY beneficial because MANY spam filters
which have to option to do RBL lookups could benefit. They could just remove
their regular rDNS feature and then do the lookup I described and use the
RBL-based return in the same way that they previously used the rDNS for.

Does anyone know of a publicly available RBL that already does this? Anyone
interested in creating such a service? (I don't think I have the expertise
to do this.)


If you cache negative results, you won't catch new dnsbl entries.

helo lookup before rDNS [Was: Faster rDNS lookups]

[mouss]

List Mail User wrote:


Certainly not (no dunce cap).  Public airing of ideas generally
has merit - quite possibly this idea can be refined to something similar
that will provide a benefit (I admit, I have not given it a lot of thought).



one thing that an MTA can do is to delay its rDNS lookup until it gets 
the helo name. then do a forward lookup on the helo name.
if one of the returned IPs matches the client IP, there will be no need 
for the classical double lookup.


if one assumes that most helo names are correct, then this would be a 
minor improvement. and the benefit is that it automatically checks for 
helo validity (I don't mean one would reject the helo, but he knows if 
it's good or not).


(I think iMail does something like this, but I can't say for sure).

Re: Anything Happened to blackholes.us ?

[mouss]

Ilan Aisic wrote:

For the last couple of days my SA wasn't able to use blackholes.us.
I tried to ping it but it seems dead.   Anyone knows the reason and
if/when it's coming back?


This is probably a timeout issue. works (slow but works) from here.

BTW, the web page says "29 March, 2005". does this mean the zones don't 
get updated?

3.1 and Net::Ident

[Steve Martin]

I'm getting ready for 3.1 and noticed one of the prerequisites is  
Net::Ident.


MacOS 10.4 no longer includes an ident daemon so I scrounged about a  
bit and found one...


http://www.macmax.org/article.php3?id_article=39

But, it fails test 6 of the Net::Ident install (it still works after  
the connection closes which makes me think is just a "fake" identd.


1) Is Net::Ident really required?
2) Does anyone know a good working identd for MacOS 10.4?

Anytyhing wrong with forcing the install despite the test failure?


--
Steve Martin  http://www.cheezmo.com/
Smart Calibration, LLC   http://www.smartcalibration.com/
The Widescreen Movie Centerhttp://www.widemovies.com/
Letterboxed Movie TV Schedule  http://www.widemovies.com/lbx.html



smime.p7s
Description: S/MIME cryptographic signature

Re: 3.1 and Net::Ident

[Michael Parker]

Steve Martin wrote:

> I'm getting ready for 3.1 and noticed one of the prerequisites is 
> Net::Ident.


Correction, one of the optional modules.It is not required.

Why gave you the idea that it was a required module?

Michael



signature.asc
Description: OpenPGP digital signature

Re: 3.1 and Net::Ident

[Steve Martin]

My mistake.

I wasn't looking to closely just saw the list of things it said  
weren't installed.


I guess it isn't a big deal, but now that I've started down the path  
of trying to get it working ...


On Aug 13, 2005, at 5:15 PM, Michael Parker wrote:


Steve Martin wrote:



I'm getting ready for 3.1 and noticed one of the prerequisites is
Net::Ident.




Correction, one of the optional modules.It is not required.

Why gave you the idea that it was a required module?

Michael




--
Steve Martin  http://www.cheezmo.com/
Smart Calibration, LLC   http://www.smartcalibration.com/
The Widescreen Movie Centerhttp://www.widemovies.com/
Letterboxed Movie TV Schedule  http://www.widemovies.com/lbx.html



smime.p7s
Description: S/MIME cryptographic signature

Re: 3.1 and Net::Ident

[John Rudd]

On Aug 13, 2005, at 3:12 PM, Steve Martin wrote:


2) Does anyone know a good working identd


this is a contradiction in terms.  "Good" and "working" cannot coexist 
with "identd".  Ok, I guess you could say that "an identd which 
accurately implements the identd protocol is 'working'", but that's 
like saying that "a car that is DESIGNED to explode into a fireball 
when you drive it, is functioning perfectly when it burns you to 
death."  (which then suggests that "identd is the pinto of 
identification/authentication protocols" would be a good description)  
But, still, "good" doesn't belong in there.


identd in general is worthless.  So, unless SA is using 'you run 
identd, so you must be a con-artist' as a means of increasing an SA 
score, I would avoid using anything based on identd.


http://www.clock.org/~fair/opinion/identd.html

Re: 3.1 and Net::Ident

[Steve Martin]

OK, I'll forget identd...

Sometimes it is hard for me to resist trying things like that  
"because they are there".


On Aug 13, 2005, at 5:33 PM, John Rudd wrote:



On Aug 13, 2005, at 3:12 PM, Steve Martin wrote:



2) Does anyone know a good working identd



this is a contradiction in terms.  "Good" and "working" cannot  
coexist with "identd".  Ok, I guess you could say that "an identd  
which accurately implements the identd protocol is 'working'", but  
that's like saying that "a car that is DESIGNED to explode into a  
fireball when you drive it, is functioning perfectly when it burns  
you to death."  (which then suggests that "identd is the pinto of  
identification/authentication protocols" would be a good  
description)  But, still, "good" doesn't belong in there.


identd in general is worthless.  So, unless SA is using 'you run  
identd, so you must be a con-artist' as a means of increasing an SA  
score, I would avoid using anything based on identd.


http://www.clock.org/~fair/opinion/identd.html




--
Steve Martin  http://www.cheezmo.com/
Smart Calibration, LLC   http://www.smartcalibration.com/
The Widescreen Movie Centerhttp://www.widemovies.com/
Letterboxed Movie TV Schedule  http://www.widemovies.com/lbx.html



smime.p7s
Description: S/MIME cryptographic signature

How do I install Mail-SpamAssassin-3.0.4.tar.gz at perl 5.005 env.

[Atami Org.]

Dear;

I like to install Mail-SpamAssassin-3.0.4.tar.gz at my Cobalt Raq4 server.  
Cobalt Raq4 has perl 5.005.  Mail-SpamAssassin looks require perl 5.6.1. 
However I am not sure how upgrade perl of Cobalt Raq4.
Then, I am try to install Mail-SpamAssassin-3.0.4.tar.gz at the perl 5.005 env.
But not suuccess yet.

Both instarations on "perl -MCPAN -e shell" and on "gunzip -c", 
same error msgs are as follow;

## Can't locate warnings.pm in @INC (@INC contains: 
/usr/lib/perl5/5.00503/i386-lin
## ux /usr/lib/perl5/5.00503 /usr/lib/perl5/site_perl/5.005/i386-linux 
/usr/lib/per
## l5/site_perl/5.005 .) at Makefile.PL line 5.

Please advice me How I install Mail-SpamAssassin-3.0.4.tar.gz at the perl 5.005 
environment.

Eiji Hamano


HyperSystems Corp.  [EMAIL PROTECTED]

Re: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!

[Theo Van Dinter]

On Sat, Aug 13, 2005 at 03:07:14PM +0530, Ramprasad A Padmanabhan wrote:
> When I build the rpm from the spec file ( on fedora core 3 ) the
> spamassassin-tools rpm is not created. Was it not a part of SA.

The tools RPM was deprecated.  There was very little in there that wasn't
development related, which is better taken out of SVN or the tarball,
so ...

-- 
Randomly Generated Tagline:
"Derive the Big Bang Theory from F=ma."- Michelle about her Physics class


pgpXWSKzDY6K5.pgp
Description: PGP signature

Re: helo lookup before rDNS [Was: Faster rDNS lookups]

[List Mail User]

>...
>
>List Mail User wrote:
>> 
>>  Certainly not (no dunce cap).  Public airing of ideas generally
>> has merit - quite possibly this idea can be refined to something similar
>> that will provide a benefit (I admit, I have not given it a lot of thought).
>> 
>
>one thing that an MTA can do is to delay its rDNS lookup until it gets 
>the helo name. then do a forward lookup on the helo name.
>if one of the returned IPs matches the client IP, there will be no need 
>for the classical double lookup.
>
>if one assumes that most helo names are correct, then this would be a 
>minor improvement. and the benefit is that it automatically checks for 
>helo validity (I don't mean one would reject the helo, but he knows if 
>it's good or not).
>
>(I think iMail does something like this, but I can't say for sure).
>
A quick check of my own stats shows (after client IP blocking), that
about 35% of HELO/EHLO arguments are bogus - i.e. don't resolve or belong
to a domain unrelated to the client or sender IP.  As a result, I have things
configured to always do the double lookup, but only log a message when they
don't match (i.e. not "strict" FCrDNS);  Still > 25% of all EHLO/HELO arguments
have no rDNS or aren't FQDNs;  Some of this is list-washing - For a year ago,
the numbers were almost double these (much worse).  And the number or client
IPs lacking rDNS is > 40% (again, after blocking on client IPs - presumably
for people who don't block at the MTA level on client IPs, the numbers are
much worse).  Of course, there is a very large overlap between requests with
invalid or bad HELO/EHLO names and invalid rDNS for the client IP (i.e. liars
lie about everything usually, at least for SMTP transactions).


Paul Shupak
[EMAIL PROTECTED]

How to reenable DCC and Razor (was RE: ANNOUNCE: SpamAssassin 3.1.0-rc1 release candidate available!)

[Dan Kohn]

Justin said:

>- - Razor: disable Razor2 support by default per our policy, since the
>  service is not free for non-personal use.  It's trivial to reenable.

Could you please change that to:

"It's trivial to reenable by uncommenting the appropriate lines in
/etc/mail/spamassassin/v310.pre".  Given that I just spent 30 minutes
figuring that out, I might even quibble with the word trivial.

I updated http://wiki.apache.org/spamassassin/UpgradeTo310 with this
info.

  - dan
--
Dan Kohn