Re: backport 3.4.0 Ubuntu 12.04 TLS

[Patrick Domack]

I had no reason to backport it, xenial didn't exist when 3.4.1 was  
released, and there is no need to backport a package just for a new  
source code change, just apply the current package to the new source  
code.



Quoting Marcus Schopen <li...@localguru.de>:


Hi Patrick,

Am Donnerstag, den 15.09.2016, 22:02 -0400 schrieb Patrick Domack:

Sounds like a lot of work for an old spamassassin version.

https://launchpad.net/%7Epatrickdk/+archive/ubuntu/production/+sourcepub/5219815/+listing-archive-extra


H ... do you think better backporting 3.4.1 from Xenial? Does it run
on Ubuntu 12.04 LTS and 14.04 LTS?

Ciao!

Re: backport 3.4.0 Ubuntu 12.04 TLS

[Patrick Domack]

Sounds like a lot of work for an old spamassassin version.

https://launchpad.net/%7Epatrickdk/+archive/ubuntu/production/+sourcepub/5219815/+listing-archive-extra


Quoting Marcus Schopen :


Hi,

I've backported 3.4.0 from Ubuntu 14.04 TLS for Ubuntu 12.04 LTS (perl
5.14.2), which comes with very old version 3.3.2 (can't upgrade the
complete host right now). Before installing it: is there anything to be
aware of, beside better wiping bayes database and starting fresh? [1]

Ciao
Marcus

[1]
http://svn.apache.org/repos/asf/spamassassin/branches/3.4/build/announcements/3.4.0.txt

Re: Quick question about training...

[Patrick Domack]

Quoting Kevin Miller kevin.mil...@juneau.org:

When a fresh spam flood comes in, sometimes 50 or more of my users  
will get hit with the same message - just a different user in the  
To: line.  When one trains the bayes database, is there a  
significant difference between training on all 50+ or just grabbing  
a few of the messages and training on them?  Will bayes be more  
convinced of the spaminess of a particular message if it sees dozens  
rather than a couple?


The last flood I had, bayes started marking them as spam, after the  
3rd email hit, each email I saw the score go up a few a point or so,  
till they all started to get marked as spam.


I only even noticed this, cause the first user to get hit, complained  
they got a spam email.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Lucio Chiappetti lu...@lambrate.inaf.it:


On Mon, 9 Jun 2014, Rob McEwen wrote:


Domain age is a good metric to factor in. But I'm always fascinated with
some people's desire to block all messages with extremely new domains.



Keep in mind that many large and famous businesses... who have fairly
good mail sending practices... sometimes launch a new products complete
with links to very newly registered domains. Same is often true for ...


Or for public research organizations which are often reformed by  
the Government, with change of name and consequential change of  
domain (even if the IP of the DNS and MX is unchanged :-))


Take my case, I've been working at the same physical place since  
1982 and the name of my institute or of the organization it belongs  
to has changed about 7 times.   And it does not only occur in this  
country (Italy), I've seen (mainly dealing with mailing list  
re-subscriptions) similar changes at least in France and UK ..


Not saying this doesn't happen. But also, how often does someone  
register a domain, move all their users to the new domain, have the  
server all reconfigured to use this new domain, all within the first  
day?


I know personally, I have always taken at least a week to do it,  
mainly just to make sure I didn't miss anything, and to double check  
everything as I go. The Last thing I do is force users to change their  
email addresses.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Axb axb.li...@gmail.com:


On 06/10/2014 12:28 PM, Patrick Domack wrote:


Not saying this doesn't happen. But also, how often does someone
register a domain, move all their users to the new domain, have the
server all reconfigured to use this new domain, all within the first day?

I know personally, I have always taken at least a week to do it, mainly
just to make sure I didn't miss anything, and to double check everything
as I go. The Last thing I do is force users to change their email
addresses.


domains don't have to have users on them.

coming up film sites, parents setting up sweet16 sites, wedding  
sites, cosmetic vendors, art festivals, etc, etc, TONS of etc,  use  
new domains for marketing, often seen in mail even before DNS is has  
fully replicated.


Yes, anything is possible.

I have yet, to see any ligit email though, I'm sure I will a few times  
a year. I have seen email before dns/whois even is updated.


But personally, one should work to establish their reputation before  
blasting out emails. You have to do this when moving ip addresses, and  
also for domains, though not as many servers track domain reputation  
as much as ip reputation.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Axb axb.li...@gmail.com:


On 06/10/2014 04:14 PM, Patrick Domack wrote:


Quoting Axb axb.li...@gmail.com:


On 06/10/2014 12:28 PM, Patrick Domack wrote:


Not saying this doesn't happen. But also, how often does someone
register a domain, move all their users to the new domain, have the
server all reconfigured to use this new domain, all within the first
day?

I know personally, I have always taken at least a week to do it, mainly
just to make sure I didn't miss anything, and to double check everything
as I go. The Last thing I do is force users to change their email
addresses.


domains don't have to have users on them.

coming up film sites, parents setting up sweet16 sites, wedding
sites, cosmetic vendors, art festivals, etc, etc, TONS of etc,  use
new domains for marketing, often seen in mail even before DNS is has
fully replicated.


Yes, anything is possible.

I have yet, to see any ligit email though, I'm sure I will a few times a
year. I have seen email before dns/whois even is updated.

But personally, one should work to establish their reputation before
blasting out emails. You have to do this when moving ip addresses, and
also for domains, though not as many servers track domain reputation as
much as ip reputation.


you honestly expect marketing drones to understand/care?

All  URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check   
track domain reputation otherwise they'd be unusable.


Their listings are not blind - they all have their secret sauce to  
process before listing a domain.


So, we are unwilling to look into any new ideas cause there might be  
an issue? that we haven't scoped or checked into?


How is progress made, when your unwilling to check and collect stats  
and figures.


This was meant to be another metric that could, or might not be used.  
I personally got tired of everyone talking about it, many shooting it  
down, and NO ONE actually looking into it, and reporting real stats  
about it.


Personally, I thought it was a pointless test, but it is proving  
useful. Does it single handily solve spam and has no side effects? No,  
but if you find that solution, you will be rich.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Rob McEwen r...@invaluement.com:


On 6/10/2014 10:21 AM, Axb wrote:

All  URI BLs I know of (SURBL/URIBL/DBL/Invaluement/etc) check  track
domain reputation otherwise they'd be unusable.
Their listings are not blind - they all have their secret sauce to
process before listing a domain.


Absolutely. As Axb and KAM and others stated, a very young domain age is
too dangerous to outright block or score high on... but might be a good
factor or good for combining with other rules.

Also, if anyone does see spam that contain domains in the clickable
links where that spam should have been blocked, but was not... then
check the domain contained within the spam again the lookup found at
http://multirbl.valli.org and/or http://mxtoolbox.com/blacklists.aspx
(some months ago, MX Toolbox upgraded their system to check domains
against URI/domain blacklists. In some cases, this could have been a
game of inches where your user caught the tip of the spear and
received the very first spams in a spam campaign that otherwise was
quickly listed by the well known URI BLs. However, you may find that one
or two good URI BLs are simply not implemented in your system--where
that would have made all the difference! Those lookup forms will point
you in the right direction.

The same can also be true for checking sending IPs--then reviewing your
current mix of sender's IP dnsbls (aka RBLs).

Of course, don't fall into the trap of using a BL that catches much, but
has too many FPs. But the list of URI BLs that Axb gave above are all
extremely low-FP URI blacklists.


In my case, Yes, I am using all the above and more.

I had a user that normally never gets spam, started receiving around  
20 per day, that where not marked.


I found that around 18per day of these where from a new domain. These  
did appear on multirbl.valli.org lists, like invaluement, and uribl  
after a day or two. I hadn't seen them hit dbl or surbl though.


This is what caused me to seriously look into if this method was  
useful, just greylisting the email for a day, would cause a huge  
benifit, for new domains, without causing an extreem backlash.


There are all kinds of way to use the infomation. I just don't  
understand why people are so against it, cause it's not 100% foolproof.


Nothing about marking spam is 100% foolproof.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Axb axb.li...@gmail.com:


On 06/10/2014 05:11 PM, Patrick Domack wrote:

There are all kinds of way to use the infomation. I just don't
understand why people are so against it, cause it's not 100% foolproof.


Nobody is against the idea, problem is scalability and trust.
To make domain age usable, the BLs I mentioned make use of it as  
well as many other daata points to gain trust that a listing won'  
tbite the globe, as well as they can.


Consider certain factors wich *can* contribute to delay in listings  
produce a positive hit,for example, mirror lag due to rsync,  
negative TTL, etc. as reasosn why you seem to see these domains  
being listed after you got the spams.

(If your size/budget permits, datafeeds would probably help a lot)

For a small site doing a few whois lookups/hour it may work, but  
what if suddenly an ISP/ASP doing many thousands of msgs/sec would  
implement this?


I did consider those factors, and they where not the problem.

I do rsync the data feeds locally, and feeds did not contain the  
lookups till hours later.

It wasn't a negative ttl issue, as the ttl is non-existant for these lookups

I fail to understand why you would be doing thousands of whois lookups  
per second. You see that many new domain names per second?
Mostly it's the same domain names over and over again, and a few new  
ones per day.
Domains don't expire, moved around, and updated a lot, and even if it  
did, that isn't really much a concern. To cache this infomation for  
atleast 3 years, would be fine, likely even longer.


Also, the point of having a central body do this, would cause the  
cached results to be even better, and less lookups needed.


I'm not a huge isp, but I don't seem to be any where near as tiny as  
you suggest.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Axb axb.li...@gmail.com:


On 06/10/2014 06:51 PM, Patrick Domack wrote:


Quoting Axb axb.li...@gmail.com:


On 06/10/2014 05:11 PM, Patrick Domack wrote:

There are all kinds of way to use the infomation. I just don't
understand why people are so against it, cause it's not 100% foolproof.


Nobody is against the idea, problem is scalability and trust.
To make domain age usable, the BLs I mentioned make use of it as well
as many other daata points to gain trust that a listing won' tbite the
globe, as well as they can.

Consider certain factors wich *can* contribute to delay in listings
produce a positive hit,for example, mirror lag due to rsync, negative
TTL, etc. as reasosn why you seem to see these domains being listed
after you got the spams.
(If your size/budget permits, datafeeds would probably help a lot)

For a small site doing a few whois lookups/hour it may work, but what
if suddenly an ISP/ASP doing many thousands of msgs/sec would
implement this?


I did consider those factors, and they where not the problem.

I do rsync the data feeds locally, and feeds did not contain the lookups
till hours later.
It wasn't a negative ttl issue, as the ttl is non-existant for these
lookups


When you come up with a couple of such cases, please post them here  
as quickly as you can so BL ops or users lurking here can check  
their  logs and maybe compare results.



I fail to understand why you would be doing thousands of whois lookups
per second. You see that many new domain names per second?
Mostly it's the same domain names over and over again, and a few new
ones per day.


You do lookups on URIS in your mailflow right? so you do it for HAM/SPAM


Domains don't expire, moved around, and updated a lot, and even if it
did, that isn't really much a concern. To cache this infomation for
atleast 3 years, would be fine, likely even longer.


Check  keep track of daily changes and you'll be surprised how  
often stuff gets moved around.



Also, the point of having a central body do this, would cause the cached
results to be even better, and less lookups needed.
if found...ok. if not found negative TTL applies and short TTL means  
evne more lookups.



I'm not a huge isp, but I don't seem to be any where near as tiny as you
suggest.


I'm not assuming/suggesting anything



I'm not interested in how much stuff gets moved around, if a domain  
has been registered, and been moved around, it will have a reputation.  
So I don't really care if the data is 100% accurate or up to date.


I'm not sure why negative ttl would cause more whois lookups? yes it  
will cause more dns lookups, but those are not an issue, expecially if  
you have a local data feed available, if you do, set your negative ttl  
to 5seconds.

RE: SPAM from a registrar

[Patrick Domack]

I have been tracking this for about 2 weeks now myself.

Comparing my list of new domains, shows that DOB seems to pick them up  
after they are 2 days old.


I also tried to compair my list to fresh.spameatingmonkey.net, but  
none of my domains in the 0-5days old would get a match for com/net  
domains. I do get some hits for info and us though. But it's normally  
com and a few us that are on my lists.


I am currently doing a whois lookups for about 30 tld's, and tracking  
their time and registar. I do minimize the lookups.


I am currently seeing, about 2 .asia, 2 .uk, and then around 100 .com  
(all the .com are ENOM) sending email to me, with an age 1day old.  
This is pretty consistant day to day.






Have you looked into Day old bread?   
http://wiki.apache.org/spamassassin/Rules/URIBL_RHS_DOB


 ...Kevin
--
Kevin Miller
Network/email Administrator, CBJ MIS Dept.
155 South Seward Street
.Juneau, Alaska 99801
Phone: (907) 586-0242, Fax: (907) 586-4500
Registered Linux User No: 307357
-Original Message-
From: James B. Byrne [mailto:byrn...@harte-lyne.ca]
Sent: Wednesday, May 14, 2014 8:52 AM
To: users@spamassassin.apache.org
Subject: SPAM from a registrar

This AM we received (and are continuing to receive) numerous spam  
messages from multiple domains
that were all registered today (2014-05-14) with a company called  
enom, inc.  This firm is
also the registrar for the the mail server domain BOSJAW.com that is  
ending some if not all

of the UCEM.  That server is hosted in CZ.

It seems likely that this is a planned UCEM campaign designed to use  
disposable domains, probably
registered with stolen credit cards or some other form of fraud, in  
order to escape blacklisting

services.  No doubt by tomorrow they will be abandoned.

Is there any test to check how long a domain name has been in  
existence and set a spam score

with that information?

Along the same lines, is there any test to determine the country of  
origin of the IP address

in the last hop before it connects to our servers?

- End forwarded message -

---BeginMessage---

I have been tracking this for about 2 weeks now myself.

Comparing my list of new domains, shows that DOB seems to pick them up  
after they are 2 days old.


I also tried to compair my list to fresh.spameatingmonkey.net, but  
none of my domains in the 0-5days old would get a match.


I am currently doing a whois lookups for about 30 tld's, and tracking  
their time and registar. I do minimize the lookups.


I am currently seeing, about 2 .asia, 2 .uk, and then around 100 .com  
(all the .com are ENOM) sending email to me, with an age 1day old.  
This is pretty consistant day to day.






Have you looked into Day old bread?   
http://wiki.apache.org/spamassassin/Rules/URIBL_RHS_DOB


 ...Kevin
--
Kevin Miller
Network/email Administrator, CBJ MIS Dept.
155 South Seward Street
.Juneau, Alaska 99801
Phone: (907) 586-0242, Fax: (907) 586-4500
Registered Linux User No: 307357
-Original Message-
From: James B. Byrne [mailto:byrn...@harte-lyne.ca]
Sent: Wednesday, May 14, 2014 8:52 AM
To: users@spamassassin.apache.org
Subject: SPAM from a registrar

This AM we received (and are continuing to receive) numerous spam  
messages from multiple domains
that were all registered today (2014-05-14) with a company called  
enom, inc.  This firm is
also the registrar for the the mail server domain BOSJAW.com that is  
ending some if not all

of the UCEM.  That server is hosted in CZ.

It seems likely that this is a planned UCEM campaign designed to use  
disposable domains, probably
registered with stolen credit cards or some other form of fraud, in  
order to escape blacklisting

services.  No doubt by tomorrow they will be abandoned.

Is there any test to check how long a domain name has been in  
existence and set a spam score

with that information?

Along the same lines, is there any test to determine the country of  
origin of the IP address

in the last hop before it connects to our servers?

---End Message---

Re: SPAM from a registrar

[Patrick Domack]

Quoting Kevin A. McGrail kmcgr...@pccc.com:


On 6/9/2014 1:23 PM, Patrick Domack wrote:

I have been tracking this for about 2 weeks now myself.

Comparing my list of new domains, shows that DOB seems to pick them  
up after they are 2 days old.


I also tried to compair my list to fresh.spameatingmonkey.net, but  
none of my domains in the 0-5days old would get a match for com/net  
domains. I do get some hits for info and us though. But it's  
normally com and a few us that are on my lists.


I am currently doing a whois lookups for about 30 tld's, and  
tracking their time and registar. I do minimize the lookups.


I am currently seeing, about 2 .asia, 2 .uk, and then around 100  
.com (all the .com are ENOM) sending email to me, with an age 1day  
old. This is pretty consistant day to day.
I wonder how we can use DNS, an RBL and distributed lookups to get  
the age of domains AND share the information so it's centrally  
available...


That could be easily done. Only issue is, if you trust the distributed  
lookups to have accurate infomation.
I suppose we could build in a trust system, where if enough  
distributed clients upload the same info, it could be trusted.


This could work out pretty good. Each dns-rbl cluster could run with  
their own shared database, and you can cross-publish to other dns-rbl  
clusters, and set your own trust rating, depending on how many copies  
you get, on if you trust the info, or do your own whois lookup for the  
info.


Bad thing is, I wonder how fast these are hammers out, and if the  
trust and replication wouldn't matter, due to latency.

Re: SPAM from a registrar

[Patrick Domack]

Quoting Kevin A. McGrail kmcgr...@pccc.com:


On 6/9/2014 2:24 PM, Patrick Domack wrote:

Quoting Kevin A. McGrail kmcgr...@pccc.com:


On 6/9/2014 1:23 PM, Patrick Domack wrote:

I have been tracking this for about 2 weeks now myself.

Comparing my list of new domains, shows that DOB seems to pick  
them up after they are 2 days old.


I also tried to compair my list to fresh.spameatingmonkey.net,  
but none of my domains in the 0-5days old would get a match for  
com/net domains. I do get some hits for info and us though. But  
it's normally com and a few us that are on my lists.


I am currently doing a whois lookups for about 30 tld's, and  
tracking their time and registar. I do minimize the lookups.


I am currently seeing, about 2 .asia, 2 .uk, and then around 100  
.com (all the .com are ENOM) sending email to me, with an age  
1day old. This is pretty consistant day to day.
I wonder how we can use DNS, an RBL and distributed lookups to get  
the age of domains AND share the information so it's centrally  
available...


That could be easily done. Only issue is, if you trust the  
distributed lookups to have accurate infomation.
I suppose we could build in a trust system, where if enough  
distributed clients upload the same info, it could be trusted.


This could work out pretty good. Each dns-rbl cluster could run  
with their own shared database, and you can cross-publish to other  
dns-rbl clusters, and set your own trust rating, depending on how  
many copies you get, on if you trust the info, or do your own whois  
lookup for the info.


Bad thing is, I wonder how fast these are hammers out, and if the  
trust and replication wouldn't matter, due to latency.
Thanks for weighing in.  These are all issues we've solved with  
other RBLs via rsync of the data and I want to keep the hurdle low  
for implementation so you are write about the trust rating, etc.


Well, while rsync works, you need a source, if the source was a feed  
from the tld's themselfs, that would work just fine.


The main thing I'm more worried about here is making sure new domains  
are noticed. Atleast I have seen 1day old domains send a lot more  
spam than 2-3day old ones.


So the new, unknown domain, is going be more important to lookup.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting David F. Skoll d...@roaringpenguin.com:


On Mon, 9 Jun 2014 11:51:21 -0700 (PDT)
John Hardin jhar...@impsec.org wrote:


 So there is merit in building a distributed look-up system using SA.



Distributed lookup of *what*, though? Can you clarify that part of
your idea? Are you referring to distributed whois queries for a
domain name, to determine its age?


Well, here's how it could be done.  Imagine someone runs a DNS zone
for newdomain.example.net.  You want to see if example.org is a new
domain, so you look up a TXT record for example.org.newdomain.example.net.

The DNS software that serves the zone newdomain.example.net runs
the following pseudo-code when example.org is looked up:

IF example.org is in my database
THEN
   return the TXT record associated with example.org
   update the last-looked-up time for example.org
ELSE
   generate a TXT record of the form MMDDHHMMSS corresponding to  
current time (UTC)

   insert it in the database
   return it
ENDIF

A background job will periodically clean out domains that haven't been
queried in a long time.

The clever part is that once lots of sites begin using this in their
SA setups, we'll very quickly build up quite an accurate database of
newly-seen domains that's completely independent of any registrar for
a data source.

Yes, spammers can poison it by specifically looking up a domain,
waiting a couple of days, and then spamming.  But I think most won't bother
(witness how effective greylisting still is.)

Furthermore, you can ignore all but the first few hundred lookups before you
enter the TXT record in the database; this will make it more expensive
for spammers to poison the data.  Or you could not enter a record in the
database until it has been looked up from 100 different IP addresses... I
can think of a few other countermeasures.

So who's volunteering to do this? :)

Regards,

David.


The point was, I have already done this, and have it in production. I  
did this cause this subject keeps coming up from time to time, and I  
was personally interested to see the results of it.


And I do agree with Rob McEwen on many points. And I would be  
hisentant to outright block. But so far, and I doubt much in real  
usage, and haven't found any yet, any issues with blocking 1day  
outright.


But then the only way to be completely sure of that, will be time.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Matthias Leisi matth...@leisi.net:


On Mon, Jun 9, 2014 at 8:43 PM, Kevin A. McGrail kmcgr...@pccc.com wrote:



I think the core issue is that age of domains is a good indicator of spam.
 So there is merit in building a distributed look-up system using SA.

I have more ideas than resources, of course...



I repeat my question: which domain? HELO, MAIL FROM, From:, ...?

-- Matthias


HELO hasn't matched anything in my tests.

MAIL FROM has matched many, though the helo's are always a different domain

From I have only started doing yesterday, and not sure exactly how I  
will track them. Likely just wait a few days, and check my ham/spam  
folders and compare what rules where hit.

Re: Domain ages (was Re: SPAM from a registrar)

[Patrick Domack]

Quoting Matthias Leisi matth...@leisi.net:


On Mon, Jun 9, 2014 at 11:31 PM, Richard Doyle lists...@islandnetworks.com
wrote:



A caching whois client (jwhois, for example) can significantly reduce
the volume of queries.



You will need to query potentially hundreds or thousands of domains *per
day* - mostly throw away domains from spammers.

1) What are the typical rate limits on public whois servers?
2) How to protect against attackers sending random non-existant domain
names your way, thus ensuring you hit rate limites early?
3) How to parse the myriads of formats sent by whois servers?
4) How do you handle TLDs which do not publish registration dates, like eg
.de? (At least they did not last time I checked.)

Whois is not a feasible data source.

-- Matthias


1) I dunno, but I am doing around 15k lookups a day, from a single ip,  
without getting limited/blocked
2) This is hard, and I don't know, currently the postfix reject  
unknown sender helps solve this for me, but won't for dns based lookups

3) This, while annoying, is solved in my code, not too hard
4) These I just don't bother doing lookups for, there is no solution,  
other than to let them bypass this system, or rate them via seen  
before method.