Re: Shortcircuit Question

2008-05-08 Thread Matt Kettler

Clayton Keller wrote:
I have been reading throught the Shortcircuit manpage as well as some 
articles within the Wiki, and the manner in which I see it performing 
within our install does not seem to coincide with how I am reading and 
presumably understanding it to work.


First off, we are using SpamAssassin 3.2.4 provided by the rpmforge 
batch of rpm's.


I have about a dozen priorities specified mainly handling URIBL, 
SURBL, as well as DCC, Razor, Pyzor and Bayes.


What I am seeing is that although the first shortcircuit rule hits and 
scores appropriately. Subsequent short circuit rules will continue to 
fire. The scores themselves are then totaled along with the original 
scores for the rules.


My understanding of how the shortcircuit should work is that once a 
shortcircuit is triggered any subsequent rules should be bypassed and 
the message wither classified as spam/ham or if set to on, it would 
use the current score specified for the rule.


As a for instance:

I have the following:

priority URIBL_BLACK-500
priority URIBL_JP_SURBL -498
priority URIBL_SC_SURBL -488
priority URIBL_OB_SURBL -487

priority SC_URIBL_SURBL -480
priority SC_URIBL_SBL   -479

priority RAZOR2_CHECK   -450
priority DCC_CHECK  -449
priority PYZOR_CHECK-448

priority SC_URIBL_HASH  -440

meta SC_URIBL_SURBL(URIBL_BLACK  (URIBL_SC_SURBL
   || URIBL_JP_SURBL || URIBL_OB_SURBL))

meta SC_URIBL_SBL((URIBL_BLACK || URIBL_SC_SURBL ||
URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


meta SC_URIBL_HASH((URIBL_BLACK || URIBL_SC_SURBL ||
 URIBL_JP_SURBL || URIBL_OB_SURBL)  
(RAZOR2_CHECK || DCC_CHECK || PYZOR_CHECK))



meta SC_URIBL_SBL   ((URIBL_BLACK || URIBL_SC_SURBL ||
  URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


shortcircuit SC_URIBL_SURBL spam
shortcircuit SC_URIBL_SBL   spam
shortcircuit SC_URIBL_HASH spam

score SC_URIBL_SURBL100.00
score SC_URIBL_HASH 100.00
score SC_URIBL_SBL  100.00


I do not have a recent debug to show but I can say that from the debug 
I do see the SC_URIBL_SURBL trigger, after the earlier priority rules 
are ran. However, the remaining priorities are then ran, and if 
meeting critera, the RAZOR, DCC, and PYZOR rules run and then the 
SC_URIBL_HASH rule would trigger. Thus giving a total score of 200 + 
the scores for the URIBL/SURBL scores that hit and if included the 
Razor, DCC, and Pyzor scores as well.


I was thinking after the SC_URIBL_SURBL was triggered remaining rules 
would not run, and the spam classification would take precendence.


Am I overlooking the obvious, have I misunderstood how the SC should 
work, is it something with the rpm that was released by rpmforge? Any 
thoughts or insight would be appreciated. 


SA is, rather fortunately, circumventing what you're trying to do 
because of how DNS is handled internally.


DO NOT try to split up the priority of DNS based tests. Priority and 
shortcircuiting is intended to be used on *fast* rules, not slow ones.


If you were successful, you would make the performance of SpamAssassin 
absurdly slow by serializing DNS queries. *OUCH*. SA normally runs these 
in parallel, and running them in serial would very seriously impact 
performance.


Currently, all DNS based tests run at their priority, but that only 
launches the DNS queries. All the results are gathered together at 
HARVEST_DNSBL_PRIORITY, which is currently set to 500. None of the rules 
will actually trigger until this point.











Re: possible idea for backscatter problem

2008-05-08 Thread Matt Kettler

.rp wrote:
One of the users (actually the boss) had the email address harvested and we got clobbered 
by backscatter. Looking at the emails of the various 'unable to deliver' type messages, I saw 
what these could be filtered on, but don't know how to write up and implement the rule 
outside of procmail. I don't want to use procmail for this since it I think it would be an 
expensive routine for procmail to run.


In the body of the 'unable to deliver' message, the original message is quoted. One of the 
lines quoted is the Message-ID: header from the original. The format of this line is always 
wrong as it does not contain the FQDN that our server appends to the end of the hash 
number , following the '@' symbol .


So, need a rule that would parse the Message-ID: in the body (or attachment) and not 
header, and look for the @FQDN 
Is this rule already out in the wild?
  
(note: your To: was the bogofilter list, but this appeared on 
spamassassin-users as well.. It looks like you bcc'ed the SA list.  
Anyway, I'm answering on the SA list because that's where I picked up 
the message from)


Not that I know of, but it would be fairly quick as a spamassassin rule.

You'd likely need a meta of some sort.

Theoretically, something like this should work. I'm leveraging some of 
the stock ruleset here, by reusing BOUNCE_MESSAGE to detect if the 
message really is a bounce, make sure it is in your ruleset.


--

body __BOUNCE_HASMSGID /Message-ID:/
body __BOUNCE_MINE   /Message-ID:.{1,[EMAIL PROTECTED]/

meta BOUNCE_NOTMINE (BOUNCE_MESSAGE  __BOUNCE_HASMSGID  !__BOUNCE_MINE)
score BOUNCE_NOTMINE 0.1
describe BOUNCE_NOTMINE  Appears to be a bounce with a message without a 
valid message ID.


--

Modify the example.com part to suit your needs.

Be sure to run spamassassin --lint after adding it.

Note: I've intentionally set to score to 0.1, as this rule isn't tested. 
It theoretically should do its job, but make sure it works properly 
before you increase the score.







Multiple X-Envelope-From and SPF

2008-05-08 Thread ram
At the MTA( postfix) I am inserting X-Envelope-From: 
If The mail had already a X-Envelope-From before landing at my MTA  then
There would be multiple lines of these 

Then SA refuses to do SPF for these  messages , and I can see in my
debug logs 


-
[18469] dbg: message: X-Envelope-From header found after 1 or more
Received lines, cannot trust envelope-from
[18469] dbg: spf: relayed through one or more trusted relays, cannot use
header-based Envelope-From, skipping
--

How do I avoid this situation. 


Thanks
Ram




Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread Justin Mason

Kevin W. Gagel writes:
 - Original Message -
 Marc Perkel wrote:
  Looking for a few volunteers who want to reduce their spambot spam and 
  at the same time help me track spambots for my black list. This is free 
  and mutual benefit. I (junkemailfilter.com) want to be your highest 
  numbered fake MX record. Here's how you would configure your domain:
 
 A generous offer and an admirable effort. But if you think I or my 
 clients are going to route mail to your servers you are mistaken. Even 
 if I knew you personally, I don't think ethics or common sense would 
 allow me to do so.
 
 DAve
 
 Personally I use the honeypot project. I recomend it. See:
 http://www.projecthoneypot.org
 For info.

btw, if you have spare spamtrap *domains* -- not just /etc/aliases
forwards -- we'd love to get a couple pointed at the SpamAssassin
spamtraps...

--j.


Re: possible idea for backscatter problem

2008-05-08 Thread Justin Mason

Matt Kettler writes:
 .rp wrote:
  One of the users (actually the boss) had the email address harvested and we 
  got clobbered 
  by backscatter. Looking at the emails of the various 'unable to deliver' 
  type messages, I saw 
  what these could be filtered on, but don't know how to write up and 
  implement the rule 
  outside of procmail. I don't want to use procmail for this since it I think 
  it would be an 
  expensive routine for procmail to run.
 
  In the body of the 'unable to deliver' message, the original message is 
  quoted. One of the 
  lines quoted is the Message-ID: header from the original. The format of 
  this line is always 
  wrong as it does not contain the FQDN that our server appends to the end of 
  the hash 
  number , following the '@' symbol .
 
  So, need a rule that would parse the Message-ID: in the body (or 
  attachment) and not 
  header, and look for the @FQDN 
  Is this rule already out in the wild?

 (note: your To: was the bogofilter list, but this appeared on 
 spamassassin-users as well.. It looks like you bcc'ed the SA list.  
 Anyway, I'm answering on the SA list because that's where I picked up 
 the message from)
 
 Not that I know of, but it would be fairly quick as a spamassassin rule.
 
 You'd likely need a meta of some sort.
 
 Theoretically, something like this should work. I'm leveraging some of 
 the stock ruleset here, by reusing BOUNCE_MESSAGE to detect if the 
 message really is a bounce, make sure it is in your ruleset.

Actually, that's overkill -- BOUNCE_MESSAGE _already_ does this.

the VBounce plugin is intended to catch backscatter -- bounces in response
to mail you didn't send -- so it'll ignore bounces in response to mail you
_did_ send, by parsing the bounced message's Received: headers and looking
for the mailserver's name in there.

See the FAQ for more info...

--j.


Re: Shortcircuit Question

2008-05-08 Thread Justin Mason

Matt Kettler writes:
 Clayton Keller wrote:
  I have been reading throught the Shortcircuit manpage as well as some 
  articles within the Wiki, and the manner in which I see it performing 
  within our install does not seem to coincide with how I am reading and 
  presumably understanding it to work.
 
  First off, we are using SpamAssassin 3.2.4 provided by the rpmforge 
  batch of rpm's.
 
  I have about a dozen priorities specified mainly handling URIBL, 
  SURBL, as well as DCC, Razor, Pyzor and Bayes.
 
  What I am seeing is that although the first shortcircuit rule hits and 
  scores appropriately. Subsequent short circuit rules will continue to 
  fire. The scores themselves are then totaled along with the original 
  scores for the rules.
 
  My understanding of how the shortcircuit should work is that once a 
  shortcircuit is triggered any subsequent rules should be bypassed and 
  the message wither classified as spam/ham or if set to on, it would 
  use the current score specified for the rule.
 
  As a for instance:
 
  I have the following:
 
  priority URIBL_BLACK-500
  priority URIBL_JP_SURBL -498
  priority URIBL_SC_SURBL -488
  priority URIBL_OB_SURBL -487
 
  priority SC_URIBL_SURBL -480
  priority SC_URIBL_SBL   -479
 
  priority RAZOR2_CHECK   -450
  priority DCC_CHECK  -449
  priority PYZOR_CHECK-448
 
  priority SC_URIBL_HASH  -440
 
  meta SC_URIBL_SURBL(URIBL_BLACK  (URIBL_SC_SURBL
 || URIBL_JP_SURBL || URIBL_OB_SURBL))
 
  meta SC_URIBL_SBL((URIBL_BLACK || URIBL_SC_SURBL ||
  URIBL_JP_SURBL || URIBL_OB_SURBL)  
  URIBL_SBL)
 
  meta SC_URIBL_HASH((URIBL_BLACK || URIBL_SC_SURBL ||
   URIBL_JP_SURBL || URIBL_OB_SURBL)  
  (RAZOR2_CHECK || DCC_CHECK || PYZOR_CHECK))
 
 
  meta SC_URIBL_SBL   ((URIBL_BLACK || URIBL_SC_SURBL ||
URIBL_JP_SURBL || URIBL_OB_SURBL)  
  URIBL_SBL)
 
  shortcircuit SC_URIBL_SURBL spam
  shortcircuit SC_URIBL_SBL   spam
  shortcircuit SC_URIBL_HASH spam
 
  score SC_URIBL_SURBL100.00
  score SC_URIBL_HASH 100.00
  score SC_URIBL_SBL  100.00
 
 
  I do not have a recent debug to show but I can say that from the debug 
  I do see the SC_URIBL_SURBL trigger, after the earlier priority rules 
  are ran. However, the remaining priorities are then ran, and if 
  meeting critera, the RAZOR, DCC, and PYZOR rules run and then the 
  SC_URIBL_HASH rule would trigger. Thus giving a total score of 200 + 
  the scores for the URIBL/SURBL scores that hit and if included the 
  Razor, DCC, and Pyzor scores as well.
 
  I was thinking after the SC_URIBL_SURBL was triggered remaining rules 
  would not run, and the spam classification would take precendence.
 
  Am I overlooking the obvious, have I misunderstood how the SC should 
  work, is it something with the rpm that was released by rpmforge? Any 
  thoughts or insight would be appreciated. 
 
 SA is, rather fortunately, circumventing what you're trying to do 
 because of how DNS is handled internally.
 
 DO NOT try to split up the priority of DNS based tests. Priority and 
 shortcircuiting is intended to be used on *fast* rules, not slow ones.
 
  If you were successful, you would make the performance of SpamAssassin 
 absurdly slow by serializing DNS queries. *OUCH*. SA normally runs these 
 in parallel, and running them in serial would very seriously impact 
 performance.
 
 Currently, all DNS based tests run at their priority, but that only 
 launches the DNS queries. All the results are gathered together at 
 HARVEST_DNSBL_PRIORITY, which is currently set to 500. None of the rules 
 will actually trigger until this point.

actually Matt, you're wrong ;)  if some of the network rules are
at a higher priority than others, and are used in shortcircuit rules,
SpamAssassin 3.2.x will indeed sleep until the results of those rules
arrive.

The idea is that, if you have the memory to support that degree of
concurrency, you can make a local policy decision to do that, instead
of doing the lookups at the MTA level which does effectively the
same thing.

This wait is logged, so you can spot it with --debug on.

--j.


Re: possible idea for backscatter problem

2008-05-08 Thread Justin Mason

Henrik Krohns writes:
 On Thu, May 08, 2008 at 09:35:31AM +0100, Justin Mason wrote:
 
  the VBounce plugin is intended to catch backscatter -- bounces in response
  to mail you didn't send -- so it'll ignore bounces in response to mail you
  _did_ send, by parsing the bounced message's Received: headers and looking
  for the mailserver's name in there.
 
 I've been trying it for myself. One thing I don't like is that all
 null-sender mail is assumed to be bounces. It creates many FPs.

Yes.  What kind of null-sender mail from machines that are not in
whitelist_bounce_relays do you get?

 Also check https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5900

saw that...

--j.


Re: possible idea for backscatter problem

2008-05-08 Thread Justin Mason

Matt Kettler writes:
 Justin Mason wrote:
  Matt Kettler writes:

  .rp wrote:
  
  One of the users (actually the boss) had the email address harvested and 
  we got clobbered 
  by backscatter. Looking at the emails of the various 'unable to deliver' 
  type messages, I saw 
  what these could be filtered on, but don't know how to write up and 
  implement the rule 
  outside of procmail. I don't want to use procmail for this since it I 
  think it would be an 
  expensive routine for procmail to run.
 
  In the body of the 'unable to deliver' message, the original message is 
  quoted. One of the 
  lines quoted is the Message-ID: header from the original. The format of 
  this line is always 
  wrong as it does not contain the FQDN that our server appends to the end 
  of the hash 
  number , following the '@' symbol .
 
  So, need a rule that would parse the Message-ID: in the body (or 
  attachment) and not 
  header, and look for the @FQDN 
  Is this rule already out in the wild?


  (note: your To: was the bogofilter list, but this appeared on 
  spamassassin-users as well.. It looks like you bcc'ed the SA list.  
  Anyway, I'm answering on the SA list because that's where I picked up 
  the message from)
 
  Not that I know of, but it would be fairly quick as a spamassassin rule.
 
  You'd likely need a meta of some sort.
 
  Theoretically, something like this should work. I'm leveraging some of 
  the stock ruleset here, by reusing BOUNCE_MESSAGE to detect if the 
  message really is a bounce, make sure it is in your ruleset.
  
 
  Actually, that's overkill -- BOUNCE_MESSAGE _already_ does this.

 
 Whoops.. good point. I didn't read the code, I just saw the name and 
 assumed it did just what it says, and nothing more.
 
 So, really all .rp needs to do is enable the vbounce plugin (which is 
 loaded by default )

Yep.  to enable it, just set whitelist_bounce_relays in the
configuration or user prefs.

--j.


Re: possible idea for backscatter problem

2008-05-08 Thread Matt Kettler

Justin Mason wrote:

Matt Kettler writes:
  

.rp wrote:

One of the users (actually the boss) had the email address harvested and we got clobbered 
by backscatter. Looking at the emails of the various 'unable to deliver' type messages, I saw 
what these could be filtered on, but don't know how to write up and implement the rule 
outside of procmail. I don't want to use procmail for this since it I think it would be an 
expensive routine for procmail to run.


In the body of the 'unable to deliver' message, the original message is quoted. One of the 
lines quoted is the Message-ID: header from the original. The format of this line is always 
wrong as it does not contain the FQDN that our server appends to the end of the hash 
number , following the '@' symbol .


So, need a rule that would parse the Message-ID: in the body (or attachment) and not 
header, and look for the @FQDN 
Is this rule already out in the wild?
  
  
(note: your To: was the bogofilter list, but this appeared on 
spamassassin-users as well.. It looks like you bcc'ed the SA list.  
Anyway, I'm answering on the SA list because that's where I picked up 
the message from)


Not that I know of, but it would be fairly quick as a spamassassin rule.

You'd likely need a meta of some sort.

Theoretically, something like this should work. I'm leveraging some of 
the stock ruleset here, by reusing BOUNCE_MESSAGE to detect if the 
message really is a bounce, make sure it is in your ruleset.



Actually, that's overkill -- BOUNCE_MESSAGE _already_ does this.
  


Whoops.. good point. I didn't read the code, I just saw the name and 
assumed it did just what it says, and nothing more.



So, really all .rp needs to do is enable the vbounce plugin (which is 
loaded by default )


Re: possible idea for backscatter problem

2008-05-08 Thread Henrik K
On Thu, May 08, 2008 at 10:03:28AM +0100, Justin Mason wrote:
 
 Henrik Krohns writes:
  On Thu, May 08, 2008 at 09:35:31AM +0100, Justin Mason wrote:
  
   the VBounce plugin is intended to catch backscatter -- bounces in response
   to mail you didn't send -- so it'll ignore bounces in response to mail you
   _did_ send, by parsing the bounced message's Received: headers and looking
   for the mailserver's name in there.
  
  I've been trying it for myself. One thing I don't like is that all
  null-sender mail is assumed to be bounces. It creates many FPs.
 
 Yes.  What kind of null-sender mail from machines that are not in
 whitelist_bounce_relays do you get?

I would assume this is common knowledge. :-)

- Mass mailings of all sorts, mailing lists, news
- Order confirmations
- Some very legimate mails from people using lotus notes, hotmail
- etc, etc..

You can just imagine a system/administrator thinking that it's wise to send
anything that doesn't require a reply as null. It's very common.

Do you want a bug opened?



Re: possible idea for backscatter problem

2008-05-08 Thread Henrik K
On Thu, May 08, 2008 at 11:35:30AM +0100, Justin Mason wrote:
 
 Not in my experience!
 
 I haven't seen anything that isn't a bounce message, an out-of-office
 notification, auto-replies, or other stuff targeted by the VBounce
 ruleset.  certainly not transactional mail.  as far as I can tell, it's
 *not* that common.

I can give you dozen sample emails from last week right now. Including
handwritten mails from people, that come from strange systems (Lotus, Live
Mail, Horde/IMP..) sending as null. There's a lot more, I don't have time to
check every mail. It *is* common. And this server only passes 15k mails a
day.

You are only talking about your own mailfeed, right? I can't believe you
would say this otherwise.

  Do you want a bug opened?
 
 Nah, that's ok ;)

Well, frankly I have too much on my plate to start working on this too. So
I'll let it sleep for now. :)



Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread ram
IOn Wed, 2008-05-07 at 08:50 -0700, Marc Perkel wrote:
 Looking for a few volunteers who want to reduce their spambot spam and 
 at the same time help me track spambots for my black list. This is free 
 and mutual benefit. I (junkemailfilter.com) want to be your highest 
 numbered fake MX record. Here's how you would configure your domain:
 
 mail.yourdomain.com MX 10
 tarbaby.junkemailfilter.com MX 20
 
 I will never actually receive your email. The recipient all always get a 
 451 error just after the DATA command. So if your servers are down you 
 won't lose anything. A 451 error is a I'm not ready, come back later 
 error.
 
 This will help you reduce your spambot spam generally by half. 

...

I use fake MX as well. But even if my lower MXes are perfectly
available. I have seen quiet a lot of legitimate traffic coming on my
fake MX and get turned down with a tempfail. 

  So If you are populating blacklists based on this data , better be
careful. (I'm sure you would have seen that yourself) 

Anyway I think moving an MX record to a third party with no bussiness
contact would not be possible for anyone

Thanks
Ram







Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread ram

On Thu, 2008-05-08 at 09:33 +0100, Justin Mason wrote:
 Kevin W. Gagel writes:
  - Original Message -
  Marc Perkel wrote:
   Looking for a few volunteers who want to reduce their spambot spam and 
   at the same time help me track spambots for my black list. This is free 
   and mutual benefit. I (junkemailfilter.com) want to be your highest 
   numbered fake MX record. Here's how you would configure your domain:
  
  A generous offer and an admirable effort. But if you think I or my 
  clients are going to route mail to your servers you are mistaken. Even 
  if I knew you personally, I don't think ethics or common sense would 
  allow me to do so.
  
  DAve
  
  Personally I use the honeypot project. I recomend it. See:
  http://www.projecthoneypot.org
  For info.
 
 btw, if you have spare spamtrap *domains* -- not just /etc/aliases
 forwards -- we'd love to get a couple pointed at the SpamAssassin
 spamtraps...
 

What should the MX'es be pointed to ? 

Also what are tricks of getting mails on your spamtrap ? 




 --j.



Re: possible idea for backscatter problem

2008-05-08 Thread Robert Müller



Henrik K schrieb:

On Thu, May 08, 2008 at 11:35:30AM +0100, Justin Mason wrote:
  

Not in my experience!

I haven't seen anything that isn't a bounce message, an out-of-office
notification, auto-replies, or other stuff targeted by the VBounce
ruleset.  certainly not transactional mail.  as far as I can tell, it's
*not* that common.



I can give you dozen sample emails from last week right now. Including
handwritten mails from people, that come from strange systems (Lotus, Live
Mail, Horde/IMP..) sending as null. There's a lot more, I don't have time to
check every mail. It *is* common. And this server only passes 15k mails a
day.

You are only talking about your own mailfeed, right? I can't believe you
would say this otherwise.

  

Do you want a bug opened?
  

Nah, that's ok ;)



Well, frankly I have too much on my plate to start working on this too. So
I'll let it sleep for now. :)


  
After my fix in the 20_vbounce.cf I had also several FPs, the most 
significant the SA-bugzilla account confirmation mail. :-| Reason for 
this was the simple match on 'daemon' in the From: -line regexp.

Is somebody already working on an enhancement of the ruleset?
Otherwise, if you can provide some of your FP's (and maybe some real 
backscatter) I can imagine to put them together with mine and try to 
improve the rules a little bit in the next few weeks.
BTW: Also for me 'null senders' are not common - never had problems with 
this, except UBE. Correctly configured servers/clients should not 
produce such mails, IMHO.

Robert


Re: possible idea for backscatter problem

2008-05-08 Thread Shane Williams

On Thu, 8 May 2008, Justin Mason wrote:


Matt Kettler writes:

.rp wrote:


So, need a rule that would parse the Message-ID: in the body (or attachment) 
and not
header, and look for the @FQDN
Is this rule already out in the wild?


You'd likely need a meta of some sort.

Theoretically, something like this should work. I'm leveraging some of
the stock ruleset here, by reusing BOUNCE_MESSAGE to detect if the
message really is a bounce, make sure it is in your ruleset.


Actually, that's overkill -- BOUNCE_MESSAGE _already_ does this.

the VBounce plugin is intended to catch backscatter -- bounces in response
to mail you didn't send -- so it'll ignore bounces in response to mail you
_did_ send, by parsing the bounced message's Received: headers and looking
for the mailserver's name in there.


Pardon my ignorance if I'm just not understanding this right, but my
impression is that there's a possibility that messages marked the
BOUNCE_MESSAGE could be legitimate bounces (just not bounces generated
by a whitelisted server).  Certainly I've read plenty of people on
this list advise against raising the vbounce rule scores as a way to
combat this new wave of seemingly intentional bounce spam, but rather
to filter all the bounces off to a separate folder.

But, doesn't the existence of a Message ID in the text of the bounce
that has a bogus or malformed email address provide a stronger
indication that this is not a valid bounce?  In which case, such email
could be scored higher, rather than just sent to a bounce folder,
which is really what many of us would rather be doing with these
messages?

--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/|  System Admin - UT iSchool
=--+---
All syllogisms contain three lines |  [EMAIL PROTECTED]
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew


Re: possible idea for backscatter problem

2008-05-08 Thread Henrik K
On Thu, May 08, 2008 at 03:11:59PM +0200, Robert Müller wrote:

 BTW: Also for me 'null senders' are not common - never had problems with
 this, except UBE.

Have you even looked at your traffic archives, if you keep one? How do you
know there isn't any problems if someone doesn't realize to report it?

I have concrete evidence.

 Correctly configured servers/clients should not produce such mails, IMHO.

That's just absurd. I could just as well say: Correctly configured servers
don't create backscatter.. Yet we have the problem.

In case of VBounce, chances of FPs are even less acceptable. You are
supposed to reject or discard backscatter, I see no point in just tagging
it. So discarding is bad, but rejecting most likely means that sending party
doesn't get any notification of failure either.

Currently VBounce is only useful as add little score and hope URIBL and
other checks match the returned body.

Hopefully someone has time to fix the too generic rules. We should only
match sure bounces.



triplets.txt

2008-05-08 Thread Jeremy Fairbrass

Hi, could someone kindly tell me what the file triplets.txt is used for, and 
if I need to have it in my rules directory or not?

Cheers,
Jeremy



Re: Shortcircuit Question

2008-05-08 Thread Clayton Keller

Justin Mason wrote:

Matt Kettler writes:

Clayton Keller wrote:
I have been reading throught the Shortcircuit manpage as well as some 
articles within the Wiki, and the manner in which I see it performing 
within our install does not seem to coincide with how I am reading and 
presumably understanding it to work.


First off, we are using SpamAssassin 3.2.4 provided by the rpmforge 
batch of rpm's.


I have about a dozen priorities specified mainly handling URIBL, 
SURBL, as well as DCC, Razor, Pyzor and Bayes.


What I am seeing is that although the first shortcircuit rule hits and 
scores appropriately. Subsequent short circuit rules will continue to 
fire. The scores themselves are then totaled along with the original 
scores for the rules.


My understanding of how the shortcircuit should work is that once a 
shortcircuit is triggered any subsequent rules should be bypassed and 
the message wither classified as spam/ham or if set to on, it would 
use the current score specified for the rule.


As a for instance:

I have the following:

priority URIBL_BLACK-500
priority URIBL_JP_SURBL -498
priority URIBL_SC_SURBL -488
priority URIBL_OB_SURBL -487

priority SC_URIBL_SURBL -480
priority SC_URIBL_SBL   -479

priority RAZOR2_CHECK   -450
priority DCC_CHECK  -449
priority PYZOR_CHECK-448

priority SC_URIBL_HASH  -440

meta SC_URIBL_SURBL(URIBL_BLACK  (URIBL_SC_SURBL
   || URIBL_JP_SURBL || URIBL_OB_SURBL))

meta SC_URIBL_SBL((URIBL_BLACK || URIBL_SC_SURBL ||
URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


meta SC_URIBL_HASH((URIBL_BLACK || URIBL_SC_SURBL ||
 URIBL_JP_SURBL || URIBL_OB_SURBL)  
(RAZOR2_CHECK || DCC_CHECK || PYZOR_CHECK))



meta SC_URIBL_SBL   ((URIBL_BLACK || URIBL_SC_SURBL ||
  URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


shortcircuit SC_URIBL_SURBL spam
shortcircuit SC_URIBL_SBL   spam
shortcircuit SC_URIBL_HASH spam

score SC_URIBL_SURBL100.00
score SC_URIBL_HASH 100.00
score SC_URIBL_SBL  100.00


I do not have a recent debug to show but I can say that from the debug 
I do see the SC_URIBL_SURBL trigger, after the earlier priority rules 
are ran. However, the remaining priorities are then ran, and if 
meeting critera, the RAZOR, DCC, and PYZOR rules run and then the 
SC_URIBL_HASH rule would trigger. Thus giving a total score of 200 + 
the scores for the URIBL/SURBL scores that hit and if included the 
Razor, DCC, and Pyzor scores as well.


I was thinking after the SC_URIBL_SURBL was triggered remaining rules 
would not run, and the spam classification would take precendence.


Am I overlooking the obvious, have I misunderstood how the SC should 
work, is it something with the rpm that was released by rpmforge? Any 
thoughts or insight would be appreciated. 
SA is, rather fortunately, circumventing what you're trying to do 
because of how DNS is handled internally.


DO NOT try to split up the priority of DNS based tests. Priority and 
shortcircuiting is intended to be used on *fast* rules, not slow ones.


 If you were successful, you would make the performance of SpamAssassin 
absurdly slow by serializing DNS queries. *OUCH*. SA normally runs these 
in parallel, and running them in serial would very seriously impact 
performance.


Currently, all DNS based tests run at their priority, but that only 
launches the DNS queries. All the results are gathered together at 
HARVEST_DNSBL_PRIORITY, which is currently set to 500. None of the rules 
will actually trigger until this point.


actually Matt, you're wrong ;)  if some of the network rules are
at a higher priority than others, and are used in shortcircuit rules,
SpamAssassin 3.2.x will indeed sleep until the results of those rules
arrive.

The idea is that, if you have the memory to support that degree of
concurrency, you can make a local policy decision to do that, instead
of doing the lookups at the MTA level which does effectively the
same thing.

This wait is logged, so you can spot it with --debug on.

--j.



With that said Justin, is the behavior I am seeing correct? Even though 
the first prioritized shortcircuit rule hits, and I see that in the 
debug log, shouldn't it be bypassing the remaining rules rather than 
continuing to process until all the shortcircuit priorities have ran?


From reading the initial bug when this was originally featured, along 
with the man page, as well as a wiki post with an example by you, that 
is how I understood it to function.


Clay


Re: Shortcircuit Question

2008-05-08 Thread Justin Mason

Clayton Keller writes:
 Justin Mason wrote:
  Matt Kettler writes:
  Clayton Keller wrote:
  I have been reading throught the Shortcircuit manpage as well as some 
  articles within the Wiki, and the manner in which I see it performing 
  within our install does not seem to coincide with how I am reading and 
  presumably understanding it to work.
 
  First off, we are using SpamAssassin 3.2.4 provided by the rpmforge 
  batch of rpm's.
 
  I have about a dozen priorities specified mainly handling URIBL, 
  SURBL, as well as DCC, Razor, Pyzor and Bayes.
 
  What I am seeing is that although the first shortcircuit rule hits and 
  scores appropriately. Subsequent short circuit rules will continue to 
  fire. The scores themselves are then totaled along with the original 
  scores for the rules.
 
  My understanding of how the shortcircuit should work is that once a 
  shortcircuit is triggered any subsequent rules should be bypassed and 
  the message wither classified as spam/ham or if set to on, it would 
  use the current score specified for the rule.
 
  As a for instance:
 
  I have the following:
 
  priority URIBL_BLACK-500
  priority URIBL_JP_SURBL -498
  priority URIBL_SC_SURBL -488
  priority URIBL_OB_SURBL -487
 
  priority SC_URIBL_SURBL -480
  priority SC_URIBL_SBL   -479
 
  priority RAZOR2_CHECK   -450
  priority DCC_CHECK  -449
  priority PYZOR_CHECK-448
 
  priority SC_URIBL_HASH  -440
 
  meta SC_URIBL_SURBL(URIBL_BLACK  (URIBL_SC_SURBL
 || URIBL_JP_SURBL || URIBL_OB_SURBL))
 
  meta SC_URIBL_SBL((URIBL_BLACK || URIBL_SC_SURBL ||
  URIBL_JP_SURBL || URIBL_OB_SURBL)  
  URIBL_SBL)
 
  meta SC_URIBL_HASH((URIBL_BLACK || URIBL_SC_SURBL ||
   URIBL_JP_SURBL || URIBL_OB_SURBL)  
  (RAZOR2_CHECK || DCC_CHECK || PYZOR_CHECK))
 
 
  meta SC_URIBL_SBL   ((URIBL_BLACK || URIBL_SC_SURBL ||
URIBL_JP_SURBL || URIBL_OB_SURBL)  
  URIBL_SBL)
 
  shortcircuit SC_URIBL_SURBL spam
  shortcircuit SC_URIBL_SBL   spam
  shortcircuit SC_URIBL_HASH spam
 
  score SC_URIBL_SURBL100.00
  score SC_URIBL_HASH 100.00
  score SC_URIBL_SBL  100.00
 
 
  I do not have a recent debug to show but I can say that from the debug 
  I do see the SC_URIBL_SURBL trigger, after the earlier priority rules 
  are ran. However, the remaining priorities are then ran, and if 
  meeting critera, the RAZOR, DCC, and PYZOR rules run and then the 
  SC_URIBL_HASH rule would trigger. Thus giving a total score of 200 + 
  the scores for the URIBL/SURBL scores that hit and if included the 
  Razor, DCC, and Pyzor scores as well.
 
  I was thinking after the SC_URIBL_SURBL was triggered remaining rules 
  would not run, and the spam classification would take precendence.
 
  Am I overlooking the obvious, have I misunderstood how the SC should 
  work, is it something with the rpm that was released by rpmforge? Any 
  thoughts or insight would be appreciated. 
  SA is, rather fortunately, circumventing what you're trying to do 
  because of how DNS is handled internally.
 
  DO NOT try to split up the priority of DNS based tests. Priority and 
  shortcircuiting is intended to be used on *fast* rules, not slow ones.
 
   If you were successful, you would make the performance of SpamAssassin 
  absurdly slow by serializing DNS queries. *OUCH*. SA normally runs these 
  in parallel, and running them in serial would very seriously impact 
  performance.
 
  Currently, all DNS based tests run at their priority, but that only 
  launches the DNS queries. All the results are gathered together at 
  HARVEST_DNSBL_PRIORITY, which is currently set to 500. None of the rules 
  will actually trigger until this point.
  
  actually Matt, you're wrong ;)  if some of the network rules are
  at a higher priority than others, and are used in shortcircuit rules,
  SpamAssassin 3.2.x will indeed sleep until the results of those rules
  arrive.
  
  The idea is that, if you have the memory to support that degree of
  concurrency, you can make a local policy decision to do that, instead
  of doing the lookups at the MTA level which does effectively the
  same thing.
  
  This wait is logged, so you can spot it with --debug on.
  
  --j.
  
 
 With that said Justin, is the behavior I am seeing correct? Even though 
 the first prioritized shortcircuit rule hits, and I see that in the 
 debug log, shouldn't it be bypassing the remaining rules rather than 
 continuing to process until all the shortcircuit priorities have ran?
 
  From reading the initial bug when this was originally featured, along 
 with the man page, as well as a wiki post with an example by you, that 
 is how I understood it to function.

actually, no, it sounds like a bug.could you open a 

Re: possible idea for backscatter problem

2008-05-08 Thread Robert Müller



Henrik K schrieb:

On Thu, May 08, 2008 at 03:11:59PM +0200, Robert Müller wrote:
  

BTW: Also for me 'null senders' are not common - never had problems with
this, except UBE.



Have you even looked at your traffic archives, if you keep one? How do you
know there isn't any problems if someone doesn't realize to report it?
  

I would have realized, if I had this problem, for sure.

I have concrete evidence.
  

I believe you, no problem.
  

Correctly configured servers/clients should not produce such mails, IMHO.



That's just absurd. I could just as well say: Correctly configured servers
don't create backscatter.. Yet we have the problem.
  
I think you misunderstood me. Such mails are  - in my experience - not 
the majority, and therefore this issue is also for me not common and not 
known as a big problem. Nothing more I wanted to say.

Now I hear different from you - fine, something learned.

In case of VBounce, chances of FPs are even less acceptable. You are
supposed to reject or discard backscatter, I see no point in just tagging
it. So discarding is bad, but rejecting most likely means that sending party
doesn't get any notification of failure either.

Currently VBounce is only useful as add little score and hope URIBL and
other checks match the returned body.
  
ACK. Because this doesn't work in most cases, and the currently (for me) 
unknown risk of FPs, I actually I sort them in a different folder.

Hopefully someone has time to fix the too generic rules. We should only
match sure bounces.
  

Sure is as always relative, but that was the goal, yes.


  


Re: possible idea for backscatter problem

2008-05-08 Thread Justin Mason

  In case of VBounce, chances of FPs are even less acceptable. You are
  supposed to reject or discard backscatter

who says?

It seems perfectly fine to me to tag vbounce-filtered mail.  In mail
filtering, there will always be FPs.

--j.


Re: IE Parse bug olso in SpamAssassin ?

2008-05-08 Thread John Hardin

On Thu, 8 May 2008, Benny Pedersen wrote:


On Thu, May 8, 2008 05:00, Joseph Brennan wrote:


Should we just call http://{; bad, and not bother checking the uri?


i belive there is parts in sa that parse the same way as ie and that 
could be used by spammers to hide there domains in multilvel obfu


Why worry about where the URI is trying to point if it's so obviously 
obfuscated?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED]
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  News flash: Lowest Common Denominator down 50 points
---
 Today: the 63rd anniversary of VE day


Re: possible idea for backscatter problem

2008-05-08 Thread Henrik K
On Thu, May 08, 2008 at 04:20:42PM +0100, Justin Mason wrote:
 
   In case of VBounce, chances of FPs are even less acceptable. You are
   supposed to reject or discard backscatter
 
 who says?
 
 It seems perfectly fine to me to tag vbounce-filtered mail.  In mail
 filtering, there will always be FPs.

Come on, we all know ISPs are not supposed to drop anything and that there
are users that insist on receiving all mail to be sure.

This has nothing to do with VBounce (which is constantly touted as being
backscatter stopping tool). In not above scenarios, why would anyone want to
receive backscatter? VBounce is currently a FP-prone match or not match
tool for that purpose. Either you want backscatter or you don't.

For other things it's perfectly fine to tag maybe spam and discard sure
spam. Now you can just hope that VBounce gets it over the discard limit.



Re: IE Parse bug olso in SpamAssassin ?

2008-05-08 Thread Benny Pedersen

On Thu, May 8, 2008 17:29, John Hardin wrote:

 Why worry about where the URI is trying to point if it's so obviously
 obfuscated?

to get more data to bayes


Benny Pedersen
Need more webspace ? http://www.servage.net/?coupon=cust37098



Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread Marc Perkel



ram wrote:

IOn Wed, 2008-05-07 at 08:50 -0700, Marc Perkel wrote:
  
Looking for a few volunteers who want to reduce their spambot spam and 
at the same time help me track spambots for my black list. This is free 
and mutual benefit. I (junkemailfilter.com) want to be your highest 
numbered fake MX record. Here's how you would configure your domain:


mail.yourdomain.com MX 10
tarbaby.junkemailfilter.com MX 20

I will never actually receive your email. The recipient all always get a 
451 error just after the DATA command. So if your servers are down you 
won't lose anything. A 451 error is a I'm not ready, come back later 
error.


This will help you reduce your spambot spam generally by half. 



...

I use fake MX as well. But even if my lower MXes are perfectly
available. I have seen quiet a lot of legitimate traffic coming on my
fake MX and get turned down with a tempfail. 


  So If you are populating blacklists based on this data , better be
careful. (I'm sure you would have seen that yourself) 


Anyway I think moving an MX record to a third party with no bussiness
contact would not be possible for anyone

Thanks
Ram


  


Hi Ram,

Being a high numbered MX in itself doesn't get you listed on this new 
server I set up. It's just a prequalifier of what I want to look at. In 
order to get listed they also have to fail to send a QUIT after the 451 
error and they have to commit some other significant sins. I'm looking 
at a number of things in the helo, the sender, the recipient, rDNS, etc. 
What I'm doing isn't going to catch as high of a percentage as I would 
if I were the official spam filtering host for the domain because I'm 
not running all my tests on it. I'm cutting them off before the data is 
sent. I'm not even seeing the message headers.


However, I do think that I'll catch a lot of what I'm looking for and 
that's virus infected spambots. That's the only think I'm targeting here 
and I think I can distinguish them well enough that I can catch most all 
the spambot traffic with no false positives on legit email. I'm hoping 
for 50% accuracy of catching spambots on the first attempt.


To participate all you have to do is set your highest numbered MX to 
point to:


tarbaby.junkemailfilter.com

Several people have asked me how I'm doing this and can they have my 
code to do it themselves. My situation is unique enough that it just 
won't work very easilly any place else and it's definitely not clean 
enough for just anyone to install. But I'll try to describe it here.


First to do what I'm doing you have to be using EXIM. If you aren't 
running exim then you just can't do it. In fact, with all due respect, I 
can't see how anyone can do spam filtering and not use exim as their MTA.


Exim has a feature where you can execute code based on how the 
connection is closed. It have a NOTQUIT acl and you can look at if the 
connection timed out and a number of other things that caused the 
connection to close without issuing a quit. Before the 451 error I store 
information in variables that I can retrieve in the notquit acl and 
based on that information I can send messages to another server that 
accumulating information from all my servers. This server is basically 
running stats on a one minute cycle to determine what data goes into my 
various white/black/yellow lists and that feeds my 4 rbldnsd servers 
which are updated every minute.


Blacklist data is stored for 5 days and then it expired. Every 6 hours 
the oldest log file is deleted and everything is moved down a slot and a 
new log file created. Thus if someone fixed the virus then they will 
eventually be cleaned off the list. Users also have a web form where 
they can get themselves removed if there is a false positive.


The list isn't perfect but it is my goal to have no false positives. 
Unlike some lists who think that some sloppy admins deserve to be 
blacklisted my attitude is if the listing is wrong it's my fault and I 
want to fix it. And unlike many other blacklisating services I focuse 
more on my white listing and yellow listing and use that information to 
reduce the chance of false positives in my blacklists.


I also see the value of being as cooperative with others because 
although I'm good at coming up with new ideas, other are better at 
taking the ideas and doing it right. So many times I'll put an idea out 
there and someone else will do it better and I get to run their better 
version.


I am of the opinion that 100% of spambot spam can be stopped because I'm 
doing it.I want to try to expand on that and get data from other sources 
and see if I can't help others make some progress too.


Hope this is helpful.



Re: IE Parse bug olso in SpamAssassin ?

2008-05-08 Thread John Hardin

On Thu, 8 May 2008, Benny Pedersen wrote:


On Thu, May 8, 2008 17:29, John Hardin wrote:


Why worry about where the URI is trying to point if it's so obviously
obfuscated?


to get more data to bayes


Bayes isn't going to parse a URI as a URI anyway, is it? It just tokenizes 
the message. Bayes will pick up the domain name string if it's delimited 
by {} as readily as it will if it's delimited by //.


To clarify: why bother trying to deobfuscate the URI and figure out what 
domain it's really pointing at, so that domain can be checked against 
URIBL lists, if the form of the obfuscation is obvious and not seen in 
legitimate emails? Why not just give that obfuscation four or five points 
and be done with it?


If that formatting *was* seen in legitimate emails, then I would say that 
it's important the URI parsers be aware of it.


Can you provide any pointers to documentation of that formatting? I didn't 
find any in a quick gargle.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED]
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The real opiate of the masses isn't religion; it's the belief that
  somewhere there is a benefit that can be delivered without a
  corresponding cost.   -- Tom of Radio Free NJ
---
 Today: the 63rd anniversary of VE day


Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread John Hardin

On Thu, 8 May 2008, Marc Perkel wrote:

To participate all you have to do is set your highest numbered MX to 
point to:


tarbaby.junkemailfilter.com

Several people have asked me how I'm doing this and can they have my 
code to do it themselves. My situation is unique enough that it just 
won't work very easilly any place else and it's definitely not clean 
enough for just anyone to install.


You should make an effort to clean it up so that others *can* install it 
as a standalone daemon, as I suggested. Why? How long will it be before 
the spambots explicitly refuse to contact your honeypot if it is listed as 
an MX for the domain they're attacking?


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED]
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The real opiate of the masses isn't religion; it's the belief that
  somewhere there is a benefit that can be delivered without a
  corresponding cost.   -- Tom of Radio Free NJ
---
 Today: the 63rd anniversary of VE day


Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread Marc Perkel



John Hardin wrote:

On Thu, 8 May 2008, Marc Perkel wrote:

To participate all you have to do is set your highest numbered MX to 
point to:


tarbaby.junkemailfilter.com

Several people have asked me how I'm doing this and can they have my 
code to do it themselves. My situation is unique enough that it just 
won't work very easilly any place else and it's definitely not clean 
enough for just anyone to install.


You should make an effort to clean it up so that others *can* install 
it as a standalone daemon, as I suggested. Why? How long will it be 
before the spambots explicitly refuse to contact your honeypot if it 
is listed as an MX for the domain they're attacking?




Good point. I suppose that if this grows I can point to my traps using 
other hostnames. I can also set up traps on virtual server under OpenVZ 
so spammers won't know what IP ranges to avoid.





Re: Shortcircuit Question

2008-05-08 Thread Clayton Keller

Justin Mason wrote:

Clayton Keller writes:

Justin Mason wrote:

Matt Kettler writes:

Clayton Keller wrote:
I have been reading throught the Shortcircuit manpage as well as some 
articles within the Wiki, and the manner in which I see it performing 
within our install does not seem to coincide with how I am reading and 
presumably understanding it to work.


First off, we are using SpamAssassin 3.2.4 provided by the rpmforge 
batch of rpm's.


I have about a dozen priorities specified mainly handling URIBL, 
SURBL, as well as DCC, Razor, Pyzor and Bayes.


What I am seeing is that although the first shortcircuit rule hits and 
scores appropriately. Subsequent short circuit rules will continue to 
fire. The scores themselves are then totaled along with the original 
scores for the rules.


My understanding of how the shortcircuit should work is that once a 
shortcircuit is triggered any subsequent rules should be bypassed and 
the message wither classified as spam/ham or if set to on, it would 
use the current score specified for the rule.


As a for instance:

I have the following:

priority URIBL_BLACK-500
priority URIBL_JP_SURBL -498
priority URIBL_SC_SURBL -488
priority URIBL_OB_SURBL -487

priority SC_URIBL_SURBL -480
priority SC_URIBL_SBL   -479

priority RAZOR2_CHECK   -450
priority DCC_CHECK  -449
priority PYZOR_CHECK-448

priority SC_URIBL_HASH  -440

meta SC_URIBL_SURBL(URIBL_BLACK  (URIBL_SC_SURBL
   || URIBL_JP_SURBL || URIBL_OB_SURBL))

meta SC_URIBL_SBL((URIBL_BLACK || URIBL_SC_SURBL ||
URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


meta SC_URIBL_HASH((URIBL_BLACK || URIBL_SC_SURBL ||
 URIBL_JP_SURBL || URIBL_OB_SURBL)  
(RAZOR2_CHECK || DCC_CHECK || PYZOR_CHECK))



meta SC_URIBL_SBL   ((URIBL_BLACK || URIBL_SC_SURBL ||
  URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


shortcircuit SC_URIBL_SURBL spam
shortcircuit SC_URIBL_SBL   spam
shortcircuit SC_URIBL_HASH spam

score SC_URIBL_SURBL100.00
score SC_URIBL_HASH 100.00
score SC_URIBL_SBL  100.00


I do not have a recent debug to show but I can say that from the debug 
I do see the SC_URIBL_SURBL trigger, after the earlier priority rules 
are ran. However, the remaining priorities are then ran, and if 
meeting critera, the RAZOR, DCC, and PYZOR rules run and then the 
SC_URIBL_HASH rule would trigger. Thus giving a total score of 200 + 
the scores for the URIBL/SURBL scores that hit and if included the 
Razor, DCC, and Pyzor scores as well.


I was thinking after the SC_URIBL_SURBL was triggered remaining rules 
would not run, and the spam classification would take precendence.


Am I overlooking the obvious, have I misunderstood how the SC should 
work, is it something with the rpm that was released by rpmforge? Any 
thoughts or insight would be appreciated. 
SA is, rather fortunately, circumventing what you're trying to do 
because of how DNS is handled internally.


DO NOT try to split up the priority of DNS based tests. Priority and 
shortcircuiting is intended to be used on *fast* rules, not slow ones.


 If you were successful, you would make the performance of SpamAssassin 
absurdly slow by serializing DNS queries. *OUCH*. SA normally runs these 
in parallel, and running them in serial would very seriously impact 
performance.


Currently, all DNS based tests run at their priority, but that only 
launches the DNS queries. All the results are gathered together at 
HARVEST_DNSBL_PRIORITY, which is currently set to 500. None of the rules 
will actually trigger until this point.

actually Matt, you're wrong ;)  if some of the network rules are
at a higher priority than others, and are used in shortcircuit rules,
SpamAssassin 3.2.x will indeed sleep until the results of those rules
arrive.

The idea is that, if you have the memory to support that degree of
concurrency, you can make a local policy decision to do that, instead
of doing the lookups at the MTA level which does effectively the
same thing.

This wait is logged, so you can spot it with --debug on.

--j.

With that said Justin, is the behavior I am seeing correct? Even though 
the first prioritized shortcircuit rule hits, and I see that in the 
debug log, shouldn't it be bypassing the remaining rules rather than 
continuing to process until all the shortcircuit priorities have ran?


 From reading the initial bug when this was originally featured, along 
with the man page, as well as a wiki post with an example by you, that 
is how I understood it to function.


actually, no, it sounds like a bug.could you open a bugzilla with
a demonstration config/test message?

--j.



Just to add, with my previous debug, I can confirm the waiting of the 
tests to finish as you mentioned from the 

Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread Kevin Parris
Well now, if a spambot actually does start recognizing and avoiding his system, 
doesn't that mean he wins and the spammer loses?


 John Hardin [EMAIL PROTECTED] 05/08/08 12:11 PM 
On Thu, 8 May 2008, Marc Perkel wrote:

 To participate all you have to do is set your highest numbered MX to 
 point to:

 tarbaby.junkemailfilter.com

 Several people have asked me how I'm doing this and can they have my 
 code to do it themselves. My situation is unique enough that it just 
 won't work very easilly any place else and it's definitely not clean 
 enough for just anyone to install.

You should make an effort to clean it up so that others *can* install it as a 
standalone daemon, as I suggested. Why? How long will it be before the spambots 
explicitly refuse to contact your honeypot if it is listed as an MX for the 
domain they're attacking?





Re: Experimental - use my server for your high fake MX record

2008-05-08 Thread Marc Perkel



Kevin Parris wrote:

Well now, if a spambot actually does start recognizing and avoiding his system, 
doesn't that mean he wins and the spammer loses?

  

I would say YES!



You should make an effort to clean it up so that others *can* install it as a 
standalone daemon, as I suggested. Why? How long will it be before the spambots 
explicitly refuse to contact your honeypot if it is listed as an MX for the 
domain they're attacking?



  


I don't see that happening. If the spammers were that sharp they would 
send quit and close the connection properly and defeat the meathod 
rather than defeating just me. But it would cost them some bandwidth and 
speed to do that. Especially if I added some delays before doing the 
rejection which would cause the spammer to have to keep the connection 
open longer which they aren't going to do.


I'm going to think about the delay thing. You inspired possibly another 
good idea.




RE: Experimental - use my server for your high fake MX record

2008-05-08 Thread Maurice Lucas
Or,

The spammers will find his host and don't use the highest MX record. Or just 
remove his host from all the results.

My best solution would be:
Marc,

-  Clean up the code

-  Write a manual howto install so every admin can install it

-  Write an extra bit of code which will send you all the information 
WITHOUT the information below.

-  Everybody who wants it can use your great software and we all win*

I have contracts with my customers that I will not use their email for other 
business then to deliver it to its destination. Some of my customers will get 
into problems if other people know their contacts.
So I can give you all information about an email message without

-  The from

-  The to

-  The body
But with all the IP addresses and with the QUIT after 451 status.


* we all know you wouldn't use it as a selling point to spammers or do 
something else with it but can/will you write that into a contract with all 
other admins. And pay a large sum of money if some data is found on the 
internet.
And do we want that type of silly contracts.
No we want to stop spam and not kill every other spamkiller (application or 
person)

met vriendelijke groet,

Maurice Lucas

TAOS-IT

Paulus Buijsstraat 191
2613 HR  Delft
www.taos-it.nlhttp://www.taos-it.nl/
KvK Haaglanden nr. 27254410

From: Marc Perkel [mailto:[EMAIL PROTECTED]
Sent: donderdag 8 mei 2008 19:07
To: Kevin Parris
Cc: users@spamassassin.apache.org
Subject: Re: Experimental - use my server for your high fake MX record



Kevin Parris wrote:

Well now, if a spambot actually does start recognizing and avoiding his system, 
doesn't that mean he wins and the spammer loses?




I would say YES!






You should make an effort to clean it up so that others *can* install it as a 
standalone daemon, as I suggested. Why? How long will it be before the spambots 
explicitly refuse to contact your honeypot if it is listed as an MX for the 
domain they're attacking?









I don't see that happening. If the spammers were that sharp they would send 
quit and close the connection properly and defeat the meathod rather than 
defeating just me. But it would cost them some bandwidth and speed to do that. 
Especially if I added some delays before doing the rejection which would cause 
the spammer to have to keep the connection open longer which they aren't going 
to do.

I'm going to think about the delay thing. You inspired possibly another good 
idea.


Re: IE Parse bug olso in SpamAssassin ?

2008-05-08 Thread Benny Pedersen

On Thu, May 8, 2008 18:07, John Hardin wrote:

 Bayes isn't going to parse a URI as a URI anyway, is it?

i belived it did use that info olso

 It just tokenizes the message.

hopefully with url that confirm to rfc olso, but i see hte point where url is
obfu not to bother now when i think more about it

 Bayes will pick up the domain name string if it's delimited
 by {} as readily as it will if it's delimited by //.

yes got it now, i was just a bit blind on the hidded url in redir.html

 To clarify: why bother trying to deobfuscate the URI and figure out what
 domain it's really pointing at, so that domain can be checked against
 URIBL lists,

the hidded url could olso be a whitelisted domain

 if the form of the obfuscation is obvious and not seen in
 legitimate emails?

obfu is genricly a spam sign

 Why not just give that obfuscation four or five points
 and be done with it?

yep i will

 If that formatting *was* seen in legitimate emails, then I would say that
 it's important the URI parsers be aware of it.

yes, my fault not thinking that long here :/

 Can you provide any pointers to documentation of that formatting? I didn't
 find any in a quick gargle.

if i know what to search for i could :/

i just started this thread to be sure IE parse bug is not in sa aswell since i
could see domains not detecked in spam, but i got it now



Benny Pedersen
Need more webspace ? http://www.servage.net/?coupon=cust37098



Re: IE Parse bug olso in SpamAssassin ?

2008-05-08 Thread John Hardin

On Thu, 8 May 2008, Benny Pedersen wrote:

i just started this thread to be sure IE parse bug is not in sa aswell 
since i could see domains not detecked in spam, but i got it now


Do you have a reference for discussion of this IE Parsing bug that led 
you to mention this oddball URI annotation format in the first place? 
There might be references in that to the definition of the format.


Thanks.

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 [EMAIL PROTECTED]FALaholic #11174 pgpk -a [EMAIL PROTECTED]
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  USMC Rules of Gunfighting #9: Accuracy is relative: most combat
  shooting standards will be more dependent on pucker factor than
  the inherent accuracy of the gun.
---
 Today: the 63rd anniversary of VE day


trusted mailing list subscriber spam

2008-05-08 Thread jidanni
Odd how mailing lists that don't obfuscate addresses don't see more
trusted mailing list subscriber spam.

All a spam program would have to do is say [EMAIL PROTECTED] posts lots
to that list. His address must be a trusted subscriber. Well, here's
one more post from him, muhahaha.

OK, I suppose that would be caught by SPF rules etc., if bob likes SPF.


Re: trusted mailing list subscriber spam

2008-05-08 Thread Benny Pedersen

On Thu, May 8, 2008 19:19, [EMAIL PROTECTED] wrote:

 OK, I suppose that would be caught by SPF rules etc., if bob likes SPF.

what are you talking about ?, to score email addresses found on maillist a bit
negative since it looks like none spammy human ?


Benny Pedersen
Need more webspace ? http://www.servage.net/?coupon=cust37098



Re: IE Parse bug olso in SpamAssassin ?

2008-05-08 Thread Kevin W. Gagel
- Original Message -
Do you have a reference for discussion of this IE Parsing bug that led 
you to mention this oddball URI annotation format in the first place? 
There might be references in that to the definition of the format.

John,

I'm not sure if this is the bug Benny refers to but here is a link for info
on what I think he is referring to:
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2006-1185


--
Kevin W. Gagel 
Postmaster for
College of New Caledonia
(250) 562-2131 loc. 5448
[EMAIL PROTECTED]
http://www.cnc.bc.ca
Anti-Spam info at:
http://gateway.cnc.bc.ca


---
The College of New Caledonia, Visit us at http://www.cnc.bc.ca
Virus scanning is done on all incoming and outgoing email.
Anti-spam information for CNC can be found at http://gateway.cnc.bc.ca
---


Re: Multiple X-Envelope-From and SPF

2008-05-08 Thread mouss

ram wrote:
At the MTA( postfix) I am inserting X-Envelope-From: 
If The mail had already a X-Envelope-From before landing at my MTA  then
There would be multiple lines of these 
  


configure postfix to replace previous ones
/^(X\-Envelope\-From:.*)/   REPLACE X-$1

I am assuming you are not adding them before header_checks.


Then SA refuses to do SPF for these  messages , and I can see in my
debug logs 



-
[18469] dbg: message: X-Envelope-From header found after 1 or more
Received lines, cannot trust envelope-from
[18469] dbg: spf: relayed through one or more trusted relays, cannot use
header-based Envelope-From, skipping
--

How do I avoid this situation. 



Thanks
Ram


  




Re: Multiple X-Envelope-From and SPF

2008-05-08 Thread Benny Pedersen

On Thu, May 8, 2008 23:19, mouss wrote:

 configure postfix to replace previous ones
 /^(X\-Envelope\-From:.*)/   REPLACE X-$1

envelope from can here be forged

better for postfix is to add

envelope_sender_header Return-Path

in local.cf



Benny Pedersen
Need more webspace ? http://www.servage.net/?coupon=cust37098



Re: Shortcircuit Question

2008-05-08 Thread Clayton Keller

Clayton Keller wrote:

Justin Mason wrote:

Clayton Keller writes:

Justin Mason wrote:

Matt Kettler writes:

Clayton Keller wrote:
I have been reading throught the Shortcircuit manpage as well as 
some articles within the Wiki, and the manner in which I see it 
performing within our install does not seem to coincide with how I 
am reading and presumably understanding it to work.


First off, we are using SpamAssassin 3.2.4 provided by the 
rpmforge batch of rpm's.


I have about a dozen priorities specified mainly handling URIBL, 
SURBL, as well as DCC, Razor, Pyzor and Bayes.


What I am seeing is that although the first shortcircuit rule hits 
and scores appropriately. Subsequent short circuit rules will 
continue to fire. The scores themselves are then totaled along 
with the original scores for the rules.


My understanding of how the shortcircuit should work is that once 
a shortcircuit is triggered any subsequent rules should be 
bypassed and the message wither classified as spam/ham or if set 
to on, it would use the current score specified for the rule.


As a for instance:

I have the following:

priority URIBL_BLACK-500
priority URIBL_JP_SURBL -498
priority URIBL_SC_SURBL -488
priority URIBL_OB_SURBL -487

priority SC_URIBL_SURBL -480
priority SC_URIBL_SBL   -479

priority RAZOR2_CHECK   -450
priority DCC_CHECK  -449
priority PYZOR_CHECK-448

priority SC_URIBL_HASH  -440

meta SC_URIBL_SURBL(URIBL_BLACK  (URIBL_SC_SURBL
   || URIBL_JP_SURBL || URIBL_OB_SURBL))

meta SC_URIBL_SBL((URIBL_BLACK || URIBL_SC_SURBL ||
URIBL_JP_SURBL || URIBL_OB_SURBL)  
URIBL_SBL)


meta SC_URIBL_HASH((URIBL_BLACK || URIBL_SC_SURBL ||
 URIBL_JP_SURBL || URIBL_OB_SURBL)  
(RAZOR2_CHECK || DCC_CHECK || PYZOR_CHECK))



meta SC_URIBL_SBL   ((URIBL_BLACK || URIBL_SC_SURBL ||
  URIBL_JP_SURBL || 
URIBL_OB_SURBL)  URIBL_SBL)


shortcircuit SC_URIBL_SURBL spam
shortcircuit SC_URIBL_SBL   spam
shortcircuit SC_URIBL_HASH spam

score SC_URIBL_SURBL100.00
score SC_URIBL_HASH 100.00
score SC_URIBL_SBL  100.00


I do not have a recent debug to show but I can say that from the 
debug I do see the SC_URIBL_SURBL trigger, after the earlier 
priority rules are ran. However, the remaining priorities are then 
ran, and if meeting critera, the RAZOR, DCC, and PYZOR rules run 
and then the SC_URIBL_HASH rule would trigger. Thus giving a total 
score of 200 + the scores for the URIBL/SURBL scores that hit and 
if included the Razor, DCC, and Pyzor scores as well.


I was thinking after the SC_URIBL_SURBL was triggered remaining 
rules would not run, and the spam classification would take 
precendence.


Am I overlooking the obvious, have I misunderstood how the SC 
should work, is it something with the rpm that was released by 
rpmforge? Any thoughts or insight would be appreciated. 
SA is, rather fortunately, circumventing what you're trying to do 
because of how DNS is handled internally.


DO NOT try to split up the priority of DNS based tests. Priority 
and shortcircuiting is intended to be used on *fast* rules, not 
slow ones.


 If you were successful, you would make the performance of 
SpamAssassin absurdly slow by serializing DNS queries. *OUCH*. SA 
normally runs these in parallel, and running them in serial would 
very seriously impact performance.


Currently, all DNS based tests run at their priority, but that 
only launches the DNS queries. All the results are gathered 
together at HARVEST_DNSBL_PRIORITY, which is currently set to 500. 
None of the rules will actually trigger until this point.

actually Matt, you're wrong ;)  if some of the network rules are
at a higher priority than others, and are used in shortcircuit rules,
SpamAssassin 3.2.x will indeed sleep until the results of those rules
arrive.

The idea is that, if you have the memory to support that degree of
concurrency, you can make a local policy decision to do that, instead
of doing the lookups at the MTA level which does effectively the
same thing.

This wait is logged, so you can spot it with --debug on.

--j.

With that said Justin, is the behavior I am seeing correct? Even 
though the first prioritized shortcircuit rule hits, and I see that 
in the debug log, shouldn't it be bypassing the remaining rules 
rather than continuing to process until all the shortcircuit 
priorities have ran?


 From reading the initial bug when this was originally featured, 
along with the man page, as well as a wiki post with an example by 
you, that is how I understood it to function.


actually, no, it sounds like a bug.could you open a bugzilla with
a demonstration config/test message?

--j.



Just to add, with my previous debug, I can confirm the waiting of the 
tests to finish as