Re: Hostkarma Blacklist Climbing the Charts
On tir 29 sep 2009 17:37:20 CEST, Warren Togami wrote
On 09/29/2009 12:27 AM, MySQL Student wrote:
header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
this one could be changed to some trusted variant for testing on local
trusted_networks
so change lastexternal to firsttrusted and if one want do please add
it to masscheck, if its not possible to test it, drop it :)
--
xpoint
Re: Hostkarma Blacklist Climbing the Charts
On 09/29/2009 12:27 AM, MySQL Student wrote:
Hi,
header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
tflags RCVD_IN_JMF_W net nice
score RCVD_IN_JMF_W -5
Hopefully my comment isn't out of place with the current discussion of
JMF/Hostkarma. I think this is not only a really bad default score,
but it should be reduced to -0.5 or perhaps not used at all.
I believe spamassassin does not assign any negative score to any
whitelist by default precisely for good reason.
USER_IN_DEF_DKIM_WL has the score -7.50 because it is a lot more certain
than a mere whitelist, having done cryptographic checking on the DKIM
signature to verify that the domain is both known non-spammer and it is
not spoofed.
Warren Togami
wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
Blaine Fleming wrote: Marc Perkel wrote: My NoBL list is similar to yellow except that you can skip black list lookup but maybe might be whitelisted somewhere. I keep seeing IPs that are on both the NoBL *and* the blacklist. An example of this 89.206.179.213. That IP currently returns 127.0.0.2 (blacklisted) and 127.0.0.5 (NoBL listed). Can you make sense of this entry? --Blaine That would be a bug in my system. I'll need to look into that.
Re: Hostkarma Blacklist Climbing the Charts
MySQL Student wrote: Hi, Hopefully my comment isn't out of place with the current discussion of JMF/Hostkarma. I think this is not only a really bad default score, but it should be reduced to -0.5 or perhaps not used at all. I have a money/fraud email that hit RCVD_IN_JMF_W that passed through these servers: Received: from 41.220.75.3 Received: from webmail.stu.qmul.ac.uk (138.37.100.37) by mercury.stu.qmul.ac.uk Received: from qmwmail2.stu.qmul.ac.uk ([138.37.100.210] Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6]) It also hit these other rules: X-Spam-Status: No, hits=1.3 tagged_above=-300.0 required=5.0 use_bayes=1 tests=AE_GBP, BAYES_50, LOTS_OF_MONEY, LOTTERY_PH_004470, LOTTO_RELATED, MONEY_TO_NO_R, RCVD_IN_DNSWL_MED, RCVD_IN_JMF_W, RELAYCOUNTRY_UK, SPF_FAIL, SPF_HELO_FAIL Unless I'm really missing something, which server has JMF/Hostkarma whitelisted that shouldn't be? This happens time after time. Yep - this isn't a perfect list. however if I got some good feedback on this I could weed out the white listes and get it more accurate. There are also a lot of hosts I could include with more data.
Re: Hostkarma Blacklist Climbing the Charts
On 29/09/2009 05:27, MySQL Student wrote:
header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
tflags RCVD_IN_JMF_W net nice
score RCVD_IN_JMF_W -5
Hopefully my comment isn't out of place with the current discussion of
JMF/Hostkarma. I think this is not only a really bad default score,
but it should be reduced to -0.5 or perhaps not used at all.
I have a money/fraud email that hit RCVD_IN_JMF_W that passed through
these servers:
Received: from 41.220.75.3
Received: from webmail.stu.qmul.ac.uk (138.37.100.37) by mercury.stu.qmul.ac.uk
Received: from qmwmail2.stu.qmul.ac.uk ([138.37.100.210]
Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6])
It also hit these other rules:
X-Spam-Status: No, hits=1.3 tagged_above=-300.0 required=5.0 use_bayes=1
tests=AE_GBP, BAYES_50, LOTS_OF_MONEY, LOTTERY_PH_004470,
LOTTO_RELATED, MONEY_TO_NO_R, RCVD_IN_DNSWL_MED, RCVD_IN_JMF_W,
RELAYCOUNTRY_UK, SPF_FAIL, SPF_HELO_FAIL
Unless I'm really missing something, which server has JMF/Hostkarma
whitelisted that shouldn't be?
This happens time after time.
I receive spam every single day from hosts listed on the HostKarma
whitelist. In comparison, it's very rare that I see any spam from hosts
listed on dnswl.org. I chose a score of -0.2 here.
--
Mike Cardwell - IT Consultant and LAMP developer
Cardwell IT Ltd. (UK Reg'd Company #06920226) http://cardwellit.com/
Re: Hostkarma Blacklist Climbing the Charts
On Tue, Sep 29, 2009 at 10:05:57AM +0200, Raymond Dijkxhoorn wrote: > Hi! > >>> Ouch, from your point of view it might be fine, but we see strange stuff >>> with DNSWL allready i certainly would not use this to shortcircuit >>> things. > >> What exactly is the strange stuff you see with DNSWL? >> >> Granted, I'm not processing millions of messages, only tens of thousands, >> but I'm not seeing anything fuzzy. I basically shortcircuit on DNSWL_MED and >> DNSWL_HI, when there aren't any suspicious rules hit (ClamAV/Sanesecurity, >> relay from africa, bayes over 60 etc). The FP rate is abysmally low. > > The regular things, whitelisted servers sending spams. So > shortcircuitting isnt an option for those and its also not whaqt DNSWL is > about. they WL sender mailservers, those could be an ISP also. You dont > want to shortcircuit them and say hey, someone put it on his whitelist, > feel free to spam me. Bad big mailservers sending mixed stuff are not supposed to be on MED/HI lists. If they are, you are supposed to report it. So I kind of disagree with you. I would imagine most people see <0.5% FP rates, even without any further meta checks.
Re: Hostkarma Blacklist Climbing the Charts
Marc Perkel wrote: > My NoBL list is similar to yellow except that you can skip black list > lookup but maybe might be whitelisted somewhere. I keep seeing IPs that are on both the NoBL *and* the blacklist. An example of this 89.206.179.213. That IP currently returns 127.0.0.2 (blacklisted) and 127.0.0.5 (NoBL listed). Can you make sense of this entry? --Blaine
Re: Hostkarma Blacklist Climbing the Charts
Hi! Ouch, from your point of view it might be fine, but we see strange stuff with DNSWL allready i certainly would not use this to shortcircuit things. What exactly is the strange stuff you see with DNSWL? Granted, I'm not processing millions of messages, only tens of thousands, but I'm not seeing anything fuzzy. I basically shortcircuit on DNSWL_MED and DNSWL_HI, when there aren't any suspicious rules hit (ClamAV/Sanesecurity, relay from africa, bayes over 60 etc). The FP rate is abysmally low. The regular things, whitelisted servers sending spams. So shortcircuitting isnt an option for those and its also not whaqt DNSWL is about. they WL sender mailservers, those could be an ISP also. You dont want to shortcircuit them and say hey, someone put it on his whitelist, feel free to spam me. Bye, Raymond.
Re: Hostkarma Blacklist Climbing the Charts
Hi!
header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
tflags RCVD_IN_JMF_W net nice
score RCVD_IN_JMF_W -5
Hopefully my comment isn't out of place with the current discussion of
JMF/Hostkarma. I think this is not only a really bad default score,
but it should be reduced to -0.5 or perhaps not used at all.
I have a money/fraud email that hit RCVD_IN_JMF_W that passed through
these servers:
Received: from 41.220.75.3
Received: from webmail.stu.qmul.ac.uk (138.37.100.37) by mercury.stu.qmul.ac.uk
Received: from qmwmail2.stu.qmul.ac.uk ([138.37.100.210]
Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6])
It also hit these other rules:
X-Spam-Status: No, hits=1.3 tagged_above=-300.0 required=5.0 use_bayes=1
tests=AE_GBP, BAYES_50, LOTS_OF_MONEY, LOTTERY_PH_004470,
LOTTO_RELATED, MONEY_TO_NO_R, RCVD_IN_DNSWL_MED, RCVD_IN_JMF_W,
RELAYCOUNTRY_UK, SPF_FAIL, SPF_HELO_FAIL
Unless I'm really missing something, which server has JMF/Hostkarma
whitelisted that shouldn't be?
You are not missing anything. Its my point also.
Bye,
Raymond.
Re: Hostkarma Blacklist Climbing the Charts
On Tue, Sep 29, 2009 at 09:29:16AM +0200, Raymond Dijkxhoorn wrote: > > Ouch, from your point of view it might be fine, but we see strange stuff > with DNSWL allready i certainly would not use this to shortcircuit > things. What exactly is the strange stuff you see with DNSWL? Granted, I'm not processing millions of messages, only tens of thousands, but I'm not seeing anything fuzzy. I basically shortcircuit on DNSWL_MED and DNSWL_HI, when there aren't any suspicious rules hit (ClamAV/Sanesecurity, relay from africa, bayes over 60 etc). The FP rate is abysmally low.
Re: Hostkarma Blacklist Climbing the Charts
Hi! If that's so, then we probably want that in the spamassassin rule name. Your wiki page suggests JMF is the name. A number of people probably already configured their spamassassin using your suggested JMF rule names and they would need to be educated to remove it. How about these for rule names, so the rule names are not too long? RCVD_HOSTKARMA_BL Black RCVD_HOSTKARMA_WL White RCVD_HOSTKARMA_YL Yellow RCVD_HOSTKARMA_BR Brown I would use the names that are advertised for months on the WIKI now, so you can override them and not duplicate lookups on installs that have it in their local.cf (or any place else). Why did you invent (Marc) completely new names out of the blue? The JMF_ stuff is there for months, please stick to it. We didnt invent those, you did Bye, Raymond.
Re: Hostkarma Blacklist Climbing the Charts
Hi! No one has actually implemented the rules for my blacklists correctly. My lists support both IP and hostname lookups. The hostname assumes that you have forward confirmed the RDNS so that you eliminate those who might spoof. Most people copy/paste from your wiki, so if this is true ... i am not sure where the real problem lies ;) Yellow means that the IP or hostname contains no useful information as to spam or no spam. On my system once I determine a host is yellow I skip all blacklists and whitelists tests. Yellow is for Yahoo, Hotmail, Gmail, etc where the IP has no information and all host tests are meaningless. My NoBL list is similar to yellow except that you can skip black list lookup but maybe might be whitelisted somewhere. Please dont combine black and whitelists together in one BL. This will trouble you. Many tools cannot look at the retern values. I think its a bad idea. You can say hey not my problem but it will give a BL a bad karma ;) If you just want to score points then Black, White, and Brown can be assigned points. Yellow should be zero points regardless of how it tests. Why would it be added to SA if the score is zero? I think the real power of my lists is in the host name lookups. It would be worthwhile to implement that. I think my white listing is very accurate at this point. The thing about white servers is that they aren't evasive like spammers. There should be some short circuiting options to reduce system load on SA for white lookups. Ouch, from your point of view it might be fine, but we see strange stuff with DNSWL allready i certainly would not use this to shortcircuit things. A question from the operational side, how many people are working on the BL? Just you i assume? Not telling this is bad, but its a risk when adding this into SA i feel personally. Same for the infra the BL is running on. I might sounds harsh, but i am rather carefull, then again, we have SA update. So it might not hurt that much. But during outages or DDoS it will hurt for hours till its gone again. Bye, Raymond.
Re: Hostkarma Blacklist Climbing the Charts
Hi,
> header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
> describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
> tflags RCVD_IN_JMF_W net nice
> score RCVD_IN_JMF_W -5
Hopefully my comment isn't out of place with the current discussion of
JMF/Hostkarma. I think this is not only a really bad default score,
but it should be reduced to -0.5 or perhaps not used at all.
I have a money/fraud email that hit RCVD_IN_JMF_W that passed through
these servers:
Received: from 41.220.75.3
Received: from webmail.stu.qmul.ac.uk (138.37.100.37) by mercury.stu.qmul.ac.uk
Received: from qmwmail2.stu.qmul.ac.uk ([138.37.100.210]
Received: from mail2.qmul.ac.uk (mail2.qmul.ac.uk [138.37.6.6])
It also hit these other rules:
X-Spam-Status: No, hits=1.3 tagged_above=-300.0 required=5.0 use_bayes=1
tests=AE_GBP, BAYES_50, LOTS_OF_MONEY, LOTTERY_PH_004470,
LOTTO_RELATED, MONEY_TO_NO_R, RCVD_IN_DNSWL_MED, RCVD_IN_JMF_W,
RELAYCOUNTRY_UK, SPF_FAIL, SPF_HELO_FAIL
Unless I'm really missing something, which server has JMF/Hostkarma
whitelisted that shouldn't be?
This happens time after time.
Thanks,
Alex
>
> header RCVD_IN_JMF_BL eval:check_rbl_sub('JMF-lastexternal', '127.0.0.2')
> describe RCVD_IN_JMF_BL Sender listed in JMF-BLACK
> tflags RCVD_IN_JMF_BL net
> score RCVD_IN_JMF_BL 3.0
>
> header RCVD_IN_JMF_BR eval:check_rbl_sub('JMF-lastexternal', '127.0.0.4')
> describe RCVD_IN_JMF_BR Sender listed in JMF-BROWN
> tflags RCVD_IN_JMF_BR net
> score RCVD_IN_JMF_BR 1.0
> ===8<---
>
> You pick the names and then the world can use them. The JMF names are out
> there today.
>
> {^_^} Joanne
>
Re: Hostkarma Blacklist Climbing the Charts
From: "Marc Perkel"
Sent: Monday, 2009/September/28 19:07
Warren Togami wrote:
On 09/28/2009 06:53 PM, Marc Perkel wrote:
...
I'd like to keep the name HOSTKARMA as standard.
If that's so, then we probably want that in the spamassassin rule name.
Your wiki page suggests JMF is the name. A number of people probably
already configured their spamassassin using your suggested JMF rule names
and they would need to be educated to remove it.
How about these for rule names, so the rule names are not too long?
RCVD_HOSTKARMA_BL Black
RCVD_HOSTKARMA_WL White
RCVD_HOSTKARMA_YL Yellow
RCVD_HOSTKARMA_BR Brown
Warren Togami
wtog...@redhat.com
Hi Warren,
No one has actually implemented the rules for my blacklists correctly. My
lists support both IP and hostname lookups. The hostname assumes that you
have forward confirmed the RDNS so that you eliminate those who might
spoof.
Yellow means that the IP or hostname contains no useful information as to
spam or no spam. On my system once I determine a host is yellow I skip all
blacklists and whitelists tests. Yellow is for Yahoo, Hotmail, Gmail, etc
where the IP has no information and all host tests are meaningless.
My NoBL list is similar to yellow except that you can skip black list
lookup but maybe might be whitelisted somewhere.
If you just want to score points then Black, White, and Brown can be
assigned points. Yellow should be zero points regardless of how it tests.
I think the real power of my lists is in the host name lookups. It would
be worthwhile to implement that.
I think my white listing is very accurate at this point. The thing about
white servers is that they aren't evasive like spammers. There should be
some short circuiting options to reduce system load on SA for white
lookups.
And - I'm hoping others will catch on to some of the things I'm doing
because when other people adopt my tricks they usually improve them.
Let me know what I need to do to help make this happen.
So what SHOULD this, which I clipped off your site, really look like
for SpamAssassin rules?
===8<---
header __RCVD_IN_JMF
eval:check_rbl('JMF-lastexternal','hostkarma.junkemailfilter.com.')
describe __RCVD_IN_JMF Sender listed in JunkEmailFilter
tflags __RCVD_IN_JMF net
header RCVD_IN_JMF_W eval:check_rbl_sub('JMF-lastexternal', '127.0.0.1')
describe RCVD_IN_JMF_W Sender listed in JMF-WHITE
tflags RCVD_IN_JMF_W net nice
score RCVD_IN_JMF_W -5
header RCVD_IN_JMF_BL eval:check_rbl_sub('JMF-lastexternal', '127.0.0.2')
describe RCVD_IN_JMF_BL Sender listed in JMF-BLACK
tflags RCVD_IN_JMF_BL net
score RCVD_IN_JMF_BL 3.0
header RCVD_IN_JMF_BR eval:check_rbl_sub('JMF-lastexternal', '127.0.0.4')
describe RCVD_IN_JMF_BR Sender listed in JMF-BROWN
tflags RCVD_IN_JMF_BR net
score RCVD_IN_JMF_BR 1.0
===8<---
You pick the names and then the world can use them. The JMF names are out
there today.
{^_^}Joanne
Re: Hostkarma Blacklist Climbing the Charts
On 09/28/2009 10:07 PM, Marc Perkel wrote: I'd like to keep the name HOSTKARMA as standard. If that's so, then we probably want that in the spamassassin rule name. Your wiki page suggests JMF is the name. A number of people probably already configured their spamassassin using your suggested JMF rule names and they would need to be educated to remove it. How about these for rule names, so the rule names are not too long? RCVD_HOSTKARMA_BL Black RCVD_HOSTKARMA_WL White RCVD_HOSTKARMA_YL Yellow RCVD_HOSTKARMA_BR Brown Hi Marc, I appreciate your desire for everyone to wholly benefit from your work, but please let us implement this for spamassassin in stages starting from the lowest hanging fruit. First please confirm that you approve of the above new rule names, if you don't want it to be known as JMF. Hi Warren, No one has actually implemented the rules for my blacklists correctly. My lists support both IP and hostname lookups. The hostname assumes that you have forward confirmed the RDNS so that you eliminate those who might spoof. Please explain in greater detail? Can this be determined wholly from the Headers and message body after the MTA had passed the mail to the MDA? Yellow means that the IP or hostname contains no useful information as to spam or no spam. On my system once I determine a host is yellow I skip all blacklists and whitelists tests. Yellow is for Yahoo, Hotmail, Gmail, etc where the IP has no information and all host tests are meaningless. My NoBL list is similar to yellow except that you can skip black list lookup but maybe might be whitelisted somewhere. Please help me better understand, what are examples of a sequence of events that would land an IP address on the NoBL? If you just want to score points then Black, White, and Brown can be assigned points. Yellow should be zero points regardless of how it tests. I am aware that Yellow isn't useful for scores. It is however useful for statistical analysis in masschecks, and it doesn't cost spamassassin any more to print if it hits. In particular I'm looking to see if there are any reliable trends of overlap between Yellow and other spamassassin rules. I think the real power of my lists is in the host name lookups. It would be worthwhile to implement that. Please describe how this is more effective than IP lookups? I think my white listing is very accurate at this point. The thing about white servers is that they aren't evasive like spammers. There should be some short circuiting options to reduce system load on SA for white lookups. Generally spamassassin does not short-circuit by default for any reason. There is an option to do so, but I think it is only to stop testing rules if the score goes beyond a certain point. Please file a separate bug for this if it is important to you. Warren Togami wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
Warren Togami wrote: On 09/28/2009 06:53 PM, Marc Perkel wrote: Warren Togami wrote: On 09/28/2009 01:32 PM, Marc Perkel wrote: I'd be interested in how well it worked. Is there anything I need to do to help? http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists Could you provide a URL redirector to this page? This URL is very long. Perhaps shorter URL in the describe of each rule like: http://hostkarma.junkemailfilter.com ? I'm working on that. Trying to figure out how to give it an A record. This URL will be in spam reports so folks can click-thru and see why their message triggered on this rule. https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212 I filed the request to add it to sandbox for testing here. The line wrapping got screwed up in Bugzilla. You might also want to consider standardizing the name of the blacklist. You called it JEF earlier in this thread. Your Wiki page calls the rules JMF. And it also seems to be called Hostkarma. It will be confusing to people if they see different names referring to the same thing. Perhaps we should call it JMF to avoid confusion? I'd like to keep the name HOSTKARMA as standard. If that's so, then we probably want that in the spamassassin rule name. Your wiki page suggests JMF is the name. A number of people probably already configured their spamassassin using your suggested JMF rule names and they would need to be educated to remove it. How about these for rule names, so the rule names are not too long? RCVD_HOSTKARMA_BL Black RCVD_HOSTKARMA_WL White RCVD_HOSTKARMA_YL Yellow RCVD_HOSTKARMA_BR Brown Warren Togami wtog...@redhat.com Hi Warren, No one has actually implemented the rules for my blacklists correctly. My lists support both IP and hostname lookups. The hostname assumes that you have forward confirmed the RDNS so that you eliminate those who might spoof. Yellow means that the IP or hostname contains no useful information as to spam or no spam. On my system once I determine a host is yellow I skip all blacklists and whitelists tests. Yellow is for Yahoo, Hotmail, Gmail, etc where the IP has no information and all host tests are meaningless. My NoBL list is similar to yellow except that you can skip black list lookup but maybe might be whitelisted somewhere. If you just want to score points then Black, White, and Brown can be assigned points. Yellow should be zero points regardless of how it tests. I think the real power of my lists is in the host name lookups. It would be worthwhile to implement that. I think my white listing is very accurate at this point. The thing about white servers is that they aren't evasive like spammers. There should be some short circuiting options to reduce system load on SA for white lookups. And - I'm hoping others will catch on to some of the things I'm doing because when other people adopt my tricks they usually improve them. Let me know what I need to do to help make this happen.
Re: Hostkarma Blacklist Climbing the Charts
On 09/28/2009 06:53 PM, Marc Perkel wrote: Warren Togami wrote: On 09/28/2009 01:32 PM, Marc Perkel wrote: I'd be interested in how well it worked. Is there anything I need to do to help? http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists Could you provide a URL redirector to this page? This URL is very long. Perhaps shorter URL in the describe of each rule like: http://hostkarma.junkemailfilter.com ? I'm working on that. Trying to figure out how to give it an A record. This URL will be in spam reports so folks can click-thru and see why their message triggered on this rule. https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212 I filed the request to add it to sandbox for testing here. The line wrapping got screwed up in Bugzilla. You might also want to consider standardizing the name of the blacklist. You called it JEF earlier in this thread. Your Wiki page calls the rules JMF. And it also seems to be called Hostkarma. It will be confusing to people if they see different names referring to the same thing. Perhaps we should call it JMF to avoid confusion? I'd like to keep the name HOSTKARMA as standard. If that's so, then we probably want that in the spamassassin rule name. Your wiki page suggests JMF is the name. A number of people probably already configured their spamassassin using your suggested JMF rule names and they would need to be educated to remove it. How about these for rule names, so the rule names are not too long? RCVD_HOSTKARMA_BL Black RCVD_HOSTKARMA_WL White RCVD_HOSTKARMA_YL Yellow RCVD_HOSTKARMA_BR Brown Warren Togami wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
Warren Togami wrote: On 09/28/2009 01:32 PM, Marc Perkel wrote: I'd be interested in how well it worked. Is there anything I need to do to help? http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists Could you provide a URL redirector to this page? This URL is very long. Perhaps shorter URL in the describe of each rule like: http://hostkarma.junkemailfilter.com ? I'm working on that. Trying to figure out how to give it an A record. This URL will be in spam reports so folks can click-thru and see why their message triggered on this rule. https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212 I filed the request to add it to sandbox for testing here. The line wrapping got screwed up in Bugzilla. You might also want to consider standardizing the name of the blacklist. You called it JEF earlier in this thread. Your Wiki page calls the rules JMF. And it also seems to be called Hostkarma. It will be confusing to people if they see different names referring to the same thing. Perhaps we should call it JMF to avoid confusion? I'd like to keep the name HOSTKARMA as standard.
Re: Hostkarma Blacklist Climbing the Charts
On 09/28/2009 01:32 PM, Marc Perkel wrote: I'd be interested in how well it worked. Is there anything I need to do to help? http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists Could you provide a URL redirector to this page? This URL is very long. Perhaps shorter URL in the describe of each rule like: http://hostkarma.junkemailfilter.com ? This URL will be in spam reports so folks can click-thru and see why their message triggered on this rule. https://issues.apache.org/SpamAssassin/show_bug.cgi?id=6212 I filed the request to add it to sandbox for testing here. The line wrapping got screwed up in Bugzilla. You might also want to consider standardizing the name of the blacklist. You called it JEF earlier in this thread. Your Wiki page calls the rules JMF. And it also seems to be called Hostkarma. It will be confusing to people if they see different names referring to the same thing. Perhaps we should call it JMF to avoid confusion? Warren Togami wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
On 09/28/2009 01:45 PM, Marc Perkel wrote: I'd be interested in how well it worked. Is there anything I need to do to help? 1) I'm waiting to hear back what it will take for me to gain commit access so I can add this to the sandbox. 2) Do you mind hundreds of thousands of rapid DNS lookups during masschecks? If not then the two largest servers doing masschecks could probably use rsync access to your data. Warren Togami wtog...@redhat.com I think I have a lot of capacity. I suppose we'll see. I should be able to handle the load. If not then I'll find out. BTW - if JEF were included in the standard distribution, about how much bandwidth and server power would I need to handle it? We don't really know how much traffic being default in spamassassin will cause. You would have to ask the other list maintainers like DNSWL if they have any statistics. If PSBL becomes enabled by default in spamassassin-3.3.0 then we may be able to estimate the jump in traffic from that. Warren Togami wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
Warren Togami wrote: On 09/28/2009 01:32 PM, Marc Perkel wrote: Warren Togami wrote: On 07/09/2009 09:57 PM, Marc Perkel wrote: For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html Hi Marc, http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists Have you considered adding your RCVD_IN_JMF* rules to the sandbox so we can easily compute some statistics from the weekly net masschecks? For example... http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail http://psbl.surriel.com/ I've been following the weekly masscheck results on PSBL. The false positives have helped us to identify problems in PSBL's trap filtering logic and made it safer to use. The statistics are looking pretty good now, so hopefully PSBL will become enabled by default in spamassassin-3.3.0. I'd be interested in how well it worked. Is there anything I need to do to help? 1) I'm waiting to hear back what it will take for me to gain commit access so I can add this to the sandbox. 2) Do you mind hundreds of thousands of rapid DNS lookups during masschecks? If not then the two largest servers doing masschecks could probably use rsync access to your data. Warren Togami wtog...@redhat.com I think I have a lot of capacity. I suppose we'll see. I should be able to handle the load. If not then I'll find out. BTW - if JEF were included in the standard distribution, about how much bandwidth and server power would I need to handle it?
Re: Hostkarma Blacklist Climbing the Charts
On 09/28/2009 01:32 PM, Marc Perkel wrote: Warren Togami wrote: On 07/09/2009 09:57 PM, Marc Perkel wrote: For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html Hi Marc, http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists Have you considered adding your RCVD_IN_JMF* rules to the sandbox so we can easily compute some statistics from the weekly net masschecks? For example... http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail http://psbl.surriel.com/ I've been following the weekly masscheck results on PSBL. The false positives have helped us to identify problems in PSBL's trap filtering logic and made it safer to use. The statistics are looking pretty good now, so hopefully PSBL will become enabled by default in spamassassin-3.3.0. I'd be interested in how well it worked. Is there anything I need to do to help? 1) I'm waiting to hear back what it will take for me to gain commit access so I can add this to the sandbox. 2) Do you mind hundreds of thousands of rapid DNS lookups during masschecks? If not then the two largest servers doing masschecks could probably use rsync access to your data. Warren Togami wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
Warren Togami wrote: On 07/09/2009 09:57 PM, Marc Perkel wrote: For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html Hi Marc, http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists Have you considered adding your RCVD_IN_JMF* rules to the sandbox so we can easily compute some statistics from the weekly net masschecks? For example... http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail http://psbl.surriel.com/ I've been following the weekly masscheck results on PSBL. The false positives have helped us to identify problems in PSBL's trap filtering logic and made it safer to use. The statistics are looking pretty good now, so hopefully PSBL will become enabled by default in spamassassin-3.3.0. Separate question for you: Would you mind if PSBL used your whitelist and yellowlist to help exclude false positive IP's? Warren Togami wtog...@redhat.com I'd be interested in how well it worked. Is there anything I need to do to help?
Re: Hostkarma Blacklist Climbing the Charts
On 07/09/2009 09:57 PM, Marc Perkel wrote: For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html Hi Marc, http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists#How_to_use_the_Lists Have you considered adding your RCVD_IN_JMF* rules to the sandbox so we can easily compute some statistics from the weekly net masschecks? For example... http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL/detail http://ruleqa.spamassassin.org/20090926-r819101-n/RCVD_IN_PSBL_2WEEKS/detail http://psbl.surriel.com/ I've been following the weekly masscheck results on PSBL. The false positives have helped us to identify problems in PSBL's trap filtering logic and made it safer to use. The statistics are looking pretty good now, so hopefully PSBL will become enabled by default in spamassassin-3.3.0. Separate question for you: Would you mind if PSBL used your whitelist and yellowlist to help exclude false positive IP's? Warren Togami wtog...@redhat.com
Re: Hostkarma Blacklist Climbing the Charts
Charles Gregory wrote: A more interesting comparison would be to see how much stuff is NOT caught by spamhaus, but caught by your list or others :) -C Some stats from my mail server, zen.spamhaus is deployed at the smtp level, so these are hits against 323 spam samples (as detected by SA) that made it through smtp restrictions: ## DNSBL Statistics ## 102 RCVD_IN_UCE_COMBINED 95 RCVD_IN_BRBL 70 RCVD_IN_JMF_BL 62 RCVD_IN_UCEPROTECT1 49 RCVD_IN_UCEPROTECT2 40 RCVD_IN_UBL_UNSUB 34 RCVD_IN_UCEPROTECT3 32 RCVD_IN_SBLXBL 24 RCVD_IN_SORBS_WEB 21 RCVD_IN_BL_SPAMCOP_NET 17 RCVD_IN_PSBL 10 RCVD_IN_JMF_BR 9 RCVD_IN_IADB_SPF 9 RCVD_IN_IADB_LISTED 4 RCVD_IN_DNSWL_LOW 3 RCVD_IN_BSP_TRUSTED 2 RCVD_IN_SORBS_DUL 2 RCVD_IN_NJABL_RELAY 2 RCVD_IN_NJABL_PROXY 1 RCVD_IN_NJABL_SPAM 1 RCVD_IN_DNSWL_MED 323 Total Spam UCE_COMBINED is a hit against any of UCEPROTECT 1, 2 or 3. In my experience UCEPROTECT can and does give occasional FPs. RCVD_IN_SBLXBL checks all IPs, not just last external, hence why we see still see some hits even though zen.spamhaus is already used. IMHO, BRBL and JMF_BL both do a good job at adding a little weight to spam making it past zen.spamhaus.org. All the easy to detect stuff has long since been blocked, so hits at this stage are against the last ~1% of spam that has slipped past everything else, so don't judge the apparent ~20% hit rate too harshly. I still only trust spamhaus to outright reject mail at the smtp level.
Re: Hostkarma Blacklist Climbing the Charts
Charles Gregory wrote: A more interesting comparison would be to see how much stuff is NOT caught by spamhaus, but caught by your list or others :) Right -- that gives you more of a sense of the value of a new list for a system which already checks other lists. -- J.D. Falk Return Path Inc http://www.returnpath.net/
Re: Hostkarma Blacklist Climbing the Charts
A more interesting comparison would be to see how much stuff is NOT caught by spamhaus, but caught by your list or others :) -C On Thu, 9 Jul 2009, Marc Perkel wrote: For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html
Re: Hostkarma Blacklist Climbing the Charts
On Fri, 2009-07-10 at 05:42 -0600, LuKreme wrote: > On 10-Jul-2009, at 05:18, rich...@buzzhost.co.uk wrote: > > There is a load of noise in NANAE about the Court coming to a > > compensation decision and Spamhaus being 'broke' hence my concern. > > Is NANAE in a time-warp? The court (in the US) has no power to compel > spamhaus (in the UK) to pay a cent. Don't you start! That's what the trolls are fighting about!
Re: Hostkarma Blacklist Climbing the Charts
On 10-Jul-2009, at 05:18, rich...@buzzhost.co.uk wrote: There is a load of noise in NANAE about the Court coming to a compensation decision and Spamhaus being 'broke' hence my concern. Is NANAE in a time-warp? The court (in the US) has no power to compel spamhaus (in the UK) to pay a cent. -- And now, the rest of the story
Re: Hostkarma Blacklist Climbing the Charts
On Fri, 2009-07-10 at 04:57 -0600, LuKreme wrote: > On 10-Jul-2009, at 01:25, rich...@buzzhost.co.uk wrote: > > On Thu, 2009-07-09 at 18:57 -0700, Marc Perkel wrote: > >> For what it's worth I'm now ahead of Barracuda on Jeff Makey's > >> blacklist > >> comparison chart. Not a scientific comparison but it's about all > >> there > >> is to compare blacklists. Now only abuseat.org and spamhaus have me > >> beat. (apews doesn't count because they blacklist everything) > >> > >> http://www.sdsc.edu/~jeff/spam/cbc.html > > > Zen still tops it - and rightly so. It's a fantastic list. The > > question > > is how much longer is spamhaus going to exists after they lost that > > e360 > > case? Could it spell the end for them? > > Spamhaus 'lost' that case a long time ago. It's made no difference, > and e360 no longer exists. > There is a load of noise in NANAE about the Court coming to a compensation decision and Spamhaus being 'broke' hence my concern.
Re: Hostkarma Blacklist Climbing the Charts
On 10-Jul-2009, at 01:25, rich...@buzzhost.co.uk wrote: On Thu, 2009-07-09 at 18:57 -0700, Marc Perkel wrote: For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html Zen still tops it - and rightly so. It's a fantastic list. The question is how much longer is spamhaus going to exists after they lost that e360 case? Could it spell the end for them? Spamhaus 'lost' that case a long time ago. It's made no difference, and e360 no longer exists. -- Otto: Apes don't read philosophy. Wanda: Yes, they do Otto, they just don't understand it.
Re: Hostkarma Blacklist Climbing the Charts
On Thu, 2009-07-09 at 18:57 -0700, Marc Perkel wrote: > For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist > comparison chart. Not a scientific comparison but it's about all there > is to compare blacklists. Now only abuseat.org and spamhaus have me > beat. (apews doesn't count because they blacklist everything) > > http://www.sdsc.edu/~jeff/spam/cbc.html > > Zen still tops it - and rightly so. It's a fantastic list. The question is how much longer is spamhaus going to exists after they lost that e360 case? Could it spell the end for them? Barracuda always intended to charge for access to their list. It's been free for around a year now and I wonder if and when that will happen. If you take spamhaus and sorbs out of the frame it green lights the digital shoplifters at Barracuda to start charging. Mind you, you have to laugh at an organisation that buys in some of it's blacklist data and ends up listing it's own customer barracuda devices LOL. Better hope that new lists spring up and Hostkarma keeps climbing. I don't have the experience of apews blacklisting everything. I've had two hits from them in six months. They are at the bottom of my lookup food chain, but I can't cite them as irresponsible in their listing.
Re: Hostkarma Blacklist Climbing the Charts
Hi! For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html Beat you with what, false positives? :-) Indeed, it doesnt tell much about -quality- of a list. So its only maths. Bye, Raymond.
Hostkarma Blacklist Climbing the Charts
For what it's worth I'm now ahead of Barracuda on Jeff Makey's blacklist comparison chart. Not a scientific comparison but it's about all there is to compare blacklists. Now only abuseat.org and spamhaus have me beat. (apews doesn't count because they blacklist everything) http://www.sdsc.edu/~jeff/spam/cbc.html
