Re: Very simple user query...

[Rob Skedgell]

On Wednesday 14 Sep 2005 22:44, jdow wrote:
> From: "Rob Skedgell" <[EMAIL PROTECTED]>
>
> On Tuesday 13 Sep 2005 21:15, Markus Eskola wrote:
> [...]
>
> > Just a quick question regarding the reporting... Do you guys report
> > all spam (including the once that SA allready caught) or only the
> > ones that got thru the net?
> >
> > Currently in my setup I have 3-4 diffrent users who move all the
> > spam that got thru into certain folders eg SPAM under IMAP. These
> > folders are scanned, emptied and reported once a night thru a
> > script. If someone has a more effectie way, I'd appreciate a hint
> > in the right direction.
>
> Most of it (5.0 <= score <= 30.0) gets LARTed by a java program that
> goes through the "confirmed spam" IMAP folder to the
^ e.g. *manually* confirmed as spam, not 
  just scored/flagged as such
[...]
> Ah, you are one of the people polluting the BLs. Thanks not.

No.

It was entirely my fault for not making it clearer that I do check the 
confirmed spam folder very carefully first, before running the 
reporting tool. It most certainly doesn't do anything like running from 
cron, nor will it ever do that. If the IMAP seen flag isn't set on a 
mail in that folder, it gets skipped as a safeguard against 
carelessness on my part - the last thing I want is a mail that's just 
been delivered to be reported without checking.

>
> Why not be a little saner and adopt a score higher than 5.0, a very
> marginal spam score, for reporting. That way you are not reporting
> false alarms and injuring innocent people.

See above. It's actually (score>=5.0 && manually_confirmed_as_spam)

I should stress that any mails I report are checked manually *first*. 
False positives do *not* go to NANAS, SpamCop, the originating ISP etc.

False positives get dragged out of the spam folder, my whitelists fixed 
(sometimes via whitelist_from_rcvd, sometimes in the PostgreSQL 
database used by a couple of ACLs, depending on the context).

You can check the NANAS posts here 

if you like. See many false positives? No, nor me.

I very rarely mis-identify a false positive as spam, and on those rare 
occasions the abuse contact who just got the LART in error gets a 
grovelling apology from me for wasting their time.

-- 
Rob Skedgell <[EMAIL PROTECTED]>


pgp6PTjZDTQMD.pgp
Description: PGP signature

Re: Very simple user query...

[jdow]

From: "Rob Skedgell" <[EMAIL PROTECTED]>

On Tuesday 13 Sep 2005 21:15, Markus Eskola wrote:
[...]

Just a quick question regarding the reporting... Do you guys report
all spam (including the once that SA allready caught) or only the  
ones that got thru the net? 


Currently in my setup I have 3-4 diffrent users who move all the spam
that got thru into certain folders eg SPAM under IMAP. These folders 
are scanned, emptied and reported once a night thru a script. 
If someone has a more effectie way, I'd appreciate a hint in the right 
direction. 


Most of it (5.0 <= score <= 30.0) gets LARTed by a java program that 
goes through the "confirmed spam" IMAP folder to the contacts.abuse.net 
addresses for the IP address that sent to my MX, SpamCop and is also 
posted to NANAS. If it scores over 30 it hits a discard ACL in exim.


Anything that sneaks through under 5.0 or went to a role account is also 
singled out for extra vindictiveness and LARTed manually to anything 
SpamTool missed and whois data checked very carefully for RFCI whois 
eligibility (and a WDPRS report).


Oh, and I have a patched Mail::SpamAssassin::Plugin::URIDNSBL to pass 
the domain names scanned over UDP to another listening application that 
tests for "missing" entries in RFCI bogusmx and automatically sends the 
submission by email. It also sends BCCs to postmaster@ and abuse@ so 
that victims of "friendly fire" (through inadvertently using a CNAME 
for their MX rather then deliberately registering 127.0.0.1) can get 
unlisted.


++
Ah, you are one of the people polluting the BLs. Thanks not.

Why not be a little saner and adopt a score higher than 5.0, a very
marginal spam score, for reporting. That way you are not reporting
false alarms and injuring innocent people.

{^_^}

Re: Very simple user query...

[Nix]

On Mon, 12 Sep 2005, Steve whispered secretively:
> Genius answer! For some reason it had completely escaped my notice
> that all of the spams missed by SA over the past month had a
> uk.geocities.com address!  I've opted for a score of 4 for any mail
> mentioning a uk.geocities.com URL - which is hopefully good enough

For me, Bayes catches them all, so a score of 1.1 for stuff mentioning
geocities is sufficient to push the evil emails over the 5.0 threshold.

-- 
`One cannot, after all, be expected to read every single word
 of a book whose author one wishes to insult.' --- Richard Dawkins

Re: Very simple user query...

[Lefteris Tsintjelis]

Anybody got an idea
how to prevent that confirmation?


Use spamcop_to_address "quick." instead of "submit." but thats something
you have to activate. The site has further info about this.

Re: Very simple user query...

[Michael Monnerie]

On Mittwoch, 14. September 2005 16:12 Lefteris Tsintjelis wrote:
> Did I also mention the use of quite a few SPAM
> traps and grey listing (both are very effective).

Oh I love those, too *beg*

> Only if you are a registered (paid) user, then it is definetly worth
> reporting and things are listed relativly fast (I have a few
> objections to the exceptions he is making in favor of a large and
> pretty well known site, SPAM is SPAM no matter where it comes from)
> but I guess overall, its as you say it is. If you are not a
> registered user though IMHO then its a waste of resources.

I registered, but do not pay. I just changed my script to use 
"spamassin" and not "sa-learn", now it reports to spamcop too. The 
problem is, I get a mail per reported mail, where I have to click on a 
link and press "confirm" on that page - annoying. Anybody got an idea 
how to prevent that confirmation?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc  ---   it-management Michael Monnerie
// http://zmi.at   Tel: 0660/4156531  Linux 2.6.11
// PGP Key:   "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952  F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879


pgplIxh6b1nxf.pgp
Description: PGP signature

Re: Very simple user query...

[Lefteris Tsintjelis]

I prefer to send it immediately which makes the updates of DCC and
razor even faster. 


How do you do it? Do you report back automatically every detected SPAM? 
That shouldn't be done, as I read from the homepage.


Not out of the box, I agree with that. I am using 3 threshold levels
and tested, trained and fined tuned the whole system for a while
before I turn on the auto reporting. Everything above a level, is auto
reported with a hit rate of 99.99%. I use a dedicated machine to
redirect, report and hold that SPAM for a while for this job only.
Everything in the middle I pass it through a couple of scripts,
analyze it, and what is left of it (not really much) manually report
it or take action against it to not enter the site again, but that
depends on the case. Did I also mention the use of quite a few SPAM
traps and grey listing (both are very effective).

What I am not so sure of is the SpamCop reporting. 
It seems that its a complete waste since the black list that

maintains is not getting updated by any of those reports.


AFAIK, spamcop sends e-mail to the admins responsible for that IP, and 
so it should help that ISPs get reports of zombies, relays, and so on. 
It fights on another level, but that one should be quite effective.


Only if you are a registered (paid) user, then it is definetly worth
reporting and things are listed relativly fast (I have a few
objections to the exceptions he is making in favor of a large and
pretty well known site, SPAM is SPAM no matter where it comes from)
but I guess overall, its as you say it is. If you are not a
registered user though IMHO then its a waste of resources.

Re: Very simple user query...

[Rob Skedgell]

On Tuesday 13 Sep 2005 21:15, Markus Eskola wrote:
[...]
> Just a quick question regarding the reporting... Do you guys report
> all spam (including the once that SA allready caught) or only the  
> ones that got thru the net? 
> 
> Currently in my setup I have 3-4 diffrent users who move all the spam
> that got thru into certain folders eg SPAM under IMAP. These folders 
> are scanned, emptied and reported once a night thru a script. 
> If someone has a more effectie way, I'd appreciate a hint in the right 
> direction. 

Most of it (5.0 <= score <= 30.0) gets LARTed by a java program that 
goes through the "confirmed spam" IMAP folder to the contacts.abuse.net 
addresses for the IP address that sent to my MX, SpamCop and is also 
posted to NANAS. If it scores over 30 it hits a discard ACL in exim.

Anything that sneaks through under 5.0 or went to a role account is also 
singled out for extra vindictiveness and LARTed manually to anything 
SpamTool missed and whois data checked very carefully for RFCI whois 
eligibility (and a WDPRS report).

Oh, and I have a patched Mail::SpamAssassin::Plugin::URIDNSBL to pass 
the domain names scanned over UDP to another listening application that 
tests for "missing" entries in RFCI bogusmx and automatically sends the 
submission by email. It also sends BCCs to postmaster@ and abuse@ so 
that victims of "friendly fire" (through inadvertently using a CNAME 
for their MX rather then deliberately registering 127.0.0.1) can get 
unlisted.

-- 
Rob Skedgell <[EMAIL PROTECTED]>


pgpY8xMqwqXAW.pgp
Description: PGP signature

Re: Very simple user query...

[Michael Monnerie]

On Mittwoch, 14. September 2005 14:40 Lefteris Tsintjelis wrote:
> I prefer to send it immediately which makes the updates of DCC and
> razor even faster. 

How do you do it? Do you report back automatically every detected SPAM? 
That shouldn't be done, as I read from the homepage.

> What I am not so sure of is the SpamCop reporting. 
> It seems that its a complete waste since the black list that
> maintains is not getting updated by any of those reports.

AFAIK, spamcop sends e-mail to the admins responsible for that IP, and 
so it should help that ISPs get reports of zombies, relays, and so on. 
It fights on another level, but that one should be quite effective.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc  ---   it-management Michael Monnerie
// http://zmi.at   Tel: 0660/4156531  Linux 2.6.11
// PGP Key:   "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952  F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879


pgpBLTRnZZjQs.pgp
Description: PGP signature

Re: Very simple user query...

[Lefteris Tsintjelis]

Michael Monnerie wrote:

On Dienstag, 13. September 2005 22:15 Markus Eskola wrote:


Just a quick question regarding the reporting... Do you guys report
all spam (including the once that SA allready caught) or only the
ones that got thru the net?


All, with no exceptions made.

I believe it should be done at least once per hour - so DCC and razor 
have it quickly detected. Otherwise, spammers have time until the night 
to send to a lot of servers. I currently do it in 10 minute intervals, 
as it doesn't really create too much load.


I prefer to send it immediately which makes the updates of DCC and
razor even faster. What I am not so sure of is the SpamCop reporting.
It seems that its a complete waste since the black list that maintains
is not getting updated by any of those reports.

Re: Very simple user query...

[Michael Monnerie]

On Dienstag, 13. September 2005 22:15 Markus Eskola wrote:
> Just a quick question regarding the reporting... Do you guys report
> all spam (including the once that SA allready caught) or only the
> ones that got thru the net?

All - because others may have other rules, probably not identifying this 
as SPAM. Imagine you get >5 points because your bayes is 100% sure, but 
there's no hit on DCC, razor, etc. It's good for the others to report 
it, so DCC and razor know it's SPAM, and therefore the next one who 
receives it knows for sure about it.

> Currently in my setup I have 3-4 diffrent users who move all the spam
> that got thru into certain folders eg SPAM under IMAP. These folders
> are scanned, emptied and reported once a night thru a script.
> If someone has a more effectie way, I'd appreciate a hint in the
> right direction.

I believe it should be done at least once per hour - so DCC and razor 
have it quickly detected. Otherwise, spammers have time until the night 
to send to a lot of servers. I currently do it in 10 minute intervals, 
as it doesn't really create too much load.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc  ---   it-management Michael Monnerie
// http://zmi.at   Tel: 0660/4156531  Linux 2.6.11
// PGP Key:   "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952  F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879


pgpvkR51AknVN.pgp
Description: PGP signature

Re: Very simple user query...

[Steve [Spamassasin]]

jdow wrote:

I absolutely do not want to report automatically - in the sense that 
I am adamant that I want human intervention before reporting.  
Conversely - given the task of establishing a remote shell; finding 
the correct email in maildir - and verifying it is indeed the mail I 
determined was a spam in my email client - followed by manually 
reporting it individually to each service... I'm inclined not to 
bother.  If, for example I had an IMAP folder into which I drop spam 
that my mail server should report on my behalf -then reporting would 
become far less of a chore.




Simple matter of coding. That is how I handle ham and spam training. I 
simply
dunk it into ham and spam folders and let a cron job run sa-learn over 
the
two folders. In this case you'd probably have to code up something 
that takes
the folder apart properly, forwards the mail appropriately, then 
tosses it.
I haven't done such a thing. But there are perl tools for reading 
messages

via IMAP that could be used as the core of a new tool.



Hmmm - given that this seems such an obvious thing to want, and because 
I'm quite laz^H^H^Hbusy these days, I'd hoped that there such  thing 
pre-existed.  It strikes me that the best way to do this would be with a 
daemon which monitors the IMAP folders for user-identified spam; salearn 
and report it - then move it to the same folder as the automatically 
identified spam.  I realise that it wouldn't be a herculean effort to 
implement this but I'm very reluctant to re-invent the wheel.

Re: Very simple user query...

[jdow]

From: "Steve [Spamassasin]" <[EMAIL PROTECTED]>


jdow wrote:


You do not say which version of spamassassin you are using. If it is not
3.04 an upgrade might help.


It's 3.04 - the latest stable build that's made it into "Gentoo Portage"


   * Is there somewhere where I can report spams which aren't caught by
 the default configuration in order to feed-back into future
 improvements?


There are places to report them manually.


I'm familiar with razor-report, for example - but it is a real pain to 
mess about with this command line tool when all my mail is managed 
remotely over IMAP



I have a strong personal bias against automating anything related to
spam REPORTING. Please examine the downsides of automatic reporting
before proceeding.


I absolutely do not want to report automatically - in the sense that I am 
adamant that I want human intervention before reporting.  Conversely - 
given the task of establishing a remote shell; finding the correct email 
in maildir - and verifying it is indeed the mail I determined was a spam 
in my email client - followed by manually reporting it individually to 
each service... I'm inclined not to bother.  If, for example I had an IMAP 
folder into which I drop spam that my mail server should report on my 
behalf -then reporting would become far less of a chore.


Simple matter of coding. That is how I handle ham and spam training. I 
simply

dunk it into ham and spam folders and let a cron job run sa-learn over the
two folders. In this case you'd probably have to code up something that 
takes

the folder apart properly, forwards the mail appropriately, then tosses it.
I haven't done such a thing. But there are perl tools for reading messages
via IMAP that could be used as the core of a new tool.
{^_^} 

Re: Very simple user query...

[Markus Eskola]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
 
Michael Monnerie wrote:

> On Dienstag, 13. September 2005 14:06 Steve [Spamassasin] wrote:
>
>> If, for example I had an IMAP folder into which I drop spam that
>> my mail server should report on my behalf -then reporting would
>> become far less of a chore.
>
>
> I'd be interested in this very much. Currently, I move SPAM into a
> folder, check that manually, and daily I move manually that
> confirmed SPAM into a special IMAP folder. That folder is scanned
> every 10 minutes, and it's content is learned by sa-learn and the
> e-mails deleted afterwards.
>
> Maybe it would be good to report that e-mails to razor, etc. too.
> I'll give it a try. Do you have a script to report from IMAP to SA?
>
>
> BTW: is pyzor good / worth the effort? It's latest release is
> September 7, 2002, for that I thought it wouldn't be used too much
> anymore. Do you get enough hits?
>
> mfg zmi

Just a quick question regarding the reporting... Do you guys report
all spam (including the once that SA allready caught) or only the ones
that got thru the net?

Currently in my setup I have 3-4 diffrent users who move all the spam
that got thru into certain folders eg SPAM under IMAP. These folders
are scanned, emptied and reported once a night thru a script.
If someone has a more effectie way, I'd appreciate a hint in the right
direction.

Sorry for hijacking ths thread a bit... ;)

/markus
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.2 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
 
iD8DBQFDJzNNY+EUV64Bqn0RAvmHAKDHfcy+mJ6BuR9GpbR5z/PsMZesrwCgidB2
gXf7TQ5sh/wEW730yPPFoTM=
=3kbx
-END PGP SIGNATURE-

Re: Very simple user query...

[Michael Monnerie]

On Dienstag, 13. September 2005 17:38 Pedro Sam wrote:
> I haven't been using spamassassin for a while, but last I check,
> "spamassassin -r" will report spam to DCC/pyzor/razor all in one go.

Ah, so it's simple. I already have a script which takes from my SPAM_yes 
folder and reports as SPAM, and from my SPAM_no folder to report a 
false positive (aka HAM).

> Unfortunately, on its own, it doesn't address the user interface issue
> from the perspective of a client remotely accessing mail over
> IMAP/SMTP... 

Here is how I do it:

# If you want to deliver the e-mail after learning back to cyrus:
fetchmail -a -s -n -p IMAP -u $user --folder 'SPAM_no' --auth 'password' 
-m 'bash -c "/usr/bin/tee >(/usr/bin/sa-learn --ham --single 
&>/dev/null)|/usr/bin/spamc|/usr/lib/cyrus-imapd/deliver"' 
imap.host.domain

# This is the way I do it on normal imapd servers. That e-mail is learnt 
and then discarded:
sudo -H -u $user fetchmail -a -s -n -p IMAP --folder 'SPAM_yes' --auth 
'password' -m "bash -c \"tee >$checkfile|sa-learn --spam --single 
&>/dev/null ; cat $headfile $checkfile >>$spamoutput.$user\"" 
imap.host.domain

Around this there's a loop with each user for who filtering is done. I 
guess I should replace the "sa-learn" with "spamassassin -r" to report 
to bayes/dcc/pyzor/razor, or does only sa-learn tell to bayes? Then I 
should do both.

mfg zmi
-- 
// Michael Monnerie, Ing.BSc  ---   it-management Michael Monnerie
// http://zmi.at   Tel: 0660/4156531  Linux 2.6.11
// PGP Key:   "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952  F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879


pgp4DsDImmdm3.pgp
Description: PGP signature

Re: Very simple user query...

[Steve [Spamassasin]]

Pedro Sam wrote:


> I'm familiar with razor-report, for example - but it is a real pain to
> mess about with this command line tool when all my mail is managed
> remotely over IMAP

I haven't been using spamassassin for a while, but last I check,
"spamassassin -r" will report spam to DCC/pyzor/razor all in one go.


(Having just checked the manual) - yes - that does seem much better than 
reporting to all the services I use individually.  Unfortunately, on its 
own, it doesn't address the user interface issue from the perspective of 
a client remotely accessing mail over IMAP/SMTP...

Re: Very simple user query...

[Steve [Spamassasin]]

Michael Monnerie wrote:

Maybe it would be good to report that e-mails to razor, etc. too. I'll 
give it a try. Do you have a script to report from IMAP to SA?
 

I don't... It can't be that hard to do using a polling approach... It 
would be neater if this was triggered by the IMAP server... but I'm not 
aware of such a facility.  I'd love to be proved wrong...


BTW: is pyzor good / worth the effort? It's latest release is 
September 7, 2002, for that I thought it wouldn't be used too much 
anymore. Do you get enough hits?
 

That's a good question... I wasn't aware that the latest release of 
pyzor was so old... but it wouldn't have concerned me if I had... I'd be 
inclined to suspect that what's important about pyzor is server-side.  
Anyway - I've compiled some statistics over the past few months... and 
it seems my installation of pyzor was really useful until sometime 
during July... thereafter no more matches were made... which is curious...


#PYZOR_CHECK   #spams

May   2794   3121
June  4402   4809
July  2713017
August0  3669
Sept  0  1546

This, I guess, might indicate part of the reason why less spam is caught 
today...  On further investigation I found pyzor crashed when run... 
un-merging then re-merging it solved the problem which was probably some 
strange python dependency.


Steve

Re: Very simple user query...

[Pedro Sam]

Steve [Spamassasin] wrote:

> I'm familiar with razor-report, for example - but it is a real pain to
> mess about with this command line tool when all my mail is managed
> remotely over IMAP

I haven't been using spamassassin for a while, but last I check,
"spamassassin -r" will report spam to DCC/pyzor/razor all in one go.

--
The authors know of one compiler that was written using only seven comments,
one of which read "This code is cursed."

page 731, The Dragon Book

Re: Very simple user query...

[Stuart Johnston]

Michael Monnerie wrote:

On Dienstag, 13. September 2005 14:06 Steve [Spamassasin] wrote:

 >
BTW: is pyzor good / worth the effort? It's latest release is September 
7, 2002, for that I thought it wouldn't be used too much anymore. Do 
you get enough hits?


On my system, pyzor gets about half as many hits as DCC but is still one 
of the more productive rules, coming in just after the SURBLs.  However, 
its default score is fairly high and it has a tendancy to FP on certain 
types of messages so you have to be careful.


Here is a meta rule I was working on to reduce pyzor FPs:

meta L_pyzor_fp (PYZOR_CHECK && (MIME_HTML_ONLY || MIME_HTML_MOSTLY) && 
!DIGEST_MULTIPLE)

score L_pyzor_fp -2

Re: Very simple user query...

[Michael Monnerie]

On Dienstag, 13. September 2005 14:06 Steve [Spamassasin] wrote:
> If, for
> example I had an IMAP folder into which I drop spam that my mail
> server should report on my behalf -then reporting would become far
> less of a chore.

I'd be interested in this very much. Currently, I move SPAM into a 
folder, check that manually, and daily I move manually that confirmed 
SPAM into a special IMAP folder. That folder is scanned every 10 
minutes, and it's content is learned by sa-learn and the e-mails 
deleted afterwards.

Maybe it would be good to report that e-mails to razor, etc. too. I'll 
give it a try. Do you have a script to report from IMAP to SA?

BTW: is pyzor good / worth the effort? It's latest release is September 
7, 2002, for that I thought it wouldn't be used too much anymore. Do 
you get enough hits?

mfg zmi
-- 
// Michael Monnerie, Ing.BSc  ---   it-management Michael Monnerie
// http://zmi.at   Tel: 0660/4156531  Linux 2.6.11
// PGP Key:   "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952  F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879


pgpk3eaUTFeGs.pgp
Description: PGP signature

Re: Very simple user query...

[Steve [Spamassasin]]

jdow wrote:


You do not say which version of spamassassin you are using. If it is not
3.04 an upgrade might help.


It's 3.04 - the latest stable build that's made it into "Gentoo Portage"


   * Is there somewhere where I can report spams which aren't caught by
 the default configuration in order to feed-back into future
 improvements?


There are places to report them manually.


I'm familiar with razor-report, for example - but it is a real pain to 
mess about with this command line tool when all my mail is managed 
remotely over IMAP



I have a strong personal bias against automating anything related to
spam REPORTING. Please examine the downsides of automatic reporting
before proceeding.


I absolutely do not want to report automatically - in the sense that I 
am adamant that I want human intervention before reporting.  Conversely 
- given the task of establishing a remote shell; finding the correct 
email in maildir - and verifying it is indeed the mail I determined was 
a spam in my email client - followed by manually reporting it 
individually to each service... I'm inclined not to bother.  If, for 
example I had an IMAP folder into which I drop spam that my mail server 
should report on my behalf -then reporting would become far less of a chore.


Steve

Re: Very simple user query...

[Fred]

You have a permissions problem, plus you are running duplicate rules..
Remove the tripwire.cf file as you are using a newer version called
99_FVGT_Tripwire.cf
That file was updated months ago with a new name, now it's called
88_FVGT_Tripwire.cf I'm not sure why we changed that but we had good
reasons...

But check your debug output, it says permission denied while trying to read
a number of your add-on rules.. this might be part of the reason you are not
getting results like before...

Frederic Tarasevicius


Steve [Spamassasin] wrote:
>> debug: using "/etc/mail/spamassassin" for site rules dir
>> debug: config: read file /etc/mail/spamassassin/70_sare_adult.cf
>> cannot open "/etc/mail/spamassassin/70_sare_bayes_poison_nxm.cf":
>> Permission denied
>> cannot open "/etc/mail/spamassassin/70_sare_genlsubj.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_genlsubj0.cf":
>> Permission denied
>> debug: config: read file /etc/mail/spamassassin/70_sare_genlsubj1.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_genlsubj2.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_genlsubj3.cf
>> cannot open "/etc/mail/spamassassin/70_sare_genlsubj_eng.cf":
>> Permission denied
>> cannot open "/etc/mail/spamassassin/70_sare_header.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_header0.cf": Permission
>> denied
>> debug: config: read file /etc/mail/spamassassin/70_sare_header1.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_header2.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_header3.cf
>> cannot open "/etc/mail/spamassassin/70_sare_header_eng.cf":
>> Permission denied
>> cannot open "/etc/mail/spamassassin/70_sare_highrisk.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_html0.cf": Permission
>> denied
>> debug: config: read file /etc/mail/spamassassin/70_sare_html1.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_html2.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_html3.cf
>> debug: config: read file /etc/mail/spamassassin/70_sare_html4.cf
>> cannot open "/etc/mail/spamassassin/70_sare_html_eng.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_oem.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_random.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_ratware.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_specific.cf": Permission
>> denied
>> cannot open "/etc/mail/spamassassin/70_sare_spoof.cf": Permission
>> denied
>> debug: config: read file /etc/mail/spamassassin/70_sare_unsub.cf
>> cannot open "/etc/mail/spamassassin/70_sare_uri.cf": Permission
>> denied
>> debug: config: read file
>> /etc/mail/spamassassin/72_sare_bml_post25x.cf
>> cannot open "/etc/mail/spamassassin/72_sare_redirect_post3.0.0.cf":
>> Permission denied
>> debug: config: read file /etc/mail/spamassassin/99_FVGT_Tripwire.cf
>> debug: config: read file
>> /etc/mail/spamassassin/99_sare_fraud_post25x.cf
>> debug: config: read file /etc/mail/spamassassin/antidrug.cf
>> debug: config: read file
>> /etc/mail/spamassassin/bogus-virus-warnings.cf
>> cannot open "/etc/mail/spamassassin/evilnumbers.cf": Permission
>> denied
>> debug: config: read file /etc/mail/spamassassin/local.cf
>> debug: config: read file /etc/mail/spamassassin/random.cf
>> debug: config: read file /etc/mail/spamassassin/random.current.cf
>> cannot open "/etc/mail/spamassassin/tripwire.cf": Permission denied

Re: Very simple user query...

[jdow]

From: "Steve [Spamassasin]" <[EMAIL PROTECTED]>

I'm using spamassassin (Razor, Pyzor, DCC) and procmail to filter all my 
mail on my (Gentoo) linux-server, to which I connect from a number of 
Windows (XP/2000) machines using Mozilla Thunderbird to access my 
(dovecot) IMAP folders on the linux server.  I configured spamassassin 
to use "Rulesdujour" and to regularly update those rules - and I was 
very happy... at least 99.99% of spam was correctly marked with only one 
incident of false positives (for which spamassasin wasn't entirely to 
blame.) in several months.


You do not say which version of spamassassin you are using. If it is not
3.04 an upgrade might help.

Lately I've been less lucky - only ~99% of my spam is marked as such... 
which sounds good but the remaining 1% gives me up-to a dozen bogus 
messages each day... which is frustrating.  To the naked eye the missed 
spam is obviously spam - but typically the only significant rule it 
triggers is the Bayesian rule...  As I've stuck to the default settings 
this alone is insufficient to identify a mail as spam.


So far Bayes 99 triggers ONLY on spam here. I use a per user Bayes. I do
not use autolearn (or autowhitelist) and I basically train only with
caught spam that is not up to Bayes 99 or with spam that escaped. It has
taken some time; but, I've made it to one in one thousand spams or less
escaping detection with about the same level of false alarms.


I'm left with several questions...

   * Is there somewhere where I can report spams which aren't caught by
 the default configuration in order to feed-back into future
 improvements?


There are places to report them manually.


   * Is there an easy way to report spam explicitly to the checksum
 services (Razor/Pyzor/DCC)?


I have a strong personal bias against automating anything related to
spam REPORTING. Please examine the downsides of automatic reporting
before proceeding.

{^_^}

Re: Very simple user query...

[Steve [Spamassasin]]

Martin Hepworth wrote:


Well if this worked. we could make sure we hit the spammers really hard 
;-)
 

While I see eliminating spammers as being one of the better 
justifications for environmental warfare, it isn't sufficiently reliable 
to get my vote.



of course those unfortunates who also live in Baton Raton (or wherever Ralski 
and his co-horts are hiding this week) would be in trouble for harboring these 
people as well ;-(
 

To a large extent (I'm sad to say) I believe that spam is the fault of 
the IT industry who have utterly failed to provide a usable PKI for the 
masses.  If ISPs required to register a certificate for every user's 
email address (at minimal cost - just like is now the case for domain 
names) then spam could become a thing of the past pretty quickly; and 
all email could be sent securely into the bargain.  Well - I can dream 
too - can't I?

RE: Very simple user query...

[Martin Hepworth]

Steve

Well if this worked.

http://www.sciam.com/article.cfm?articleID=000593AE-704B-1151-B57F83414B7F00
00

we could make sure we hit the spammers really hard ;-)

of course those unfortunates who also live in Baton Raton (or wherever
Ralski and his co-horts are hiding this week) would be in trouble for
harboring these people as well ;-(

--
Martin Hepworth 
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300

-Original Message-
From: Steve [Spamassasin] [mailto:[EMAIL PROTECTED] 
Sent: 12 September 2005 17:08
To: users@spamassassin.apache.org
Cc: Martin Hepworth
Subject: Re: Very simple user query...

Martin Hepworth wrote:

>Steve
>
>OK looks like these are both uk.geocities.com abuse spam.
>
>If you look at the archive you'll find some extra rulesets for these little
>blighters (and their variants).
>  
>
Genius answer! For some reason it had completely escaped my notice that 
all of the spams missed by SA over the past month had a uk.geocities.com 
address!  I've opted for a score of 4 for any mail mentioning a 
uk.geocities.com URL - which is hopefully good enough to avoid this kind 
of problem without too great a risk of loosing a mail that happens to 
reference a homepage on uk.geocites.com in an innocent way.

What still surprises me is that DCC/Razor/Pyzor don't pick these up... 
I'd still like to know what would be the easiest way to report these 
spams in order that in future they might be caught without falling back 
on a vicious static check for any mail referencing a URL at a free provider.

Thanks,
Steve



**

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   

**

Re: Very simple user query...

[Steve [Spamassasin]]

Martin Hepworth wrote:


Steve

OK looks like these are both uk.geocities.com abuse spam.

If you look at the archive you'll find some extra rulesets for these little
blighters (and their variants).
 

Genius answer! For some reason it had completely escaped my notice that 
all of the spams missed by SA over the past month had a uk.geocities.com 
address!  I've opted for a score of 4 for any mail mentioning a 
uk.geocities.com URL - which is hopefully good enough to avoid this kind 
of problem without too great a risk of loosing a mail that happens to 
reference a homepage on uk.geocites.com in an innocent way.


What still surprises me is that DCC/Razor/Pyzor don't pick these up... 
I'd still like to know what would be the easiest way to report these 
spams in order that in future they might be caught without falling back 
on a vicious static check for any mail referencing a URL at a free provider.


Thanks,
Steve

RE: Very simple user query...

[Martin Hepworth]

Steve

OK looks like these are both uk.geocities.com abuse spam.

If you look at the archive you'll find some extra rulesets for these little
blighters (and their variants).

--
Martin Hepworth 
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300

-Original Message-
From: Steve [Spamassasin] [mailto:[EMAIL PROTECTED] 
Sent: 12 September 2005 14:22
To: Martin Hepworth
Cc: users@spamassassin.apache.org
Subject: Re: Very simple user query...

Martin Hepworth wrote:

>Steve
>
>Ok looks good. If you can drop an example of a spam that 'gets through' to
a web page somewhere, I can run it over my system and see what happens.
>
>I've got loads of extra rules (most of rulesemporium.com etc etc so we'll
>see what hits...
>  
>
I suspect that the rulesemporium rules are what I refer to as Gentoo's 
"rulesdujour" - though I can't be sure that my automated script picks 
the same rules as you have.

I've attached a zip file containing two spams (sensitive details removed 
with '#' characters... this shouldn't confuse spamassassin) These two 
spams are typical of what's annoying me... Both these examples have 
DATE_IN_PAST_12_24, but this is not the case for all of what is slipping 
past.



**

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   

**

Re: Very simple user query...

[Steve [Spamassasin]]

Martin Hepworth wrote:


Steve

Ok looks good. If you can drop an example of a spam that 'gets through' to a
web page somewhere, I can run it over my system and see what happens.

I've got loads of extra rules (most of rulesemporium.com etc etc so we'll
see what hits...



I should have read your suggestion more carefully - I tried mailing a 
zip file as an attachment - which seems to have been eaten.


   http://www.shic.dynalias.net/spam.zip

Contains two spams...  The eaten message would have said:

--
I suspect that the rulesemporium rules are what I refer to as Gentoo's 
"rulesdujour" - though I can't be sure that my automated script picks 
the same rules as you have.


I've attached a zip file containing two spams (sensitive details removed 
with '#' characters... this shouldn't confuse spamassassin) These two 
spams are typical of what's annoying me... Both these examples have 
DATE_IN_PAST_12_24, but this is not the case for all of what is slipping 
past.

--

RE: Very simple user query...

[Martin Hepworth]

Steve

Ok looks good. If you can drop an example of a spam that 'gets through' to a
web page somewhere, I can run it over my system and see what happens.

I've got loads of extra rules (most of rulesemporium.com etc etc so we'll
see what hits...

--
Martin Hepworth 
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300
-Original Message-
From: Steve [Spamassasin] [mailto:[EMAIL PROTECTED] 
Sent: 12 September 2005 11:43
To: Martin Hepworth; users@spamassassin.apache.org
Subject: Re: Very simple user query...

Martin Hepworth wrote:

>Steve
>
>OK - what do you get for "spamassassin -D --lint" ??
>  
>
Output attached: sdlint.txt...

>This will give you the list of tests etc its triggering along with things
>that might be causing ptoblems. The URI-RBLs are enabled by default in most
config's, but Gentoo might have removed this from the init.pre (as it is in
the RH rpms) which is a right PITA.
>
>In /etc/mail/spamassassin there should be a init.pre file and the following
>line should be enabled to make the URI-RBL's work..
>
>loadplugin Mail::SpamAssassin::Plugin::URIDNSBL
>
>if it doesn't exist or has a # in the front then that will not help at all.
>
I've got that line... and I can confirm that some RBLs do work - for 
example - a spam was classified today with these matches:

> 0.5 SARE_MSGID_ADDED   Message ID added by later system
> 1.7 MSGID_FROM_MTA_ID  Message-Id for external message added locally
> 0.1 RAZOR2_CF_RANGE_51_100 BODY: Razor2 gives confidence level above 50%
>[cf: 100]
> 3.5 BAYES_99   BODY: Bayesian spam probability is 99 to 100%
>[score: 1.]
> 1.5 RAZOR2_CHECK   Listed in Razor2 (http://razor.sf.net/)
> 2.2 DCC_CHECK  Listed in DCC
(http://rhyolite.com/anti-spam/dcc/)
> 2.0 RCVD_IN_SORBS_DUL  RBL: SORBS: sent directly from dynamic IP
address
>[213.106.39.160 listed in dnsbl.sorbs.net]
> 1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
>  [Blocked - see
<http://www.spamcop.net/bl.shtml?213.106.39.160>]
> 3.1 RCVD_IN_XBLRBL: Received via a relay in Spamhaus XBL
>[213.106.39.160 listed in sbl-xbl.spamhaus.org]
> 1.6 DNS_FROM_RFC_POST  RBL: Envelope sender in
postmaster.rfc-ignorant.org
> 0.1 RCVD_IN_NJABL_DUL  RBL: NJABL: dialup sender did non-local SMTP
>[213.106.39.160 listed in combined.njabl.org]
> 1.0 URIBL_SBL  Contains an URL listed in the SBL blocklist
>[URIs: e4v.net]
> 0.4 URIBL_AB_SURBL Contains an URL listed in the AB SURBL
blocklist
>[URIs: e4v.net]
> 2.5 URIBL_JP_SURBL Contains an URL listed in the JP SURBL
blocklist
>[URIs: e4v.net]
> 1.5 URIBL_WS_SURBL Contains an URL listed in the WS SURBL
blocklist
>[URIs: e4v.net]
> 3.2 URIBL_OB_SURBL Contains an URL listed in the OB SURBL
blocklist
>[URIs: e4v.net]
> 4.3 URIBL_SC_SURBL Contains an URL listed in the SC SURBL
blocklist
>[URIs: e4v.net]
> 0.1 DIGEST_MULTIPLEMessage hits more than one network digest check
> 1.7 SARE_SPEC_ROLEXRolex watch spam
> 2.3 SARE_SPEC_ROLEX_REPRolex Replic
>
>



**

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   

**

Re: Very simple user query...

[Steve [Spamassasin]]

Martin Hepworth wrote:


Steve

OK - what do you get for "spamassassin -D --lint" ??
 


Output attached: sdlint.txt...


This will give you the list of tests etc its triggering along with things
that might be causing ptoblems. The URI-RBLs are enabled by default in most 
config's, but Gentoo might have removed this from the init.pre (as it is in the 
RH rpms) which is a right PITA.

In /etc/mail/spamassassin there should be a init.pre file and the following
line should be enabled to make the URI-RBL's work..

loadplugin Mail::SpamAssassin::Plugin::URIDNSBL

if it doesn't exist or has a # in the front then that will not help at all.

I've got that line... and I can confirm that some RBLs do work - for 
example - a spam was classified today with these matches:



0.5 SARE_MSGID_ADDED   Message ID added by later system
1.7 MSGID_FROM_MTA_ID  Message-Id for external message added locally
0.1 RAZOR2_CF_RANGE_51_100 BODY: Razor2 gives confidence level above 50%
   [cf: 100]
3.5 BAYES_99   BODY: Bayesian spam probability is 99 to 100%
   [score: 1.]
1.5 RAZOR2_CHECK   Listed in Razor2 (http://razor.sf.net/)
2.2 DCC_CHECK  Listed in DCC (http://rhyolite.com/anti-spam/dcc/)
2.0 RCVD_IN_SORBS_DUL  RBL: SORBS: sent directly from dynamic IP address
   [213.106.39.160 listed in dnsbl.sorbs.net]
1.2 RCVD_IN_BL_SPAMCOP_NET RBL: Received via a relay in bl.spamcop.net
 [Blocked - see ]
3.1 RCVD_IN_XBLRBL: Received via a relay in Spamhaus XBL
   [213.106.39.160 listed in sbl-xbl.spamhaus.org]
1.6 DNS_FROM_RFC_POST  RBL: Envelope sender in postmaster.rfc-ignorant.org
0.1 RCVD_IN_NJABL_DUL  RBL: NJABL: dialup sender did non-local SMTP
   [213.106.39.160 listed in combined.njabl.org]
1.0 URIBL_SBL  Contains an URL listed in the SBL blocklist
   [URIs: e4v.net]
0.4 URIBL_AB_SURBL Contains an URL listed in the AB SURBL blocklist
   [URIs: e4v.net]
2.5 URIBL_JP_SURBL Contains an URL listed in the JP SURBL blocklist
   [URIs: e4v.net]
1.5 URIBL_WS_SURBL Contains an URL listed in the WS SURBL blocklist
   [URIs: e4v.net]
3.2 URIBL_OB_SURBL Contains an URL listed in the OB SURBL blocklist
   [URIs: e4v.net]
4.3 URIBL_SC_SURBL Contains an URL listed in the SC SURBL blocklist
   [URIs: e4v.net]
0.1 DIGEST_MULTIPLEMessage hits more than one network digest check
1.7 SARE_SPEC_ROLEXRolex watch spam
2.3 SARE_SPEC_ROLEX_REPRolex Replic




debug: SpamAssassin version 3.0.4
debug: Score set 0 chosen.
debug: running in taint mode? no
debug: diag: module not installed: DBI ('require' failed)
debug: diag: module installed: DB_File, version 1.811
debug: diag: module installed: Digest::SHA1, version 2.10
debug: diag: module installed: IO::Socket::UNIX, version 1.21
debug: diag: module installed: MIME::Base64, version 3.05
debug: diag: module installed: Net::DNS, version 0.49
debug: diag: module installed: Net::LDAP, version 0.33
debug: diag: module installed: Razor2::Client::Agent, version 2.77
debug: diag: module installed: Storable, version 2.13
debug: diag: module installed: URI, version 1.35
debug: ignore: using a test message to lint rules
debug: using "/etc/mail/spamassassin/init.pre" for site rules init.pre
debug: config: read file /etc/mail/spamassassin/init.pre
debug: using "/usr/share/spamassassin" for default rules dir
debug: config: read file /usr/share/spamassassin/10_misc.cf
debug: config: read file /usr/share/spamassassin/11_gentoo.cf
debug: config: read file /usr/share/spamassassin/20_anti_ratware.cf
debug: config: read file /usr/share/spamassassin/20_body_tests.cf
debug: config: read file /usr/share/spamassassin/20_compensate.cf
debug: config: read file /usr/share/spamassassin/20_dnsbl_tests.cf
debug: config: read file /usr/share/spamassassin/20_drugs.cf
debug: config: read file /usr/share/spamassassin/20_fake_helo_tests.cf
debug: config: read file /usr/share/spamassassin/20_head_tests.cf
debug: config: read file /usr/share/spamassassin/20_html_tests.cf
debug: config: read file /usr/share/spamassassin/20_meta_tests.cf
debug: config: read file /usr/share/spamassassin/20_phrases.cf
debug: config: read file /usr/share/spamassassin/20_porn.cf
debug: config: read file /usr/share/spamassassin/20_ratware.cf
debug: config: read file /usr/share/spamassassin/20_uri_tests.cf
debug: config: read file /usr/share/spamassassin/23_bayes.cf
debug: config: read file /usr/share/spamassassin/25_body_tests_es.cf
debug: config: read file /usr/share/spamassassin/25_hashcash.cf
debug: config: read file /usr/share/spamassassin/25_spf.cf
debug: config: read file /usr/share/spamassassin/25_uribl.cf

RE: Very simple user query...

[Martin Hepworth]

Steve

What version of SA and what URI-RBL's are you using??

--
Martin Hepworth 
Snr Systems Administrator
Solid State Logic
Tel: +44 (0)1865 842300

-Original Message-
From: Steve [Spamassasin] [mailto:[EMAIL PROTECTED] 
Sent: 12 September 2005 10:27
To: users@spamassassin.apache.org
Subject: Very simple user query...

I'm using spamassassin (Razor, Pyzor, DCC) and procmail to filter all my 
mail on my (Gentoo) linux-server, to which I connect from a number of 
Windows (XP/2000) machines using Mozilla Thunderbird to access my 
(dovecot) IMAP folders on the linux server.  I configured spamassassin 
to use "Rulesdujour" and to regularly update those rules - and I was 
very happy... at least 99.99% of spam was correctly marked with only one 
incident of false positives (for which spamassasin wasn't entirely to 
blame.) in several months.

Lately I've been less lucky - only ~99% of my spam is marked as such... 
which sounds good but the remaining 1% gives me up-to a dozen bogus 
messages each day... which is frustrating.  To the naked eye the missed 
spam is obviously spam - but typically the only significant rule it 
triggers is the Bayesian rule...  As I've stuck to the default settings 
this alone is insufficient to identify a mail as spam.

I'm left with several questions...

* Is there somewhere where I can report spams which aren't caught by
  the default configuration in order to feed-back into future
  improvements?
* Is there an easy way to report spam explicitly to the checksum
  services (Razor/Pyzor/DCC)?

Any other suggestions are welcome...

Steve



**

This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote confirms that this email message has been swept
for the presence of computer viruses and is believed to be clean.   

**

Re: Very simple user query...

[Michael Monnerie]

On Montag, 12. September 2005 11:27 Steve [Spamassasin] wrote:
> Lately I've been less lucky - only ~99% of my spam is marked as
> such...

Same for me: Getting some russian SPAM, which even is only sometimes 
recognised by bayes. Could it be problems with cyrillic?

But I even get english SPAM that doesn't trigger. Even worse, it sayes 
"bayes_00" and gives -2.599 points, effectively marking it as HAM...

mfg zmi
-- 
// Michael Monnerie, Ing.BSc  ---   it-management Michael Monnerie
// http://zmi.at   Tel: 0660/4156531  Linux 2.6.11
// PGP Key:   "lynx -source http://zmi.at/zmi2.asc | gpg --import"
// Fingerprint: EB93 ED8A 1DCD BB6C F952  F7F4 3911 B933 7054 5879
// Keyserver: www.keyserver.net Key-ID: 0x70545879


pgprufROm274c.pgp
Description: PGP signature

Very simple user query...

[Steve [Spamassasin]]

I'm using spamassassin (Razor, Pyzor, DCC) and procmail to filter all my 
mail on my (Gentoo) linux-server, to which I connect from a number of 
Windows (XP/2000) machines using Mozilla Thunderbird to access my 
(dovecot) IMAP folders on the linux server.  I configured spamassassin 
to use "Rulesdujour" and to regularly update those rules - and I was 
very happy... at least 99.99% of spam was correctly marked with only one 
incident of false positives (for which spamassasin wasn't entirely to 
blame.) in several months.


Lately I've been less lucky - only ~99% of my spam is marked as such... 
which sounds good but the remaining 1% gives me up-to a dozen bogus 
messages each day... which is frustrating.  To the naked eye the missed 
spam is obviously spam - but typically the only significant rule it 
triggers is the Bayesian rule...  As I've stuck to the default settings 
this alone is insufficient to identify a mail as spam.


I'm left with several questions...

   * Is there somewhere where I can report spams which aren't caught by
 the default configuration in order to feed-back into future
 improvements?
   * Is there an easy way to report spam explicitly to the checksum
 services (Razor/Pyzor/DCC)?

Any other suggestions are welcome...

Steve