OT: MailFrontier
Is anyone here familiar with MailFrontier? I got a message from them on Tuesday claiming they'd seen a lot of spam from our mail server's IP address, and that they have "marked this IP in [their] central database to protect [their] customers." Following it was a list of hash values and report times which, unfortunately, are completely useless when it comes to checking our logs to see if the mail even came from us, never mind figuring out whether we've got a rogue user sending spam, or a backup MX or forwarding customer receiving spam, or bogus reports. Meanwhile, we've been getting complaints about spam which, on analysis, clearly contains forged Received headers. They have our IP but the wrong HELO, and no or wrong reverse DNS...and of course they don't show up in our logs. So we know spammers are out there forging our IP address. (Why ours? I have no idea. Probably the same reason they like forging our domain name and sending us 90,000 bounces a day.) Anyway, since I can't answer the "charges" witout additional info (which they were unwilling or unable to give me) I asked them some follow-up questions on Wednesday morning, but they haven't replied. These boiled down to: Which IP addresses do they extract from a message? Do they block by IP or just use it as part of a more complex system? If they block mail, to they discard it silently or reject it in SMTP? Unfortunately, I suspect they may have silently discarded my questions. Does anyone here know more about them, or have any suggestions on what to do next? -- Kelson Vibber SpeedGate Communications
RE: cannot write to /root/.spamassassin/bayes_journal, Bayes db update ignored: Permission denied
Ok, now I'm noticing this Creating default_prefs [/root/.spamassassin/user_prefs] Creating default_prefs [/root/.spamassassin/user_prefs] Is there a file path I can set so that the new working directory is my new .spamassasin directory I created? I specifically set the bayes and the autowhitelist paths, I'm assuming there's a general path statement I can set also, although I didn't see it in the manuals.
RE: cannot write to /root/.spamassassin/bayes_journal, Bayes db update ignored: Permission denied
> > First, I assume you're using a bayes_path statement to force the bayes DB > for all users to be in roots homedir. Yep! > > If so, DO NOT proceed.. > In order for your bayes DB to be wide open, ALL users must have r_x access > to /root... that's a bad thing that you don't want to give them. Done. I just opened up the directory rather than the files within it. They still have the same permissions. > move it to someplace in /var, /etc, /usr/share, or some other directory > normal users can safely have read access to the directory. Changed to a different location as you suggested. > Also, be sure to set SA's bayes_file_mode to 777 in your local.cf, > otherwise SA will just change the permisisons every time it updates the > file. Did that too. Now in the log I see it looks much better. debug: bayes token 'H*c:NHxtPHrt' => 0.152853685441601 debug: bayes: score = 4.01130517690973e-10 debug: bayes: 21457 untie-ing debug: bayes: 21457 untie-ing db_toks debug: bayes: 21457 untie-ing db_seen debug: madiff: left: 0, orig: 11, max-difference: 0.00% The strange thing is I temporarily had opened up /root/.spamassassin to be wide open but still got the same results. So I can only think it must have been the file_mode setting which made a fundamental difference. Anyway, looks much better now thanks! Chris
Re: OT - MAPS
At 01:40 PM 1/28/2005, Ade Fewings wrote: We don't pay for it directly, being ac.uk - Janet pays for it - we use MAPS as one of our Sendmail rejecters. However, in the last ~7 days, in the order we check the lists MAPS RBL rejected: 131983 SpamHaus SBL rejected: 8076 Ordb Relays rejected: 497 Of course, be sure to recognize that these numbers are not a good indicator of relative performance of the lists. Since they are in-order filters, SBL's results are inherently biased by the exclusion of MAPS's hits. In general, unless your first RBL really sucks, it will always have the most hits because it will get sole credit for any hits that are also listed in the other RBLs. Most decent RBLs have a lot of overlap. MAPS still probably performs better than SBL, but not by the margin reflected above.. Also, SBL isn't one of the better performing lists from looking at the STATISTICS-set1.txt in SA 3.0.2.. It hit 8% of spam.. it might be more useful to compare MAPS to better performing free lists like DSBL (67%), XBL (53%), NJABL dialup (47%), or Spamcop (25%).
Re: cannot write to /root/.spamassassin/bayes_journal, Bayes db update ignored: Permission denied
At 04:26 PM 1/28/2005, Chris Harvey wrote: cannot write to /root/.spamassassin/bayes_journal, Bayes db update ignored: Permission denied This is right after all the bayes token statements. It suggests it's a problem, but I don't seem to be able to fix it. My default bayes location is /root/.spamassassin and I've tried making permissions on it wide open (777). The files in it are: First, I assume you're using a bayes_path statement to force the bayes DB for all users to be in roots homedir. If so, DO NOT proceed.. In order for your bayes DB to be wide open, ALL users must have r_x access to /root... that's a bad thing that you don't want to give them. move it to someplace in /var, /etc, /usr/share, or some other directory normal users can safely have read access to the directory. Also, be sure to set SA's bayes_file_mode to 777 in your local.cf, otherwise SA will just change the permisisons every time it updates the file.
Re: autolearn never learns ham
At 02:06 PM 1/28/2005, breena wrote: Thank you Matt =). So most of the heuristics seem to be looking for SPAM. What are the ones that would push a mail towards being HAM (and that are not ignored by autolearn bayes)? So far I have found one: ALL_TRUSTED. A few network tests also qualify: RCVD_IN_BSP_OTHER RCVD_IN_BSP_TRUSTED HABEAS_USER Other than that, SA is just relying on not hitting many spam rules. The default autolearn threshold is slightly positive for this reason... I myself use a slightly negative autolearn threshold, and I have a bunch of custom rules with small (no less than -0.2) negative scores that help place mail into my ham autolearning.. Even so, ham autolearning is quite a bit less frequent than spam autolearning... # grep "autolearn=spam" /var/log/maillog |wc -l 9030 # grep "autolearn=not spam" /var/log/maillog |wc -l 478 (note: I use MailScanner which uses this log format... IRC, normal SA logs as "ham" instead of "not spam")
cannot write to /root/.spamassassin/bayes_journal, Bayes db update ignored: Permission denied
I'm seeing in my maillog file (SA 3.x running in Debug mode) the following: cannot write to /root/.spamassassin/bayes_journal, Bayes db update ignored: Permission denied This is right after all the bayes token statements. It suggests it's a problem, but I don't seem to be able to fix it. My default bayes location is /root/.spamassassin and I've tried making permissions on it wide open (777). The files in it are: -rw--- 1 root root 618496 Jan 28 13:26 auto-whitelist -rw--- 1 root root 1323008 Jan 28 14:05 bayes_seen -rw--- 1 root root 10489856 Jan 28 14:05 bayes_toks -rw-r--r-- 1 root root 1313 Jan 28 13:26 user_prefs And spam assassin is being started with service spamassassin start (via root) So I'm a little lost as to why this error would be coming up. Any suggestions? As I said, I'm not sure this is actually a problem, but it does suggest that the bayes DB's are not being updated. Thanks for any help in advance, Chris --- The most important tool in your toolbox is the question "why?".
Re: HELO_DYNAMIC_IPADDR matches wrongly on hotmail
Tony Finch wrote: Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan 2005 19:41:14 - The order and spacing of the items after the from keyword is wrong. The specification for Received: lines is in RFC 2821. A correctly formatted line would be something like sure Received: from bay22-dav1.bay22.hotmail.com.[64.4.16.181].#30781 (EHLO ...) would be the correct writing according to rfc821. but that wouldn't help parsers nor a person reading the headers:) Anyway, there are a lot of malformed Received headers out there. One can't just drop the messages since Received headers were initially designed for debugging. Moreover, the doc says that HELO_DYNAMIC_IPADDR means that a "relay helo'd using a suspicious hostname", which is clearly not the case here as can be seen from the "(EHLO hotmail.com)". Would it be easy to detect "simple" cases of explicit helo such as this one (i.e. the case when we have "(helo foo)" as first or second comment in the from params) and in the case of "( helo foo)" and "(... helo=foo ...)", and when detected consider that the param of "from" is not the heloname but some hostname as viewed by the gateway that added the header (generally an rdns hostname, but it can be ignored if it's in some special form).
Re: bayes db - export/import
On Fri, 28 Jan 2005 11:48:32 -0800, Justin Mason <[EMAIL PROTECTED]> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > > Rodney Green writes: > > Hello, > > > > I'm setting up a temporary mail server so I can do some work on the > > regular production machine, without interrupting service. > > > > I'd like to copy the bayes db to the temporary mail server so it can > > continue to be used and continue learning. > > > > Will I need to do some special export/import procedure or will I be > > able to just copy the db files into the directory, set permissions and > > be good to go? > > If it's the same architecture, and the same OS release, you can > probably just copy. For safety I'd recommend using sa-learn --backup > and --restore. > Thanks Justin. I'll use sa-learn --backup and --restore. Rod
Re: HELO_DYNAMIC_IPADDR matches wrongly on hotmail
At 01:05 PM 1/28/2005, Tony Finch wrote:
> > > Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO
> > > hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure
> > > Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan
> > > 2005 19:41:14 -
> >
> > F-Secure Anti-Virus for Internet Mail is producing incorrectly-formatted
> > Received: lines.
>
> Erm, Where is that header invalid? As far as I can see it's following
all of
> the rules of RFC 2822.
The order and spacing of the items after the from keyword is wrong. The
specification for Received: lines is in RFC 2821. A correctly formatted
line would be something like
Received: from hotmail.com (bay22-dav1.bay22.hotmail.com [64.4.16.181]:30781)
by mailgateway.sitc.dk ([195.231.241.98]:25)
(F-Secure Anti-Virus for Internet Mail 6.41.149 Release)
with SMTP; Wed, 19 Jan 2005 19:41:14 -
I do see the differences between your example and theirs, however, I don't
see where theirs deviates from the spec.
The RFC 2821 section 4.4 appears to allow the either the Domain and the
Address Literal in the From-domain part. This having:
Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781
by itself as the from clause is legitimate.
The RFC does prohibit the "("Tcp-Info")" field from being based on anything
but the TCP connection, but you are not required to insert it, and
immediately following the from clause is a CFWS, allowing a comment of any
sort to be inserted before the by clause.
Or am I misreading section 4.4?
Stamp = From-domain By-domain Opt-info ";" FWS date-time
From-domain = "FROM" FWS Extended-Domain CFWS
Extended-Domain = Domain /
( Domain FWS "(" TCP-info ")" ) /
( Address-literal FWS "(" TCP-info ")" )
TCP-info = Address-literal / ( Domain FWS Address-literal )
; Information derived by server from TCP connection
; not client EHLO.
Or is it that folding in the middle of a comment is not allowed?
Re: bayes db - export/import
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rodney Green writes: > Hello, > > I'm setting up a temporary mail server so I can do some work on the > regular production machine, without interrupting service. > > I'd like to copy the bayes db to the temporary mail server so it can > continue to be used and continue learning. > > Will I need to do some special export/import procedure or will I be > able to just copy the db files into the directory, set permissions and > be good to go? If it's the same architecture, and the same OS release, you can probably just copy. For safety I'd recommend using sa-learn --backup and --restore. - --j. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Exmh CVS iD8DBQFB+pcQMJF5cimLx9ARAronAJ9R00cpm3kAZoa143nNRLfCU/AV8wCcC5oJ TDOiZ5cFN5j+yk5pzrcRHAc= =hs43 -END PGP SIGNATURE-
Re: autolearn never learns ham
Thank you Matt =). So most of the heuristics seem to be looking for SPAM. What are the ones that would push a mail towards being HAM (and that are not ignored by autolearn bayes)? So far I have found one: ALL_TRUSTED. Thanks! Breena On Wed, 26 Jan 2005 13:41:12 -0500, Matt Kettler wrote > At 05:37 PM 1/25/2005, breena wrote: > >I have been checking my logs and it seems that even when an email comes in > >with a whitelisted address (making its score below the default threshold for > >autolearn ham), it is not learned as ham. According to the logs, nothing is > >ever autolearned as ham. Does autolearn ignore the whitelisted mails? > > Autolearn ignores white and blacklist statements. The autolearner > evaluates based on the score of the message with bayes disabled and > all "noautolearn" rules disabled (ie: white/blacklists, AWL, and > GTUBE). > > It might still autolearn a whitelisted message, but it won't learn > it on the basis of being whitelisted or not. > > >What would have to be true about the email for it to be autolearned as ham? > > Here you go: > > 1) bayes_auto_learn must not be set to 0 > > 2) The pre-bayes/noautolearn corrected score of the message must be > below bayes_auto_learn_threshold_nonspam (defaults to 0.1) > > 3) The existing bayes rules (actually those with the "learn" tflag) > must not total to a score over +1.0 for the message as it is now. > This creates the "don't auto-learn as ham anything you previously > thought to be spam" rule. > > 4) the bayes DB write lock needs to be available (ie: nobody else > can be writing at this time). -- SPAM and Webmail solutions from CompanyV.com
bayes db - export/import
Hello, I'm setting up a temporary mail server so I can do some work on the regular production machine, without interrupting service. I'd like to copy the bayes db to the temporary mail server so it can continue to be used and continue learning. Will I need to do some special export/import procedure or will I be able to just copy the db files into the directory, set permissions and be good to go? Thanks, Rod -- Get Firefox Web Browser at the link below! You won't regret it! http://tinyurl.com/4cqbv
Re: OT - MAPS
Tom Gwilt wrote: Hi, Sorry for the brief off-topic post. Is anyone using MAPS? If so, is it worth the cost? Tom We don't pay for it directly, being ac.uk - Janet pays for it - we use MAPS as one of our Sendmail rejecters. However, in the last ~7 days, in the order we check the lists MAPS RBL rejected: 131983 SpamHaus SBL rejected: 8076 Ordb Relays rejected: 497 Cheers Ade -- ___ Ade Fewings MEng School of Informatics, University of Wales, Bangor, Dean Street, Bangor, Gwynedd. LL57 1UT. UK. [EMAIL PROTECTED] www.informatics.bangor.ac.uk/~ade Tel: +44 (0)1248 382736 Fax: +44 (0)1248 361429 ___ smime.p7s Description: S/MIME Cryptographic Signature
OT - MAPS
Hi, Sorry for the brief off-topic post. Is anyone using MAPS? If so, is it worth the cost? Tom
Re: HELO_DYNAMIC_IPADDR matches wrongly on hotmail
> > > Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO > > > hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure > > > Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan > > > 2005 19:41:14 - > > > > F-Secure Anti-Virus for Internet Mail is producing incorrectly-formatted > > Received: lines. > > Erm, Where is that header invalid? As far as I can see it's following all of > the rules of RFC 2822. The order and spacing of the items after the from keyword is wrong. The specification for Received: lines is in RFC 2821. A correctly formatted line would be something like Received: from hotmail.com (bay22-dav1.bay22.hotmail.com [64.4.16.181]:30781) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan 2005 19:41:14 - Tony. -- f.a.n.finch <[EMAIL PROTECTED]> http://dotat.at/ CULLERCOATS: STRONG WEST OR NORTHWEST WINDS FOR MUCH OF THE TIME, THE WINDS OCCASIONALLY VEERING NORTH IN DIRECTION.
Re: HELO_DYNAMIC_IPADDR matches wrongly on hotmail
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Matt Kettler writes: > At 09:23 AM 1/28/2005, Tony Finch wrote: > > > Hi, it seems that HELO_DYNAMIC_IPADDR fires wrongly on this header: > > > > > > Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO > > > hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure > > > Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan > > > 2005 > > > 19:41:14 - > > > >F-Secure Anti-Virus for Internet Mail is producing incorrectly-formatted > >Received: lines. > > Erm, Where is that header invalid? As far as I can see it's following all > of the rules of RFC 2822. It's not what SpamAssassin expects a line to look like ;) generally the receiving site doesn't give its IP address; a lot of faked headers added by spamware do, though. hence the rule. Ole, could you open a bug in the SpamAssassin bugzilla about this? - --j. -BEGIN PGP SIGNATURE- Version: GnuPG v1.2.5 (GNU/Linux) Comment: Exmh CVS iD8DBQFB+nsNMJF5cimLx9ARAsoBAJ41IsXID6iP2GqFC3N+Mu3bWhYBCgCfXFJz +x8Ym8glOGB0VPzW5qPPp2Y= =vEq/ -END PGP SIGNATURE-
Re: HELO_DYNAMIC_IPADDR matches wrongly on hotmail
At 09:23 AM 1/28/2005, Tony Finch wrote: > Hi, it seems that HELO_DYNAMIC_IPADDR fires wrongly on this header: > > Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO > hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure > Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan 2005 > 19:41:14 - F-Secure Anti-Virus for Internet Mail is producing incorrectly-formatted Received: lines. Erm, Where is that header invalid? As far as I can see it's following all of the rules of RFC 2822.
Re: Alt text getting through
At 10:23 AM 1/28/2005, Ray Anderson wrote: I made a custom rule in local.cf to score the following with 5: describe custom_body_checksCustom Body Checks score custom_body_checks5 rawbody __bc_0 /%RND_ALT/I meta custom_body_checks ( __bc_0 ) But it is not catching that phrase in the inbound e-mail. (below) Can anyone tell me why? Because you ended the __bc_0 with a /I, which isn't a valide modifier. Perhaps you meant lower-case /i? try running spamassassin --lint on your rules.. $ spamassassin --lint Bareword found where operator expected at (no file), rule __bc_0, line 1, near "/%RND_ALT/I" (Missing operator before I?)
Alt text getting through
I made a custom rule in local.cf to score the following with 5: describe custom_body_checksCustom Body Checks score custom_body_checks5 rawbody __bc_0 /%RND_ALT/I meta custom_body_checks ( __bc_0 ) But it is not catching that phrase in the inbound e-mail. (below) Can anyone tell me why? I have tons of other rules that get caught and marked, but this one seems to sneak by. Stuck on spamassassin-2.55-2.1.92 for now on a Mandrake 9.2 machine. Thanks, -=Ray Good flying never killed [an enemy] yet. Major Edward "Mick" Mannock, RAF, WWI, 50-73 Victories Here is a snip of the original e-mail (hope it doesn't get caught) = SNIP X-Spam-Status: No, hits=0.1 required=3.0 tests=HTML_MESSAGE version=2.55 X-Spam-Level: X-Spam-Checker-Version: SpamAssassin 2.55 (1.174.2.19-2003-05-19-exp) A40863880133651 Content-Type: multipart/alternative; boundary="--A95370304846963" A95370304846963 Content-Type: text/plain; Charset = "us-ascii" Content-Transfer-Encoding: 7bit --- cut html crap - %RND_ALT%RND_ALT%RND_ALT --- cut html crap - A95370304846963 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit --- cut html crap - %RND_ALT%RND_ALT%RND_ALT --- cut html crap - A95370304846963-- A40863880133651 Content-Type: image/gif; name="vicodinad.gif" Content-Transfer-Encoding: base64
Re: Scalar modifiers
At 06:54 AM 1/28/2005, Gray, Richard wrote: My concern regard processing time. This is basically going to double the number of rules in the SA files. Is SA's meta rule logic greedy? E.g. by putting the DUL rule first if it fails on this will it check the other aspects of the rule? Are there any other issues that I need to bear in mind? Wait.. First, you need to understand something. Meta rules never execute anything. All the normal rules execute first, then meta rules run. Meta rules examine the results of the rules from their previous execution. Thus, it does not matter which parts of a meta rule do or do not match. The sub rules have all already been executed before the meta rule is evaluated. This means that SA's meta logic is greedy in that all parts must always be evaluated, however, it also means that SA's meta logic never double-executes anything.
Re: HELO_DYNAMIC_IPADDR matches wrongly on hotmail
On Fri, 28 Jan 2005, Ole Nomann Thomsen wrote: > Hi, it seems that HELO_DYNAMIC_IPADDR fires wrongly on this header: > > Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO > hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure > Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan 2005 > 19:41:14 - F-Secure Anti-Virus for Internet Mail is producing incorrectly-formatted Received: lines. Tony. -- f.a.n.finch <[EMAIL PROTECTED]> http://dotat.at/ FORTIES CROMARTY FORTH TYNE DOGGER: NORTH BACKING NORTHWEST 5 OR 6, DECREASING 4 FOR A TIME. SHOWERS. MODERATE OR GOOD.
HELO_DYNAMIC_IPADDR matches wrongly on hotmail
Hi, it seems that HELO_DYNAMIC_IPADDR fires wrongly on this header: Received: from bay22-dav1.bay22.hotmail.com[64.4.16.181]:30781 (EHLO hotmail.com) by mailgateway.sitc.dk ([195.231.241.98]:25) (F-Secure Anti-Virus for Internet Mail 6.41.149 Release) with SMTP; Wed, 19 Jan 2005 19:41:14 - Specifically bay22-dav1.bay22.hotmail.com, wich should be legit(?). As it packs 4.4 spampoints, I find this pretty serious. I've set mine to 0.0, is there any explanation? Spamssassin v3.0.2 - Ole.
Scalar modifiers
Hi all, I'd like to implement within SpamAssassin (2.64) the ability to scale a spam score based on a certain rule (specifically, I want to scale the spam score by 1.5 if its from an IP listed as a DUL) My basic theory is that if I take every rule and build a meta rule from it that includes the original rule, and the DUL rule, then and adds half the spam score on again, this produce what I am after. My concern regard processing time. This is basically going to double the number of rules in the SA files. Is SA's meta rule logic greedy? E.g. by putting the DUL rule first if it fails on this will it check the other aspects of the rule? Are there any other issues that I need to bear in mind? I'd also like to propose this as a feature for future versions. I can think of a number of reasons why one might want to weight the spam score based on certain rules (scalable trust relationships etc). Your feedback is, as always, most appreciated. Richard --- This email from dns has been validated by dnsMSS Managed Email Security and is free from all known viruses. For further information contact [EMAIL PROTECTED]
RE: Regular expression expanding
Loren, Bob, Mike
Awesome explanations! Mike hit the nail on the head for the bit that I was
uncertain about, but the explanations cleared up a lot of extra uncertainty
surrounding the whole thing.
Thanks for your help,
Richard
-Original Message-
From: Matt Kettler [mailto:[EMAIL PROTECTED]
Sent: 28 January 2005 02:51
To: Gray, Richard; users@spamassassin.apache.org
Subject: Re: Regular expression expanding
At 09:23 AM 1/27/2005, Gray, Richard wrote:
>body
>MANGLED_CASH/(?!cash)\b[cǩ\(][_\W]{0,[EMAIL PROTECTED],5}[sz
>5\$][_\W]{0,5}h\b/i My understanding of rule matching was that the
>'(?!cash' bit required an |
>(or) in order to work. Can anyone break down the logic of how SA tests
>this line?
Heh.. I think your used to seeing things like (?:a|b) which is an or operation
with backreferencing disabled.
However, you can also have (?:a) without the | and you can have (a|b).
The deal is that (?: disables the ability to later use backreferencing, which
is the ability to use \1 later in a expression to require a duplicate of a
previous match.
| is just an or.
Put the two together and you have an or without backreferencing. Disabling
backreferencing saves memory if you're not going to use it, so it's commonly
done in SA rules.
The bit used in the MANGLED_CASH rule is a completely different syntax, despite
it's similar appearance. (?!a) is a negative look-ahead assertion.
ie: when evaluating the rest of the regex line, do not match if you match this.
Here it's used to exclude "cash" from being considered a match for the mangled
string.
There's lots of different operation modifiers that start with (?. (?: is much
different than (?! , (?=, or (?http://perlmonks.thepen.com/236866.html
In the context of SA rules, you usually only see (?: and (?!
---
This email from dns has been validated by dnsMSS Managed Email Security and is
free from all known viruses.
For further information contact [EMAIL PROTECTED]
SARE: Subject header rules updated
Just a quick note that the SARE subject header files, 70_sare_genlsubj*.cf, have been updated. Information and links at http://www.rulesemporium.com/rules.htm#genlsubj Bob Menschel
Re: Whitelisting Groups/Lists
On Thursday, January 27, 2005, 9:51:41 PM, Jeff Chan wrote: > As a practical matter an N of 1 seems to > stop most spammers and probably prevents most from even > trying in the first place, which is even better. (But that's with the manual un-moderating, and not auto un-moderating.) Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: Whitelisting Groups/Lists
On Thursday, January 27, 2005, 9:34:09 PM, Daniel Quinlan wrote: > Jeff Chan <[EMAIL PROTECTED]> writes: >> Yahoo Groups has a "moderate new members" setting which leaves new >> members in a moderated state until the owner manually changes it. >> It's a deterrent against spam since initial posts are moderated. >> Works great. > I've been a moderator too many times, that's definitely too much work. Yeah I suppose it depends on the message activity and the number of members. None of my lists are too huge, so this is manageable. I can see how auto-unmoderation after N messages could be a timesaver. But if spammers knew number N, they could just post N legitimate looking messages, then start spamming. With a manual unmoderation, they know someone is checking their postings. As a practical matter an N of 1 seems to stop most spammers and probably prevents most from even trying in the first place, which is even better. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: Whitelisting Groups/Lists
Jeff Chan <[EMAIL PROTECTED]> writes: > Yahoo Groups has a "moderate new members" setting which leaves new > members in a moderated state until the owner manually changes it. > It's a deterrent against spam since initial posts are moderated. > Works great. I've been a moderator too many times, that's definitely too much work. /me calls patent attorney. Daniel -- Daniel Quinlan http://www.pathname.com/~quinlan/
Re: Whitelisting Groups/Lists
On Thursday, January 27, 2005, 8:50:25 PM, Daniel Quinlan wrote: > "Loren Wilton" <[EMAIL PROTECTED]> writes: >> Then again, I belong to a fairly esoteric list that requires a conversation >> with the moderator in able to even be able to join the list. Part of the >> conversation is stating that you Will Not Spam. >> >> About one in 5 new members is a spammer, and gets deleted from the list >> after their first post (or first 3 simultaneous posts) all of which are >> spam. > Wow, sounds like Yahoo! needs a moderator option to "moderate first N > messages" from new posters where N can be set at will by the moderator > (so spammers don't know how long to pose before spamming). > I'm patenting the idea now... > Daniel Yahoo Groups has a "moderate new members" setting which leaves new members in a moderated state until the owner manually changes it. It's a deterrent against spam since initial posts are moderated. Works great. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: Whitelisting Groups/Lists
"Loren Wilton" <[EMAIL PROTECTED]> writes: > Then again, I belong to a fairly esoteric list that requires a conversation > with the moderator in able to even be able to join the list. Part of the > conversation is stating that you Will Not Spam. > > About one in 5 new members is a spammer, and gets deleted from the list > after their first post (or first 3 simultaneous posts) all of which are > spam. Wow, sounds like Yahoo! needs a moderator option to "moderate first N messages" from new posters where N can be set at will by the moderator (so spammers don't know how long to pose before spamming). I'm patenting the idea now... Daniel -- Daniel Quinlan http://www.pathname.com/~quinlan/
Re: Whitelisting Groups/Lists
> One interesting tidbit -- a group I manage used to get hit by Step 6 > style spam pretty regularly. I turned on "first post requires moderator > approval". Interestingly enough, I haven't had to reject any spam. > Apparently just turning on that flag is enough to ward off a lot of > spammers. Then again, I belong to a fairly esoteric list that requires a conversation with the moderator in able to even be able to join the list. Part of the conversation is stating that you Will Not Spam. About one in 5 new members is a spammer, and gets deleted from the list after their first post (or first 3 simultaneous posts) all of which are spam.
Re: Whitelisting Groups/Lists
On Thursday, January 27, 2005, 8:01:46 PM, David Brodbeck wrote: > Kelson wrote: >> 1. You sign up for a group about vintage widgets. >> 2. Spammer sends a message to your vintage widget list. >> 3. You get the spam through a whitelisted, opt-in channel. >> 4. List members & owner get up in arms, flame war ensues over whether >> the list should be closed or kept open, whether Yahoo isn't doing its >> job filtering posts, whether it's Yahoo's business to filter posts, etc. >> 5. Repeat steps 2-4 until the owner decides to limit posting to members. >> 6. Spammer signs up for Yahoo account, signs up for list, posts spam. >> 7. Same as step 4. >> 8. Repeat steps 6-7 until list owner decides to enable some degree of >> moderation. > One interesting tidbit -- a group I manage used to get hit by Step 6 > style spam pretty regularly. I turned on "first post requires moderator > approval". Interestingly enough, I haven't had to reject any spam. > Apparently just turning on that flag is enough to ward off a lot of > spammers. I've done the same thing on all of the Yahoo Groups I run. Spam no longer gets through to the lists. The software also has a convenient "Ban" button on the new message moderation page. Jeff C. -- Jeff Chan mailto:[EMAIL PROTECTED] http://www.surbl.org/
Re: Whitelisting Groups/Lists
Kelson wrote: 1. You sign up for a group about vintage widgets. 2. Spammer sends a message to your vintage widget list. 3. You get the spam through a whitelisted, opt-in channel. 4. List members & owner get up in arms, flame war ensues over whether the list should be closed or kept open, whether Yahoo isn't doing its job filtering posts, whether it's Yahoo's business to filter posts, etc. 5. Repeat steps 2-4 until the owner decides to limit posting to members. 6. Spammer signs up for Yahoo account, signs up for list, posts spam. 7. Same as step 4. 8. Repeat steps 6-7 until list owner decides to enable some degree of moderation. One interesting tidbit -- a group I manage used to get hit by Step 6 style spam pretty regularly. I turned on "first post requires moderator approval". Interestingly enough, I haven't had to reject any spam. Apparently just turning on that flag is enough to ward off a lot of spammers.
Re: Regular expression expanding
At 09:23 AM 1/27/2005, Gray, Richard wrote:
body
MANGLED_CASH/(?!cash)\b[cǩ\(][_\W]{0,[EMAIL PROTECTED],5}[sz5\$][_\W]{0,5}h\b/i
My understanding of rule matching was that the '(?!cash' bit required an |
(or) in order to work. Can anyone break down the logic of how SA tests
this line?
Heh.. I think your used to seeing things like (?:a|b) which is an or
operation with backreferencing disabled.
However, you can also have (?:a) without the | and you can have (a|b).
The deal is that (?: disables the ability to later use backreferencing,
which is the ability to use \1 later in a expression to require a duplicate
of a previous match.
| is just an or.
Put the two together and you have an or without backreferencing. Disabling
backreferencing saves memory if you're not going to use it, so it's
commonly done in SA rules.
The bit used in the MANGLED_CASH rule is a completely different syntax,
despite it's similar appearance. (?!a) is a negative look-ahead assertion.
ie: when evaluating the rest of the regex line, do not match if you match
this. Here it's used to exclude "cash" from being considered a match for
the mangled string.
There's lots of different operation modifiers that start with (?. (?: is
much different than (?! , (?=, or (?
This really is getting into advanced perl regex syntax, but if you really
want to know about them look up:
http://perlmonks.thepen.com/236866.html
In the context of SA rules, you usually only see (?: and (?!
Re: Regular expression expanding
Hello Richard,
Thursday, January 27, 2005, 6:23:53 AM, you wrote:
GR> I'm trying to get my head around regular expression matching.
GR> body MANGLED_CASH
GR> /(?!cash)\b[cǩ\(][_\W]{0,[EMAIL PROTECTED],5}[sz5\$][_\W]{0,5}h\b/i
GR> My understanding of rule matching was that the '(?!cash' bit
GR> required an | (or) in order to work. Can anyone break down the
GR> logic of how SA tests this line?
GR> /(?!cash)
Do NOT match "cash"
GR> \b
What ever does match needs to begin at the beginning of a word. There
must be a beginning of line or non-word character to the left, and a
word character to the right.
GR> [cǩ\(]
First character matched must be a C or some variation thereof
GR> [_\W]{0,5}
Next character(s) matched must be some non-alphanumeric character.
There may or may not be any, and no more than 5.
GR> [EMAIL PROTECTED]
Next letter is an A
GR> [_\W]{0,5}
GR> [sz5\$]
Next letter is an S
GR> [_\W]{0,5}
GR> h
Next letter is an H
GR> \b
That H has to be followed by a non-word character or end of line
GR> /i
Ignore case -- treat CA$H the same as ca$h.
Bob Menschel
Re: how to call procmail for spam delete?
I got a little carried away... Procmail is now moving everything tagged
spam to the spam folder like it is supposed to, but anything that does
match the spam tag is getting bounced...
Kyle Reynolds
972-731-4731
[EMAIL PROTECTED]
[EMAIL PROTECTED]
splace.com To: "jdow" <[EMAIL
PROTECTED]>
cc:
users@spamassassin.apache.org
01/27/2005 08:02 Subject: Re: how to call
procmail for spam delete?
PM
I have it working now. Does this seem to make sense? Does anyone see any
potential performance issues with this?
I changed the master.cf from this:
spamassassin unix - n n - - pipe
user=filter argv=/usr/local/bin/sa-filter.csh -f ${sender} -- ${recipient}
#
==
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
#
==
smtp inet n - n - - smtpd -o
content_filter=spamassassin
pickupfifo n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
rewrite unix - - n - - trivial-rewrite
bounceunix - - n - 0 bounce
defer unix - - n - 0 bounce
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - n - - smtp -o
content_filter=spamassassin
relay unix - - n - - smtp
showq unix n - n - - showq
error unix - - n - - error
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
-
to this:
procmail unix - n n - - pipe
user=filter argv=/usr/local/bin/procmail -pm /etc/procmailrc
#
==
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
#
==
smtp inet n - n - - smtpd -o
content_filter=procmail
pickupfifo n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
rewrite unix - - n - - trivial-rewrite
bounceunix - - n - 0 bounce
defer unix - - n - 0 bounce
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - n - - smtp -o
content_filter=procmail
relay unix - - n - - smtp
showq unix n - n - - showq
error unix - - n - - error
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
-
before, procmail was never getting called, it seemed that postfix was
calling spamassassin and then delivering the messages, but now, pos
Re: Spamassassin conflits - help me please! (SOLVED!)
> I've been seeing a LOT of reports recently of problems caused by the > standard rules being in /etc/spamassassin or /etc/mail/spamassassin, > something that should not happen. I'm wondering if it's old broken distro > packages, or user error... It might be worth adding a few lines to the SA startup code that checks for one or two of the standard rules files in /etc... and output a debugging complaint. Maybe this check would only be made if the -D switch was given. That could maybe help a few people at least realize that they have a problem. Possibly there could even be something at the end of the SA install that would check for this error and output an error message. That should make it really obvious to whoever is doing the install that something is in the wrong place. If it happens for everyone that installs package X, it might become fairly obvious that something is wrong with package X or its instructions. Loren
Re: Regular expression expanding
I'm trying to get my head around regular _expression_ matching.
body MANGLED_CASH
/(?!cash)\b[cǩ\(][_\W]{0,[EMAIL PROTECTED],5}[sz5\$][_\W]{0,5}h\b/i
My
understanding of rule matching was that the '(?!cash' bit required an | (or)
in order to work. Can anyone break down the logic of how SA tests this
line?
Not sure why you think an OR is required.
OTOH, I'm not at all sure why there is a \b there between (?!cash) and the
mangled matching code. That \b either should be inside the parends with
cash, or shoudn't be there at all. Given the overall rule it would be
more efficient to have it inside the parends. There should also be
another \b before the '(?!' part to keep from matching 'cash' inside the
middle of some other word, I suppose.
Then again, I don't really see a reason to have
the \b check there at all. If someone is going to spell cash using
mangled letters, I don't see that you care much if it is a stand-alone
word.
In any case, what the (?!cash) part is saying
is 'the word 'cash' does not appear here', followed by a word break (the \b)
followed by a mangled spelling of cash, followed by another word break.
Which doesn't really work, but the intent was to catch a mangled spelling of
cash, but not a non-mangled spelling.
A better version would probably be
body MANGLED_CASH
/(?!cash)[cǩ\(][_\W]{0,[EMAIL PROTECTED],5}[sz5\$][_\W]{0,5}h/i
Loren
Re: how to call procmail for spam delete?
I have it working now. Does this seem to make sense? Does anyone see any
potential performance issues with this?
I changed the master.cf from this:
spamassassin unix - n n - - pipe
user=filter argv=/usr/local/bin/sa-filter.csh -f ${sender} -- ${recipient}
#
==
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
#
==
smtp inet n - n - - smtpd -o
content_filter=spamassassin
pickupfifo n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
rewrite unix - - n - - trivial-rewrite
bounceunix - - n - 0 bounce
defer unix - - n - 0 bounce
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - n - - smtp -o
content_filter=spamassassin
relay unix - - n - - smtp
showq unix n - n - - showq
error unix - - n - - error
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
-
to this:
procmail unix - n n - - pipe
user=filter argv=/usr/local/bin/procmail -pm /etc/procmailrc
#
==
# service type private unpriv chroot wakeup maxproc command + args
# (yes) (yes) (yes) (never) (100)
#
==
smtp inet n - n - - smtpd -o
content_filter=procmail
pickupfifo n - n 60 1 pickup
cleanup unix n - n - 0 cleanup
qmgr fifo n - n 300 1 qmgr
rewrite unix - - n - - trivial-rewrite
bounceunix - - n - 0 bounce
defer unix - - n - 0 bounce
flush unix n - n 1000? 0 flush
proxymap unix - - n - - proxymap
smtp unix - - n - - smtp -o
content_filter=procmail
relay unix - - n - - smtp
showq unix n - n - - showq
error unix - - n - - error
local unix - n n - - local
virtual unix - n n - - virtual
lmtp unix - - n - - lmtp
-
before, procmail was never getting called, it seemed that postfix was
calling spamassassin and then delivering the messages, but now, postfix
calls procmail, and then procmail calls spamc with this in procmailrc:
-
LOGFILE=/var/mail/procmaillog
VERBOSE=YES
DROPPRIVS=yes
:0fw: spamc.lock
* < 256000
| /usr/local/bin/spamc
:0:
* ^X-Spam-Level: \*\*\*\*
/tmp/spam
# Work around procmail bug: any output on stderr will cause the "F" in
"From"
# to be dropped. This will re-add it.
:0
* ^^rom[ ]
{
LOG="*** Dropped F off From_ header! Fixing up. "
:0 fhw
| sed -e '1s/^/F/'
}
--
and it does work. My concern now is performance. I got this working in
our test environment, which gets only the email I push through...) so I'm
curious to know how this will perform under a full mail load. Anyone have
any predictions based on their own experience? Any changes I should make
to ease the burden on the cpu?
Thanks, and thanks to "jdow" for the patient suggestions.
Kyle Reynolds
972-731-4731
[EMAIL PROTECTED]
Re: Spamassassin Reporting Qn
Title: RE: Spamassassin Reporting Qn Don't have the users FORWARD the mail to the account of the SA box. That will screw things up, especially with Exchange. Instead, make a public folder on the SA box, probably IMAP, and have users COPY or MOVE spam messages into this folder. They can do this with drag-n-drop, or by right-clicking on the message and selecting Move or maybe Copy. Run a cron script on the SA box to learn the stuff in the folder every so often, then empty it out. I believe there is a script on the wiki for managing the SA learning end of this, and a description in more detail on how to set all this up. Loren
RE: SA3.0.2, rewrite and transform spam
At 05:21 PM 1/27/2005, Rakotomandimby (R12y) Mihamina wrote: ( Thu, 27 Jan 2005 17:04:47 -0500 ) Chris Santerre : > If I misunderstood this, I'm sorry. But can you sip the first server from > scanning the messege with SA? Seems the logical solution. Not really. Because SA still have tu run on the first server. I leave it, but the users stay :-) Well I would like SA: - not to alter the body of the messages it found spam - not to check spam-tagged messages (It stil has to check NON-SPAM tagged) I found on google that report_header could be the solution of the first point. But how about the second point? report_header is deprecated.. use report_safe 0 in SA 2.50 and up. As for the second issue, use a procmail rule to bypass calling SA if the "X-Spam-Flag: Yes" header is present. SA itself has no mechanism for "aborting" the scan of a message. If SA is given a message, it gets scanned. Also, if you're having problems where the second server doesn't tag messages the first server does tag, check to see if it's because the DUL lists are no longer hitting. If that's the case, you should manualy declare a "trusted_networks" and "internal_networks" (sa 3.0+) and include the first server. This way, SA will trust the Received: header generated there, and check if a dialup node direct-delivered mail there. Otherwise, SA will assume the first server is the ISP MTA and it's properly relayed.
