SA not receiving fixed FORGED_MUA_MOZILLA update?
I am having problems with false positives for FORGED_MUA_MOZILLA for Yahoo emails. I see this has been already dealt with here and pushed to the 3.4 and trunk branches: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7411 However, even after running sa-update, the file 20_meta_tests.cf still doesn't have the changes on my servers. Has this bugfix not been applied for some reason? 20_meta_tests.cf on my machines is dated May 17 and June 24 for the 3.00xx and 4.00xx dirs under /var/lib/spamassassin - which is after the date the bugfix was pushed (2017-04-22). I run SA 3.4.1 and 4.0.0 on different machines
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
On 9/15/2017 6:11 AM, Sebastian Arcus wrote: I am having problems with false positives for FORGED_MUA_MOZILLA for Yahoo emails. I see this has been already dealt with here and pushed to the 3.4 and trunk branches: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7411 However, even after running sa-update, the file 20_meta_tests.cf still doesn't have the changes on my servers. Has this bugfix not been applied for some reason? 20_meta_tests.cf on my machines is dated May 17 and June 24 for the 3.00xx and 4.00xx dirs under /var/lib/spamassassin - which is after the date the bugfix was pushed (2017-04-22). I run SA 3.4.1 and 4.0.0 on different machines Hi Sebastian, The Rule Promotion and Generation of tarballs with correct scores has been an ongoing issue for the SpamAssassin team. I would suggest you simply copy the rule to your local.cf at this time. Regards, KAM
FORGED_YAHOO_RCVD still causing false positives
I see this has come up again and again. Since FORGED_YAHOO_RCVD seems to work by checking the address of the Yahoo smtp server in the headers against a predefined list of Yahoo servers in SA, and Yahoo seems to add new servers all the time - which causes false positives, is there much point to this check? If not, maybe the default score should be lowered at least to something like 0.2 or 0.3 (I think is at 1.5 at the moment).
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
On 15/09/17 11:41, Kevin A. McGrail wrote: On 9/15/2017 6:11 AM, Sebastian Arcus wrote: I am having problems with false positives for FORGED_MUA_MOZILLA for Yahoo emails. I see this has been already dealt with here and pushed to the 3.4 and trunk branches: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7411 However, even after running sa-update, the file 20_meta_tests.cf still doesn't have the changes on my servers. Has this bugfix not been applied for some reason? 20_meta_tests.cf on my machines is dated May 17 and June 24 for the 3.00xx and 4.00xx dirs under /var/lib/spamassassin - which is after the date the bugfix was pushed (2017-04-22). I run SA 3.4.1 and 4.0.0 on different machines Hi Sebastian, The Rule Promotion and Generation of tarballs with correct scores has been an ongoing issue for the SpamAssassin team. I would suggest you simply copy the rule to your local.cf at this time. Hi Kevin, Thank you for the reply. Does that mean that no new rules have been pushed to SA installations in the past 5 months - or only some rules get pushed through?
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
On 9/15/2017 6:54 AM, Sebastian Arcus wrote: Thank you for the reply. Does that mean that no new rules have been pushed to SA installations in the past 5 months - or only some rules get pushed through? The system has been "down" since March 15 in that everything is working but we are purposefully not changing the DNS entries. We've resurrected it a few times and Dave Jones has done some work to get a new system running but it published incorrect score files. He did some massaging and published a new rule set with the old score file a few months ago. But since them we've been battling the machine randomly hanging. And work to resurrect the old boxes failed. We are looking for some little darn insidious issue not processing rule scores right. Regards, KAM
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
> On 15/09/17 11:41, Kevin A. McGrail wrote: >> On 9/15/2017 6:11 AM, Sebastian Arcus wrote: >>> I am having problems with false positives for FORGED_MUA_MOZILLA for >>> Yahoo emails. I see this has been already dealt with here and pushed >>> to the 3.4 and trunk branches: >>> >>> https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7411 >>> >>> However, even after running sa-update, the file 20_meta_tests.cf still >>> doesn't have the changes on my servers. Has this bugfix not been >>> applied for some reason? >>> >>> 20_meta_tests.cf on my machines is dated May 17 and June 24 for the >>> 3.00xx and 4.00xx dirs under /var/lib/spamassassin - which is after >>> the date the bugfix was pushed (2017-04-22). >>> >>> I run SA 3.4.1 and 4.0.0 on different machines >> >> Hi Sebastian, >> >> The Rule Promotion and Generation of tarballs with correct scores has >> been an ongoing issue for the SpamAssassin team. I would suggest you >> simply copy the rule to your local.cf at this time. > > Hi Kevin, > > Thank you for the reply. Does that mean that no new rules have been > pushed to SA installations in the past 5 months - or only some rules get > pushed through? > Basically there have been no rule changes pushed since march. That is, at some point new rules were pushed, but as they were incorrect, those changes were reverted and the old set (march) was pushed again.
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
> On 9/15/2017 6:54 AM, Sebastian Arcus wrote: >> Thank you for the reply. Does that mean that no new rules have been >> pushed to SA installations in the past 5 months - or only some rules >> get pushed through? > > The system has been "down" since March 15 in that everything is working > but we are purposefully not changing the DNS entries. > > We've resurrected it a few times and Dave Jones has done some work to > get a new system running but it published incorrect score files. He did > some massaging and published a new rule set with the old score file a > few months ago. But since them we've been battling the machine randomly > hanging. And work to resurrect the old boxes failed. > > We are looking for some little darn insidious issue not processing rule > scores right. It sounds a bit like you guys are hitting a wall? Could any help from the community get things going again? If so, what kind of skillset would be useful to tackle this thing? > > Regards, > > KAM > >
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
On 15/09/17 12:21, Kevin A. McGrail wrote: On 9/15/2017 6:54 AM, Sebastian Arcus wrote: Thank you for the reply. Does that mean that no new rules have been pushed to SA installations in the past 5 months - or only some rules get pushed through? The system has been "down" since March 15 in that everything is working but we are purposefully not changing the DNS entries. We've resurrected it a few times and Dave Jones has done some work to get a new system running but it published incorrect score files. He did some massaging and published a new rule set with the old score file a few months ago. But since them we've been battling the machine randomly hanging. And work to resurrect the old boxes failed. We are looking for some little darn insidious issue not processing rule scores right. Thank you for the update Kevin - and for the hard work of everyone involved. Let's hope the rules updates will be operational again soon.
Re: SA not receiving fixed FORGED_MUA_MOZILLA update?
On 9/15/2017 7:43 AM, Merijn van den Kroonenberg wrote: It sounds a bit like you guys are hitting a wall? Could any help from the community get things going again? If so, what kind of skillset would be useful to tackle this thing? Yes, help is always welcome. More hands make light work. We setup a SysAdmins group at the SA project to help manage the SA Infrastructure just because of this issue and because I was the last holder of too much arcania. Off the cuff, the skills needed are: - Ethical SysAdmin agreement since you have access to private emails, etc. donated to the project. - Ubuntu SysAdmin Skills since that's the OS in question. - Redhat/Solaris/etc. skills helpful since old platform was on those platforms. And the applicant has to then be approved by the SA PMC as they need committer access. Right now, we have 2 that passed with a 3rd whose vote is still pending but failed. Out of the 2, the system complexity required more attention than he can give at this time. So it's been me and Dave. Some others have offered to help like Eric Wolleson. I'll see if I can unlogjam that PMC process. Regards, KAM
Re: getting help with SA sysadmin (was: SA not receiving fixed FORGED_MUA_MOZILLA update?)
> On 9/15/2017 7:43 AM, Merijn van den Kroonenberg wrote: >> It sounds a bit like you guys are hitting a wall? >> >> Could any help from the community get things going again? If so, what >> kind >> of skillset would be useful to tackle this thing? > > Yes, help is always welcome. More hands make light work. We setup a > SysAdmins group at the SA project to help manage the SA Infrastructure > just because of this issue and because I was the last holder of too much > arcania. > > Off the cuff, the skills needed are: > > - Ethical SysAdmin agreement since you have access to private emails, > etc. donated to the project. > > - Ubuntu SysAdmin Skills since that's the OS in question. > > - Redhat/Solaris/etc. skills helpful since old platform was on those > platforms. > > And the applicant has to then be approved by the SA PMC as they need > committer access. Right now, we have 2 that passed with a 3rd whose > vote is still pending but failed. Out of the 2, the system complexity > required more attention than he can give at this time. So it's been me > and Dave. So one of the problems with solving this is because getting help requires a complicated enrollment. Maybe there are chunks of work which can be offloaded to the community without requiring full sysadmin enrollment? I can imagine lots of scripts are involved into running the rule generation. Maybe some need reviewing or something? (just throwning thing in the air here) > > Some others have offered to help like Eric Wolleson. I'll see if I can > unlogjam that PMC process. > > Regards, > > KAM > >
Re: getting help with SA sysadmin
On 9/15/2017 8:22 AM, Merijn van den Kroonenberg wrote: So one of the problems with solving this is because getting help requires a complicated enrollment. Maybe there are chunks of work which can be offloaded to the community without requiring full sysadmin enrollment? I can imagine lots of scripts are involved into running the rule generation. Maybe some need reviewing or something? (just throwning thing in the air here) I could be forest for the trees but I haven't seen the ability for anyone to make progress with minimal access. Dave, tell me if you agree, please? Regards, KAM
Re: FORGED_YAHOO_RCVD still causing false positives
On Fri, 15 Sep 2017 11:50:25 +0100 Sebastian Arcus wrote: > I see this has come up again and again. Since FORGED_YAHOO_RCVD seems > to work by checking the address of the Yahoo smtp server in the > headers against a predefined list of Yahoo servers in SA, and Yahoo > seems to add new servers all the time - which causes false positives, It's based on Yahoo received header formats, but they are liable to change. > is there much point to this check? The rule was created and scored when spoofing Yahoo was very common, but it isn't any more. I don't think it's worth keeping as it is - high maintenance and error prone.
Re: In anyone else getting 325KB spams from cont...@cron-job.org?
On Fri, 15 Sep 2017 00:39:35 +0100 Sebastian Arcus wrote: > I had to add on my systems a while ago an > /etc/mail/spamassassin/spamc.conf containing: > > -s 200 > > to increase the maximum size of emails passed to SA. It seems some > spammers have cottoned onto the fact that 256KB is still hardwired > somewhere in SA, and started sending spam just above that threshold > to bypass the filter. The default is 500kB for spamc, 256kB is a default for sa-learn.
Re: FORGED_YAHOO_RCVD still causing false positives
On 9/15/2017 8:26 AM, RW wrote: The rule was created and scored when spoofing Yahoo was very common, but it isn't any more. I don't think it's worth keeping as it is - high maintenance and error prone. Agreed. Score FORGED_YAHOO_RCVD to zero locally and will get a bug open to deprecate it. Regards, KAM
Re: getting help with SA sysadmin
On 09/15/2017 07:26 AM, Kevin A. McGrail wrote: On 9/15/2017 8:22 AM, Merijn van den Kroonenberg wrote: So one of the problems with solving this is because getting help requires a complicated enrollment. Maybe there are chunks of work which can be offloaded to the community without requiring full sysadmin enrollment? I can imagine lots of scripts are involved into running the rule generation. Maybe some need reviewing or something? (just throwning thing in the air here) I could be forest for the trees but I haven't seen the ability for anyone to make progress with minimal access. Dave, tell me if you agree, please? Regards, KAM I hit a wall with my troubleshooting a couple of months ago. Normally I am able to solve very detailed issues like this without much problem. Here are the issues: 1. These are very old scripts written in disjointed pieces over a long period of time by different people with different styles. There is very little consistency and no proper modularity by function. 2. It takes a long time, 40+ minutes, to do a single run making troubleshooting very slow and difficult. The scripts are mostly using very verbose bash "set -x" output for logging which is hard to dig through the massive output and correlate with problems. They need to have proper debug logging after being rewritten in modular parts. 3. I have narrowed down the problem to the general area of a perl Makefile which builds a custom garescorer.c file which does some statistical analysis to determine the best score for rules in the 72_scores.cf. These 72_scores.cf are excluded from 50_scores.cf (static scores) and are currently incomplete making these rules default to 1.0. Most of theses missing rules should be much higher than 1.0 causing SA to allow spam through on most installations that don't have an optimized MTA in front of SA. https://wiki.apache.org/spamassassin/InfraNotes2017#mkupdates ~/svn/trunk/build/mkupdates/mkupdate-with-scores masses -> perl Makefile.PL && make (complete build of SA and test) - perl hit-frequencies - garescorer - compiles and runs it, requires build/pga <--- THE PROBLEM IS NEAR HERE I think the problem is somewhere behind line 127 and 128 which does many things/steps: http://svn.apache.org/viewvc/spamassassin/trunk/build/mkupdates/ 4. Most of the ruleqa/masscheck scripts are sh or bash. I am not a perl person. I am good at bash, Python, and a few others but not perl. I can hack my way though reading and updating existing perl but that's about it. I have been thinking about this the past few weeks and maybe there is a "band-aid" option where we could merge the current 72_scores.cf rules that are missing with the shorter version that is being generated today to keep the 72_scores.cf complete and get the rule updates happening again. Incomplete version from last night: http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?revision=1808406&view=markup&sortby=date Last good version: http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?revision=1786976&view=markup&sortby=date Dave -- David Jones
Re: getting help with SA sysadmin
On Sep 15, 2017, at 9:46 AM, David Jones wrote: > 3. I have narrowed down the problem to the general area of a perl Makefile > which builds a custom garescorer.c file which does some statistical analysis > to determine the best score for rules in the 72_scores.cf. These > 72_scores.cf are excluded from 50_scores.cf (static scores) and are currently > incomplete making these rules default to 1.0. Most of theses missing rules > should be much higher than 1.0 causing SA to allow spam through on most > installations that don't have an optimized MTA in front of SA. > > https://wiki.apache.org/spamassassin/InfraNotes2017#mkupdates > > ~/svn/trunk/build/mkupdates/mkupdate-with-scores > > masses -> perl Makefile.PL && make (complete build of SA and test) > - perl hit-frequencies > - garescorer - compiles and runs it, requires build/pga ><--- THE PROBLEM IS NEAR HERE where are hit-frequencies and garescorer? > I think the problem is somewhere behind line 127 and 128 which does many > things/steps: line 127 and 128 of mkupdate-withscores? (that just looks like it's building a version of SpamAssassin from trunk svn? maybe you are referring to something else?) -- Daniel J. Luke
Re: getting help with SA sysadmin
> On Sep 15, 2017, at 9:46 AM, David Jones wrote: >> 3. I have narrowed down the problem to the general area of a perl >> Makefile which builds a custom garescorer.c file which does some >> statistical analysis to determine the best score for rules in the >> 72_scores.cf. These 72_scores.cf are excluded from 50_scores.cf (static >> scores) and are currently incomplete making these rules default to 1.0. >> Most of theses missing rules should be much higher than 1.0 causing SA >> to allow spam through on most installations that don't have an optimized >> MTA in front of SA. >> >> https://wiki.apache.org/spamassassin/InfraNotes2017#mkupdates >> >> ~/svn/trunk/build/mkupdates/mkupdate-with-scores >> >> masses -> perl Makefile.PL && make (complete build of SA and test) >> - perl hit-frequencies >> - garescorer - compiles and runs it, requires build/pga >> <--- THE PROBLEM IS NEAR HERE > > where are hit-frequencies and garescorer? Seems its here: http://svn.apache.org/repos/asf/spamassassin/trunk/masses/ Not sure where its tied in tho. > >> I think the problem is somewhere behind line 127 and 128 which does many >> things/steps: > > line 127 and 128 of mkupdate-withscores? (that just looks like it's > building a version of SpamAssassin from trunk svn? maybe you are referring > to something else?) > > -- > Daniel J. Luke > > > >
Re: getting help with SA sysadmin
On 09/15/2017 09:52 AM, Merijn van den Kroonenberg wrote: On Sep 15, 2017, at 9:46 AM, David Jones wrote: 3. I have narrowed down the problem to the general area of a perl Makefile which builds a custom garescorer.c file which does some statistical analysis to determine the best score for rules in the 72_scores.cf. These 72_scores.cf are excluded from 50_scores.cf (static scores) and are currently incomplete making these rules default to 1.0. Most of theses missing rules should be much higher than 1.0 causing SA to allow spam through on most installations that don't have an optimized MTA in front of SA. https://wiki.apache.org/spamassassin/InfraNotes2017#mkupdates ~/svn/trunk/build/mkupdates/mkupdate-with-scores masses -> perl Makefile.PL && make (complete build of SA and test) - perl hit-frequencies - garescorer - compiles and runs it, requires build/pga <--- THE PROBLEM IS NEAR HERE where are hit-frequencies and garescorer? Seems its here: http://svn.apache.org/repos/asf/spamassassin/trunk/masses/ Not sure where its tied in tho. I think the problem is somewhere behind line 127 and 128 which does many things/steps: line 127 and 128 of mkupdate-withscores? (that just looks like it's building a version of SpamAssassin from trunk svn? maybe you are referring to something else?) 1. Actually start here with the runGA call: http://svn.apache.org/viewvc/spamassassin/trunk/masses/rule-update-score-gen/generate-new-scores.sh?revision=1798589&view=markup#l271 2. Here is the runGA script (not changed in almost 8 years): http://svn.apache.org/viewvc/spamassassin/trunk/masses/runGA?view=log 3. Somewhere in the bottom of the runGA script there is a problem generating a complete scores-set0 which becomes the 72_scores.cf: http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/ 4. This shows the resulting different in the last good 72_scores.cf and the latest version: http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?r1=1786976&r2=1808406 You kinda have to work backwards through the scripts to find what is generating the scores-set0 file and turning it into 72_scores.cf. I am grep'ing through the work dir on the SA server now but it contains a lot of files. I need to find the large dirs and exclude them. -- Daniel J. Luke -- David Jones
Re: Ends with string
Hi! Thanks! I didnt find this info in Writing rules tutorial. I see uri __KAM_SHORT /(\/|^|\b)(?:j\.mp|bit\.ly|goo\.gl|x\.co|t\.co|t\.cn|tinyurl\.com|hop\.kz|urla\.ru|fw\.to)(\/|$|\b)/i Seems a bit complicated. It would be to make this rule check that suffixes are at the end of URI. uri __TEST_URLS /\b(\.vn|\.pl|\.my|\.lu|\.vn|\.ar)\b/i I believe this does it, correct? uri __TEST_URLS /\b(\.vn$|\.pl$|\.my$|\.lu$|\.vn$|\.ar$)\b/i Thanks. Rob 2017-09-08 14:03 GMT-03:00 Kevin A. McGrail : > On 9/8/2017 12:24 PM, Robert Boyl wrote: > >> Hello, everyone! >> >> Is there a way to create a Spamassassin rule that checks for a certain >> URL suffix such as .ru but makes sure it has to be at the end of the URI? >> Ends with string. >> >> Thanks! >> Rob >> > > Yes, it's called an anchor and Shane Williams a long time ago gave me some > advice on that I used in this rule: > > uri __KAM_SHORT /(\/|^|\b)(?:j\.mp|bit\.ly|goo > \.gl|x\.co|t\.co|t\.cn|tinyurl\.com|hop\.kz|urla\.ru|fw\.to)(\/|$|\b)/i > > Regards, > KAM > >
Re: In anyone else getting 325KB spams from cont...@cron-job.org?
On 2017-09-15 13:32, RW wrote: > The default is 500kB for spamc, 256kB is a default for sa-learn. I have asked this before: Does this mean 500 * 1000 bytes or 512 * 1024 bytes, or something else still? (this is relevant when configuring other stuff which only understands straight byte counts with no suffixes) -- Please don't Cc: me privately on mailing lists and Usenet, if you also post the followup to the list or newsgroup. Do obvious transformation on domain to reply privately _only_ on Usenet.
Re: FORGED_YAHOO_RCVD still causing false positives
On 15/09/17 14:34, Kevin A. McGrail wrote: On 9/15/2017 8:26 AM, RW wrote: The rule was created and scored when spoofing Yahoo was very common, but it isn't any more. I don't think it's worth keeping as it is - high maintenance and error prone. Agreed. Score FORGED_YAHOO_RCVD to zero locally and will get a bug open to deprecate it. Regards, KAM Much appreciated - thank you both!
Re: FORGED_YAHOO_RCVD still causing false positives
Hi, On Fri, Sep 15, 2017 at 9:34 AM, Kevin A. McGrail wrote: > On 9/15/2017 8:26 AM, RW wrote: >> >> The rule was created and scored when spoofing Yahoo was very common, >> but it isn't any more. I don't think it's worth keeping as it is - high >> maintenance and error prone. > > > Agreed. Score FORGED_YAHOO_RCVD to zero locally and will get a bug open to > deprecate it. This then invalidates KAM_GRABBAG5 and KAM_UAH_YAHOOGROUP_SENDER from KAM.cf.
Re: Ends with string
Something along the following still seems the easiest to read approach to me
enlist_uri_host (BADTLDS) vn
enlist_uri_host (BADTLDS) pl
enlist_uri_host (BADTLDS) my
enlist_uri_host (BADTLDS) lu
enlist_uri_host (BADTLDS) ar
header __TEST_URLS eval:check_uri_host_listed('BADTLDS')
Paul
From: Robert Boyl
Date: Friday, 15 September 2017 at 17:48
To: "users@spamassassin.apache.org"
Subject: Re: Ends with string
Hi!
Thanks! I didnt find this info in Writing rules tutorial.
I see
uri __KAM_SHORT
/(\/|^|\b)(?:j\.mp|bit\.ly|goo\.gl|x\.co|t\.co|t\.cn|tinyurl\.com|hop\.kz|urla\.ru|fw\.to)(\/|$|\b)/i
Seems a bit complicated.
It would be to make this rule check that suffixes are at the end of URI.
uri __TEST_URLS /\b(\.vn|\.pl|\.my|\.lu|\.vn|\.ar)\b/i
I believe this does it, correct?
uri __TEST_URLS /\b(\.vn$|\.pl$|\.my$|\.lu$|\.vn$|\.ar$)\b/i
Thanks.
Rob
--
Paul Stead
Systems Engineer
Zen Internet
Re: Ends with string
On Fri, 15 Sep 2017, Paul Stead wrote:
Something along the following still seems the easiest to read approach to me
enlist_uri_host (BADTLDS) vn
enlist_uri_host (BADTLDS) pl
enlist_uri_host (BADTLDS) my
enlist_uri_host (BADTLDS) lu
enlist_uri_host (BADTLDS) ar
header __TEST_URLS eval:check_uri_host_listed('BADTLDS')
If you're only looking at uris, it probably is (though I wonder a
little about processing time between a long list of such entries and a
single (if also long) regular expression). I have rules for "bad"
tlds that look in headers as well (Received, From, Env_From being the
main ones), so these wouldn't help with that. If there's something
similar for those cases, I'd love to know about it.
--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/| System Admin - UT CompSci
=--+---
All syllogisms contain three lines | sha...@shanew.net
Therefore this is not a syllogism | www.ischool.utexas.edu/~shanew
Re: Ends with string
On 15/09/2017, 20:57, "sha...@shanew.net" wrote: If you're only looking at uris, it probably is (though I wonder a little about processing time between a long list of such entries and a single (if also long) regular expression). I have rules for "bad" tlds that look in headers as well (Received, From, Env_From being the main ones), so these wouldn't help with that. If there's something similar for those cases, I'd love to know about it. The following patch works for me: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7354 -- Paul Stead Systems Engineer Zen Internet
Re: Ends with string
On 15/09/2017, 20:59, "Paul Stead" wrote: On 15/09/2017, 20:57, "sha...@shanew.net" wrote: If you're only looking at uris, it probably is (though I wonder a little about processing time between a long list of such entries and a single (if also long) regular expression). I have rules for "bad" tlds that look in headers as well (Received, From, Env_From being the main ones), so these wouldn't help with that. If there's something similar for those cases, I'd love to know about it. The following patch works for me: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=7354 … though not with Received -- Paul Stead Systems Engineer Zen Internet
Re: getting help with SA sysadmin
On Sep 15, 2017, at 12:24 PM, David Jones wrote: > 1. Actually start here with the runGA call: > > http://svn.apache.org/viewvc/spamassassin/trunk/masses/rule-update-score-gen/generate-new-scores.sh?revision=1798589&view=markup#l271 > > 2. Here is the runGA script (not changed in almost 8 years): > > http://svn.apache.org/viewvc/spamassassin/trunk/masses/runGA?view=log > > 3. Somewhere in the bottom of the runGA script there is a problem generating > a complete scores-set0 which becomes the 72_scores.cf: > > http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/ > > 4. This shows the resulting different in the last good 72_scores.cf and the > latest version: > > http://svn.apache.org/viewvc/spamassassin/trunk/rulesrc/scores/72_scores.cf?r1=1786976&r2=1808406 I don't know if it helps, but I (manually) removed the lines that looked just like score changes from that diff and I get the following (on the assumption we want to have a good example of what to look for after making changes to know that this is 'fixed'): -score AC_SPAMMY_URI_PATTERNS1 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS10 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS11 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS12 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS2 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS3 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS4 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS8 1.000 1.000 1.000 1.000 -score AC_SPAMMY_URI_PATTERNS9 1.000 1.000 1.000 1.000 -score ADVANCE_FEE_2_NEW_FORM1.000 1.000 1.000 1.000 -score ADVANCE_FEE_2_NEW_MONEY 1.997 0.001 1.997 0.001 -score AXB_XM_FORGED_OL2600 1.190 2.699 1.190 2.699 -score BODY_EMPTY1.997 1.999 1.997 1.999 -score CANT_SEE_AD 2.996 0.500 2.996 0.500 -score CN_B2B_SPAMMER0.001 0.001 0.001 0.001 -score COMMENT_GIBBERISH 1.498 1.499 1.498 1.499 -score ENCRYPTED_MESSAGE -1.000 -1.000 -1.000 -1.000 -score FORM_LOW_CONTRAST 1.000 1.000 1.000 1.000 -score FREEMAIL_DOC_PDF_BCC 2.596 2.599 2.596 2.599 -score FREEMAIL_FORGED_FROMDOMAIN0.001 0.199 0.001 0.199 -score FROM_MISSPACED0.001 0.001 0.001 0.001 -score FROM_MISSP_SPF_FAIL 0.001 1.000 0.001 1.000 -score FROM_MISSP_XPRIO 0.001 0.001 0.001 0.001 -score FROM_WORDY_SHORT 1.000 1.000 1.000 1.000 -score GOOGLE_DOCS_PHISH 1.000 1.000 1.000 1.000 -score GOOGLE_DOCS_PHISH_MANY1.000 1.000 1.000 1.000 -score GOOG_MALWARE_DNLD 1.000 1.000 1.000 1.000 -score HEADER_FROM_DIFFERENT_DOMAINS 0.001 0.001 0.001 0.001 -score HEXHASH_WORD 1.000 1.000 1.000 1.000 -score HK_RANDOM_FROM0.998 0.001 0.998 0.001 -score HK_SCAM_N15 1.935 2.499 1.935 2.499 -score HTML_OFF_PAGE 1.000 1.000 1.000 1.000 -score LIST_PRTL_PUMPDUMP1.000 1.000 1.000 1.000 -score LONG_HEX_URI 2.194 2.290 2.194 2.290 -score LOTS_OF_MONEY 0.001 0.001 0.001 0.001 -score LOTTO_DEPT0.001 0.001 0.001 0.001 -score LUCRATIVE 1.000 1.000 1.000 1.000 -score MIMEOLE_DIRECT_TO_MX 1.445 0.381 1.445 0.381 -score MIME_NO_TEXT 1.000 1.000 1.000 1.000 -score MONEY_FRAUD_3 2.896 0.001 2.896 0.001 -score MONEY_FRAUD_5 3.096 0.001 3.096 0.001 -score MONEY_FRAUD_8 2.548 0.001 2.548 0.001 -score MONEY_LOTTERY 2.498 1.611 2.498 1.611 -score MSGID_NOFQDN1 2.395 3.299 2.395 3.299 -score MSM_PRIO_REPTO2.497 0.180 2.497 0.180 -score NSL_RCVD_FROM_USER0.548 0.001 0.548 0.001 -score NSL_RCVD_HELO_USER1.273 0.001 1.273 0.001 -score PHP_NOVER_MUA 1.000 1.000 1.000 1.000 -score PHP_ORIG_SCRIPT 0.502 2.499 0.502 2.499 -score PHP_SCRIPT_MUA1.000 1.000 1.000 1.000 -score PP_MIME_FAKE_ASCII_TEXT 0.429 0.001 0.429 0.001 -score PP_TOO_MUCH_UNICODE02 0.500 0.500 0.500 0.500 -score PP_TOO_MUCH_UNICODE05 1.000 1.000 1.000 1.000 -score PUMPDUMP 1.000 1.000 1.000 1.000 -score PUMPDUMP_MULTI1.000 1.000 1.000 1.000 -score RAND_HEADER_MANY 1.000 1.000 1.000 1.000 -score RCVD_IN_MSPIKE_BL 0.001 0.010 0.001 0.010 -score RCVD
Re: getting help with SA sysadmin
On Sep 15, 2017, at 12:24 PM, David Jones wrote: > You kinda have to work backwards through the scripts to find what is > generating the scores-set0 file and turning it into 72_scores.cf. I am > grep'ing through the work dir on the SA server now but it contains a lot of > files. I need to find the large dirs and exclude them. you may have already done this, but if you modify the scripts to not overwrite (or save a copy) of the intermediate files (which may clue into exactly where the problem is being introduced). ie. runGA lines 57-59, 124-132 (for 50_scores.cf) another 'easy' test I would try would be to set numcpus in runGA to 1 just in case the problem is that somewhere there are multiple writers overwriting parts of the same file -- Daniel J. Luke
Re: Ends with string
On 9/15/2017 12:48 PM, Robert Boyl wrote: Thanks! I didnt find this info in Writing rules tutorial. Yeah, I rewrote the rule a bit already. Thanks! It's in the latest KAM.cf.
Re: Ends with string
On Fri, 15 Sep 2017, Robert Boyl wrote:
uri
__KAM_SHORT/(\/|^|\b)(?:j\.mp|bit\.ly|goo\.gl|x\.co|t\.co|t\.cn|tinyurl\.com|hop\.kz|u
rla\.ru|fw\.to)(\/|$|\b)/i
Seems a bit complicated.
It would be to make this rule check that suffixes are at the end of URI.
uri __TEST_URLS /\b(\.vn|\.pl|\.my|\.lu|\.vn|\.ar)\b/i
I believe this does it, correct?
uri __TEST_URLS /\b(\.vn$|\.pl$|\.my$|\.lu$|\.vn$|\.ar$)\b/i
As Paul said, if you're just looking at uris, the enlist_uri might be
the better way to go. And it has the advantage that you don't have to
use (some might say abuse) regular expressions.
I believe URIs as collected for the uri tests consist of more than
just the server part of the URI, but maybe I'm wrong (or maybe the
list includes the server part only as well as the full URI). If I'm
correct, then using the "$" will not work where URIs have a local part
and might not work where there's only a trailing "/".
In the case where you're only looking at the TLD, you don't have to
worry about the front word boundary because you're explicitly
anchoring the front of the match with the "\." part. At the end, you
need to make sure that you're not allowing characters that would
indicate the server part of the URI continues past your intended match
(to avoid things like matching "blah.com" when you're really trying to
match ".co"). In my estimation, the characters that might indicate
continuation of the URI are letters, numbers, underscores, hyphens,
and the literal ".".
So, my rule for just matching TLDs looks like:
uri __TEST_URLS /\.(vn|pl|my|lu|vn|ar)\b[^\.-]/i
The "\b" part excludes the letters, numbers and underscore because
those wouldn't be a word boundary. The "[^\.-]" part excludes the
hyphen and literal "." from being on the right side of that word
boundary.
And now that I'm looking at it, I'm wondering if it would match a
URI like "https://legit.domain.com/great.beer/"; ("beer" being one of
the TLDs my rule contains). Like I said, the enlist_uri method might
be worth it just to avoid regular expressions.
--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/| System Admin - UT CompSci
=--+---
All syllogisms contain three lines | sha...@shanew.net
Therefore this is not a syllogism | www.ischool.utexas.edu/~shanew
Re: Ends with string
On Fri, 15 Sep 2017 15:46:31 -0500 (CDT)
sha...@shanew.net wrote:
> So, my rule for just matching TLDs looks like:
>
> uri __TEST_URLS /\.(vn|pl|my|lu|vn|ar)\b[^\.-]/i
>
> The "\b" part excludes the letters, numbers and underscore because
> those wouldn't be a word boundary. The "[^\.-]" part excludes the
> hyphen and literal "." from being on the right side of that word
> boundary.
note that [^\.-] has to match a character after the tld so it wouldn't
match "http://example.vn";
> And now that I'm looking at it, I'm wondering if it would match a
> URI like "https://legit.domain.com/great.beer/"; ("beer" being one of
> the TLDs my rule contains).
Yes it would, you can use something like ^[a-z]+\/\/:[^\/]* at the
beginning to avoid that.
An alternative is to use the URIDetail plugin and just test the domain.
https://spamassassin.apache.org/full/3.4.x/doc/Mail_SpamAssassin_Plugin_URIDetail.html
> Like I said, the enlist_uri method might be worth it just to avoid
> regular expressions.
In this case it is.
bb.barracudacentral.org
I wasn't quite sure what to put for the subject line. I didn't want to include the whole issue I'm seeing as I wanted to put it below. Every time SA is run on a message I'm seeing: spamd[15128]: spamd: processing message <1f44a5c57d9ff7ab8a90e109b.6708 c37fa7.20170915204926.d9a014b97f.eaede...@mail190.atl81.rsgsv.net> for chris:1000 localhost named[1284]: connection refused resolving '190.129.2.198.bb.barracudacentral.org/A/IN': 64.235.154.72#53 Is this an issue with an SA rule or some kind of an issue with Bind? I'm only using it as a caching name server. When running the message through spamassassin -D -t I see: dns: checking RBL bb.barracudacentral.org., set brbl-lastexternal async: launching A/190.129.2.198.bb.barracudacentral.org for dns:A:190.129.2.198.bb.barracudacentral.org dns: providing a callback for id: 28604/IN/A/190.129.2.198.bb.barracudacentral.org async: starting: DNSBL-A, dns:A:190.129.2.198.bb.barracudacentral.org (timeout 10.0s, min 2.0s) dns: dns reply to 28604/IN/A/190.129.2.198.bb.barracudacentral.org: NXDOMAIN async: calling callback on key dns:A:190.129.2.198.bb.barracudacentral.org async: completed in 1.024 s: DNSBL-A, dns:A:190.129.2.198.bb.barracudacentral.org async: timing: 1.024 . dns:A:190.129.2.198.bb.barracudacentral.org It's not a 'show stopper' it's just annoying to keep seeing this and wondering what the cause is. -- Chris KeyID 0xE372A7DA98E6705C 31.11972; -97.90167 (Elev. 1092 ft) 16:38:35 up 7 days, 6 min, 1 user, load average: 0.72, 0.61, 0.66 Description:Ubuntu 16.04.3 LTS, kernel 4.10.0-33-generic signature.asc Description: This is a digitally signed message part
Re: bb.barracudacentral.org
On 9/15/2017 5:50 PM, Chris wrote: It's not a 'show stopper' it's just annoying to keep seeing this and wondering what the cause is. You have configured your installation with the Baraccuda Reputation Black List but likely not subscribed your IP address. See http://barracudacentral.org/rbl I don't think rules are enabled for that by default.
Re: bb.barracudacentral.org
On 15.09.17 23:50, Chris wrote: > localhost named[1284]: connection refused resolving > '190.129.2.198.bb.barracudacentral.org/A/IN': 64.235.154.72#53 According to http://barracudacentral.org/rbl/how-to-use that should be b.barracudacentral.org, not bb.barracudacentral.org (single 'b'). -Ralph
Re: bb.barracudacentral.org
On Fri, 2017-09-15 at 18:20 -0400, Kevin A. McGrail wrote:
> On 9/15/2017 5:50 PM, Chris wrote:
> >
> > It's not a 'show stopper' it's just annoying to keep seeing this
> > and
> > wondering what the cause is.
> You have configured your installation with the Baraccuda Reputation
> Black List but likely not subscribed your IP address.
>
> See http://barracudacentral.org/rbl
>
> I don't think rules are enabled for that by default.
>
Thanks Kevin, problem is I have a dynamic IP therefore unless I'm
misunderstanding something I can't list IP addresses on the access
form. Here is the only rule that I can find in my local.cf referencing
Barracuda but I have it commented out:
# header __RCVD_IN_BRBL eval:check_rbl('brbl',
'bb.barracudacentral.org')
# describe __RCVD_IN_BRBL received via a relay in
bb.barracudacentral.org
# header RCVD_IN_BRBL_RELAY eval:check_rbl_sub('brbl',
'127.0.0.2')
# tflags RCVD_IN_BRBL_RELAY net
# describeRCVD_IN_BRBL_RELAY received via a relay rated as
poor by Barracuda
# score RCVD_IN_BRBL_RELAY 0
# header RCVD_IN_BRBL eval:check_rbl('brbl-
lastexternal', 'b.barracudacentral.org.', '127.0.0.2')
# describe RCVD_IN_BRBL Received via relay listed in
Barracuda RBL
# score RCVD_IN_BRBL 0
# tflags RCVD_IN_BRBL net
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
17:40:56 up 7 days, 1:09, 1 user, load average: 0.69, 0.79, 0.72
Description:Ubuntu 16.04.3 LTS, kernel 4.10.0-33-generic
signature.asc
Description: This is a digitally signed message part
Re: bb.barracudacentral.org
On 9/15/2017 6:46 PM, Chris wrote: On Fri, 2017-09-15 at 18:20 -0400, Kevin A. McGrail wrote: On 9/15/2017 5:50 PM, Chris wrote: It's not a 'show stopper' it's just annoying to keep seeing this and wondering what the cause is. You have configured your installation with the Baraccuda Reputation Black List but likely not subscribed your IP address. See http://barracudacentral.org/rbl I don't think rules are enabled for that by default. Thanks Kevin, problem is I have a dynamic IP therefore unless I'm misunderstanding something I can't list IP addresses on the access form. Here is the only rule that I can find in my local.cf referencing Barracuda but I have it commented out: I agree. You are likely not a candidate for BRBL usage with DynIP. I do not think that any stock rules include BRBL so I do not know where it is coming from. Anyone verify it's NOT included? Regards, KAM
Re: bb.barracudacentral.org
On Fri, 15 Sep 2017 18:51:22 -0400
Kevin A. McGrail wrote:
> On 9/15/2017 6:46 PM, Chris wrote:
> > On Fri, 2017-09-15 at 18:20 -0400, Kevin A. McGrail wrote:
> >> On 9/15/2017 5:50 PM, Chris wrote:
> >>> It's not a 'show stopper' it's just annoying to keep seeing this
> >>> and
> >>> wondering what the cause is.
> >> You have configured your installation with the Baraccuda Reputation
> >> Black List but likely not subscribed your IP address.
> >>
> >> See http://barracudacentral.org/rbl
> >>
> >> I don't think rules are enabled for that by default.
> >>
> > Thanks Kevin, problem is I have a dynamic IP therefore unless I'm
> > misunderstanding something I can't list IP addresses on the access
> > form. Here is the only rule that I can find in my local.cf
> > referencing Barracuda but I have it commented out:
> I agree. You are likely not a candidate for BRBL usage with DynIP. I
> do not think that any stock rules include BRBL so I do not know where
> it is coming from. Anyone verify it's NOT included?
$ grep -ri barracudacentral /var/db/spamassassin/
/var/db/spamassassin/3.004001/updates_spamassassin_org/72_active.cf:header
RCVD_IN_BRBL_LASTEXT
eval:check_rbl('brbl-lastexternal','bb.barracudacentral.org')
Re: bb.barracudacentral.org
On 9/15/2017 8:22 PM, RW wrote:
$ grep -ri barracudacentral /var/db/spamassassin/
/var/db/spamassassin/3.004001/updates_spamassassin_org/72_active.cf:header
RCVD_IN_BRBL_LASTEXT
eval:check_rbl('brbl-lastexternal','bb.barracudacentral.org')
That sounds like a mistake. Does it meet our default RBL inclusion
policy? Will look into this.
Re: bb.barracudacentral.org
On Fri, 2017-09-15 at 20:35 -0400, Kevin A. McGrail wrote:
> On 9/15/2017 8:22 PM, RW wrote:
> >
> > $ grep -ri barracudacentral /var/db/spamassassin/
> > /var/db/spamassassin/3.004001/updates_spamassassin_org/72_active.cf
> > :header RCVD_IN_BRBL_LASTEXT eval:check_rbl('brbl-
> > lastexternal','bb.barracudacentral.org')
> >
> That sounds like a mistake. Does it meet our default RBL inclusion
> policy? Will look into this.
Ok, I see now that in 72_active.cf there is
ifplugin Mail::SpamAssassin::Plugin::DNSEval
header RCVD_IN_BRBL_LASTEXT eval:check_rbl('brbl-
lastexternal','bb.barracudacentral.org')
tflags RCVD_IN_BRBL_LASTEXT net
endif
##} RCVD_IN_BRBL_LASTEXT ifplugin Mail::SpamAssassin::Plugin::DNSEval
The DNSEval plugin is called here:
# Plugins which used to be EvalTests.pm
# broken out into separate plugins
loadplugin Mail::SpamAssassin::Plugin::Bayes
loadplugin Mail::SpamAssassin::Plugin::BodyEval
loadplugin Mail::SpamAssassin::Plugin::DNSEval
loadplugin Mail::SpamAssassin::Plugin::HTMLEval
loadplugin Mail::SpamAssassin::Plugin::HeaderEval
loadplugin Mail::SpamAssassin::Plugin::MIMEEval
loadplugin Mail::SpamAssassin::Plugin::RelayEval
loadplugin Mail::SpamAssassin::Plugin::URIEval
loadplugin Mail::SpamAssassin::Plugin::WLBLEval
in /etc/mail/spamassassin/v320.pre
I can't just comment this out
ifplugin Mail::SpamAssassin::Plugin::DNSEval
header
RCVD_IN_BRBL_LASTEXT eval:check_rbl('brbl-
lastexternal','bb.barracudacentral.org')
tflags
RCVD_IN_BRBL_LASTEXT net
endif
since whenever a new 72_active.cf rule is downloaded it will be overwritten
anyway. In the same token I can't just turn off the DNSEval plugin since there
are other rules that use that plugin. It looks to me that I'll just ignore it
since as I said at first it's not causing any other issues it's just something
that had been nagging at me since I keep seeing it and wondering why.
--
Chris
KeyID 0xE372A7DA98E6705C
31.11972; -97.90167 (Elev. 1092 ft)
19:44:07 up 7 days, 3:12, 1 user, load average: 0.55, 2.25, 2.56
Description:Ubuntu 16.04.3 LTS, kernel 4.10.0-33-generic
signature.asc
Description: This is a digitally signed message part
Re: bb.barracudacentral.org
On 9/15/2017 8:51 PM, Chris wrote: Ok, I see now that in 72_active.cf there is Does adding a score RCVD_IN_BRBL_LASTEXT 0 to your local.cf work?
