Re: Feature idea: Expiring rules

[Dianne Skoll]

On Wed, 14 Jun 2017 00:52:15 +0100
RW  wrote:

> If you want it to work that way it can be done in an external script
> in about 10 lines.

*SIGH*

Yes.  I'm perfectly aware of that.

My point is that we can have hundreds of sysadmins writing hacky little
scripts that all do things slightly differently and making up directory
hierarchies of their own that mean nothing to anyone else, OR we could
have a standard, supported, maintained way to do it in SpamAssassin.

> If you want a rule, or rules, to expire on a specific day put it in a
> file that starts with "YYMMDD-" in a separate directory. The script
> just needs to deal with the expired files, warn about any badly named
> files and generate a file containing a list of include lines.

Why not do it properly, once, so everyone can benefit?  And not put the
burden on sysadmins who have to document all these little hacks for their
successors?

Regards,

Dianne.

Re: Feature idea: Expiring rules

[RW]

On Tue, 13 Jun 2017 18:15:50 -0400
Dianne Skoll wrote:

 If you make a rule that you *know* will be effective for a
> pretty limited timespan, setting the expiry at the time of creation
> seems more efficient to me than having to remember to go back and
> expire it.

If you want it to work that way it can be done in an external script
in about 10 lines.

If you want a rule, or rules, to expire on a specific day put it in a
file that starts with "YYMMDD-" in a separate directory. The script just
needs to deal with the expired files, warn about any badly named files
and generate a file containing a list of include lines.

Re: Feature idea: Expiring rules

[Dianne Skoll]

On Tue, 13 Jun 2017 23:10:25 +0100
RW  wrote:

> Then why not write a script to parse your logs and determine when that
> happens.

Because that's more work, and I'm lazy, just like all true sysadmins.

> > What if we did something like:
> > expire MYRULE_FOO 2017-09-01

> This seems sub-optimal to me.

How so?  If you make a rule that you *know* will be effective for a pretty
limited timespan, setting the expiry at the time of creation seems more
efficient to me than having to remember to go back and expire it.

Of course, if you prefer to do the work, then you don't have to use
the expiration feature.

Regards,

Dianne.

Re: Feature idea: Expiring rules

[RW]

On Tue, 13 Jun 2017 11:39:10 -0400
Dianne Skoll wrote:

> Hi,
> 
> Something I and possibly others might find useful would be rules that
> expire.  Quite often, we might make some very specific rules to handle
> a particular spam run and they lose their effectiveness pretty
> quickly. 

Then why not write a script to parse your logs and determine when that
happens.

> What if we did something like:
> expire MYRULE_FOO 2017-09-01

This seems sub-optimal to me.

Re: Feature idea: Expiring rules

[John Hardin]

On Tue, 13 Jun 2017, Bowie Bailey wrote:


On 6/13/2017 3:53 PM, Dianne Skoll wrote:


 2) If a rule has an expiry set and then is used to build a meta rule,
 then the expiry is ignored and the parser issues a warning or even a
 fatal error.  I'm partial to the fatal error because warnings are usually
 ignored. :)


Or require that the meta rule must have the same (or possibly earlier) 
expiration date as the dependency.  This would allow more complex temporary 
rules to be created.


That sounds like a reasonable approach to me.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  So Microsoft's invented the ASCII equivalent to ugly ink spots that
  appear on your letter when your pen is malfunctioning.
 -- Greg Andrews, about Microsoft's way to encode apostrophes
---
 5 days until SWMBO's Birthday

Re: Feature idea: Expiring rules

[Bowie Bailey]

On 6/13/2017 3:53 PM, Dianne Skoll wrote:


2) If a rule has an expiry set and then is used to build a meta rule,
then the expiry is ignored and the parser issues a warning or even a
fatal error.  I'm partial to the fatal error because warnings are usually
ignored. :)


Or require that the meta rule must have the same (or possibly earlier) 
expiration date as the dependency.  This would allow more complex 
temporary rules to be created.


--
Bowie

Re: Feature idea: Expiring rules

[Benny Pedersen]

Kevin A. McGrail skrev den 2017-06-13 21:45:

On 6/13/2017 3:38 PM, Noel wrote:

Maybe expired rules could automatically score as 0.01 rather than
invalid.   Then log a warning to remind the admin.


I think that would defeat the purpose since the goal is likely to make
the core engine run more efficiently for RDJ items that are no longer
relevant and just waste cycles.


otoh, if rules expire, there would be less problems add new rules later

corpus score process is good dokumented, would that mean no more corpus 
use ?


life would indeed be lot more simple

working with spamassassin is like trying to catch 0day virus in a zip 
file, with a yara rule

Re: Feature idea: Expiring rules

[Noel]

On 6/13/2017 2:45 PM, John Hardin wrote:
> On Tue, 13 Jun 2017, Noel wrote:
>
>> On 6/13/2017 12:10 PM, Dianne Skoll wrote:
>>> On Tue, 13 Jun 2017 08:59:27 -0700 (PDT)
>>> John Hardin  wrote:
>>>
 Dependencies.
>>> Yes, that would mess things up.  Probably shouldn't be able to
>>> expire
>>> rules that others depend on.  The parser could check for that
>>> and make
>>> them non-expiring (with a warning.)
>>
>> Maybe expired rules could automatically score as 0.01 rather than
>> invalid.   Then log a warning to remind the admin.
>
> How do you adjust the scores of the meta rule(s) depending on it?
>

A non-expiring meta depending on an expiring rule?  Aim the gun away
from your foot.

But this shows that "expiring rules" isn't a simple thing.

A meta that expires would drop.  A valid meta that depends on
expired rules would have to be a fatal error; there's no reasonable
default action.

I guess that goes back to dropping expired rules, with a warning in
the docs about what happens with dependencies.

Re: Feature idea: Expiring rules

[Dianne Skoll]

On Tue, 13 Jun 2017 14:38:21 -0500
Noel  wrote:

> Maybe expired rules could automatically score as 0.01 rather than
> invalid.   Then log a warning to remind the admin.

No, I don't like that.  As others mentioned, that does nothing for dependent
rules.  I think a sensible use case for this would be:

1) It's meant for site rules, not the core rules or any other set of
automatically-updated or widely-distributed rules.

2) If a rule has an expiry set and then is used to build a meta rule,
then the expiry is ignored and the parser issues a warning or even a
fatal error.  I'm partial to the fatal error because warnings are usually
ignored. :)

3) Expired rules are removed from the list if already in memory or completely
ignored at parse time.

Regards,

Dianne.

Re: Feature idea: Expiring rules

[Martin Gregorie]

On Tue, 2017-06-13 at 14:38 -0500, Noel wrote:
> On 6/13/2017 12:10 PM, Dianne Skoll wrote:
> > On Tue, 13 Jun 2017 08:59:27 -0700 (PDT)
> > John Hardin  wrote:
> > 
> > > Dependencies.
> > 
> > Yes, that would mess things up.  Probably shouldn't be able to
> > expire
> > rules that others depend on.  The parser could check for that and
> > make
> > them non-expiring (with a warning.)
> > 
> 
> Maybe expired rules could automatically score as 0.01 rather than
> invalid.   Then log a warning to remind the admin.
> 
Nice idea: this also takes care of the dependency problem.


Martin

Re: Feature idea: Expiring rules

[Kevin A. McGrail]

On 6/13/2017 3:38 PM, Noel wrote:

Maybe expired rules could automatically score as 0.01 rather than
invalid.   Then log a warning to remind the admin.


I think that would defeat the purpose since the goal is likely to make 
the core engine run more efficiently for RDJ items that are no longer 
relevant and just waste cycles.

Re: Feature idea: Expiring rules

[John Hardin]

On Tue, 13 Jun 2017, Noel wrote:


On 6/13/2017 12:10 PM, Dianne Skoll wrote:

On Tue, 13 Jun 2017 08:59:27 -0700 (PDT)
John Hardin  wrote:


Dependencies.

Yes, that would mess things up.  Probably shouldn't be able to expire
rules that others depend on.  The parser could check for that and make
them non-expiring (with a warning.)


Maybe expired rules could automatically score as 0.01 rather than
invalid.   Then log a warning to remind the admin.


How do you adjust the scores of the meta rule(s) depending on it?

--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The tree of freedom must be freshened from time to time
  with the blood of tyrants and tyrannosaurs.
 -- DW, commenting on the GM6 Lynx .50BMG bullpup
---
 5 days until SWMBO's Birthday

Re: Feature idea: Expiring rules

[Noel]

On 6/13/2017 12:10 PM, Dianne Skoll wrote:
> On Tue, 13 Jun 2017 08:59:27 -0700 (PDT)
> John Hardin  wrote:
>
>> Dependencies.
> Yes, that would mess things up.  Probably shouldn't be able to expire
> rules that others depend on.  The parser could check for that and make
> them non-expiring (with a warning.)
>

Maybe expired rules could automatically score as 0.01 rather than
invalid.   Then log a warning to remind the admin.

Re: Feature idea: Expiring rules

[shanew]

On Tue, 13 Jun 2017, Dianne Skoll wrote:


Hi,

Something I and possibly others might find useful would be rules that
expire.  Quite often, we might make some very specific rules to handle
a particular spam run and they lose their effectiveness pretty quickly.


I would love this for private rules, especially if it could be applied
to blacklist (or whitelist, I suppose) entries.  We regularly
blacklist specific addresses when they've obviously fallen victim to
some form of compromise.  If I could set those to expire rather than
add an annotation that I have to manually remember (or more likely
forget) to remove later, it would be fantastic.


--
Public key #7BBC68D9 at| Shane Williams
http://pgp.mit.edu/|  System Admin - UT CompSci
=--+---
All syllogisms contain three lines |  sha...@shanew.net
Therefore this is not a syllogism  | www.ischool.utexas.edu/~shanew

Re: Feature idea: Expiring rules

[John Hardin]

On Tue, 13 Jun 2017, Kevin A. McGrail wrote:


On 6/13/2017 1:13 PM, Dianne Skoll wrote:

>  Brilliant idea but how to keep that information from spammers?

 Would it matter?  Especially for private site rules.  I wouldn't advocate
 this for centrally-distributed rules


I don't think it would matter except that it's functionality in the 
centrally-distributed or publicly available rules would be lessened from the 
public information.  It's an attack vector I would use as a bad guy.


I agree with you both. That expiry is implemented doesn't mean it needs to 
be used in the base rules, where spammers could leverage it.


I think the masscheck dynamically evaluating rules based on collected 
corpora is as close to rule expiry as the base rules need to come.


--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  One death is a tragedy; thirty is a media sensation;
  a million is a statistic.  -- Joseph Stalin, modernized
---
 5 days until SWMBO's Birthday

Re: Feature idea: Expiring rules

[Kevin A. McGrail]

On 6/13/2017 1:13 PM, Dianne Skoll wrote:

Brilliant idea but how to keep that information from spammers?

Would it matter?  Especially for private site rules.  I wouldn't advocate
this for centrally-distributed rules, which are in any event expired out by
removing the rules.


I don't think it would matter except that it's functionality in the 
centrally-distributed or publicly available rules would be lessened from 
the public information.  It's an attack vector I would use as a bad guy.

Re: Feature idea: Expiring rules

[Bryan Vest]

Our company does something similar with external scripts that we wrote. Our
small but powerful internal RBL works somewhat the same way. Very useful.

--Bryan

On Tue, Jun 13, 2017 at 1:13 PM, Dianne Skoll 
wrote:

> On Tue, 13 Jun 2017 11:56:57 -0400
> "Kevin A. McGrail"  wrote:
>
> > Brilliant idea but how to keep that information from spammers?
>
> Would it matter?  Especially for private site rules.  I wouldn't advocate
> this for centrally-distributed rules, which are in any event expired out by
> removing the rules.
>
> Regards,
>
> Dianne.
>

Re: Feature idea: Expiring rules

[Dianne Skoll]

On Tue, 13 Jun 2017 11:56:57 -0400
"Kevin A. McGrail"  wrote:

> Brilliant idea but how to keep that information from spammers?

Would it matter?  Especially for private site rules.  I wouldn't advocate
this for centrally-distributed rules, which are in any event expired out by
removing the rules.

Regards,

Dianne.

Re: Feature idea: Expiring rules

[Dianne Skoll]

On Tue, 13 Jun 2017 08:59:27 -0700 (PDT)
John Hardin  wrote:

> Dependencies.

Yes, that would mess things up.  Probably shouldn't be able to expire
rules that others depend on.  The parser could check for that and make
them non-expiring (with a warning.)

Regards,

Dianne.

Re: Feature idea: Expiring rules

[Benny Pedersen]

Kevin A. McGrail skrev den 2017-06-13 17:56:


I've been thinking a lot about hive protection on rules and
centralizing this type of data.


i will like to go the other way around, make spamassassin rules more 
decentraly, with feks ip repution, and uri repution based on maybe to 
see dmarc pass, say a spammail have 1 uri links to the payload, this 
would be tracked from dmarc as a spam for the dmarc record signed domain 
if learned with spamassassin -r


if its not dmarc pass, learn all ips and urls as spam

but not for url domains that are skipped testing, my above algo will 
imho not be beaten from spammmers side, but we miss dmarc still and 
repution localy tracked


just my own 3 euro, maybe i should have it in bitcoin :)

Re: Feature idea: Expiring rules

[John Hardin]

On Tue, 13 Jun 2017, Dianne Skoll wrote:


Hi,

Something I and possibly others might find useful would be rules that
expire.  Quite often, we might make some very specific rules to handle
a particular spam run and they lose their effectiveness pretty quickly.
What if we did something like:

expire MYRULE_FOO 2017-09-01

or maybe

tflags MYRULE_FOO expire=2017-09-01

Then the file parser would ignore expired rules, and if expired rules
have already been parsed into memory from before they expired, the run-time
would skip them.

Thoughts?


Dependencies. Rules that depend on expired rules would suddenly become 
invalid. That could be caught at lint time and avoid impacting a running 
system, but if the expiry is being checked and enforced dynamically at 
runtime as well that's a bit of a problem - either the rule goes invalid, 
or the dependency becomes an always-false (or always-true if the 
dependency is !negated).


Perhaps, for simplicity: expiry cannot be set on a rule that has other 
rules depending on it / you cannot use a rule having expiry in a META? 
Force that to be a lint-time check and avoid runtime complications.



--
 John Hardin KA7OHZhttp://www.impsec.org/~jhardin/
 jhar...@impsec.orgFALaholic #11174 pgpk -a jhar...@impsec.org
 key: 0xB8732E79 -- 2D8C 34F4 6411 F507 136C  AF76 D822 E6E6 B873 2E79
---
  The promise of nuclear power: electricity too cheap to meter
  The reality of nuclear power: FUD too cheap to meter
---
 5 days until SWMBO's Birthday

Re: Feature idea: Expiring rules

[Kevin A. McGrail]

On 6/13/2017 11:39 AM, Dianne Skoll wrote:

Something I and possibly others might find useful would be rules that
expire.  Quite often, we might make some very specific rules to handle
a particular spam run and they lose their effectiveness pretty quickly.
What if we did something like:

expire MYRULE_FOO 2017-09-01

or maybe

tflags MYRULE_FOO expire=2017-09-01

Then the file parser would ignore expired rules, and if expired rules
have already been parsed into memory from before they expired, the run-time
would skip them.


Brilliant idea but how to keep that information from spammers?

My thoughts on this topic have been an automatic gradiated disablement 
for efficiency.  Adding your idea of a date and the rule is only tested 
for every 1 out of X messages but goes back to 100% if it gets a certain 
# of hits.  This would allow SA to run on a subset of rules but increase 
the subset based on in the wild hits.


I've been thinking a lot about hive protection on rules and centralizing 
this type of data.


Regards,

KAM

Feature idea: Expiring rules

[Dianne Skoll]

Hi,

Something I and possibly others might find useful would be rules that
expire.  Quite often, we might make some very specific rules to handle
a particular spam run and they lose their effectiveness pretty quickly.
What if we did something like:

expire MYRULE_FOO 2017-09-01

or maybe

tflags MYRULE_FOO expire=2017-09-01

Then the file parser would ignore expired rules, and if expired rules
have already been parsed into memory from before they expired, the run-time
would skip them.

Thoughts?

Regards,

Dianne.

Re: version 3.4.1 with block TLD

[Robert Kudyba]

> On Jun 12, 2017, at 9:44 PM, Joseph Brennan  wrote:
> 
> 
> 
> --On June 8, 2017 at 12:07:43 PM -0400 Robert Kudyba  
> wrote:
> 
> I would like
>> to block *@*.us but allow the cities and schools that use them so allow
>> examples like @ci.boston.ma.us and corunna.k12.mi.us. I don’t think
>> this can be done with the access.db file in sendmail.
> 
> 
> Sendmail access.db? It's easy:
> 
> From:us REJECT
> From:ci.boston.ma.us OK
> From:corunna.k12.mi.usOK
> 
> Or name the states:
> 
> From:us   REJECT
> From:ma.usOK
> From:mi.usOK

Thanks for the suggestion didn’t realize it worked with the top level domain as 
well as the prefix before the TLD. Does the order matter? Meaning should the:
From:us REJECT 
come before the “OK” lines?

Also where can I add a rejection for the hostname colocrossing.com when it 
appears in the relay line like this. 
v5CB8tYp022453: ruleset=check_mail,  relay=198-23-201-144-host.colocrossing.com 
[198.23.201.144] (may be forged), 

I tried:
colocrossing.comREJECT

But that doesn’t seem to work.