Re: mailspike in 72_scores
You should rant about rspamd on list so you can shame moee into working on it :-) Regards, KAM On November 5, 2017 12:21:03 PM EST, Dave Jones wrote: >On 11/05/2017 10:26 AM, Merijn van den Kroonenberg wrote: >> Note to self: investigate why in do-nightly-rescore-example.sh is >called >> like this: >> merge-scoresets $SCORESET >> With a $SCORESET param. It will cause all rules which are not in that >> scores-set$SCORESET file to be ignored and not written to the >> 72_scores.cf. >> >> The dos version of the script: >> >http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/dos/new-rule-score-gen/do-nightly-rescore-example >> Also has the $SCORESET param, but it does not seem to be set anywhere >so >> it might have functioned as if no param was given. >> >> To me it sounds illogical to ignore the net rules, they are just >> masschecked less often, but should still be taken into consideration, >> right? >> >> >> And I just noticed: scores-set1 is still truncated at MILLION_USD, >but >> maybe because the masscheck is only once a week? >> >> > >Not sure either. I would like to get something to work even if it's >not >perfect so we can get sa-update updating again with "good enough" >72_scores.cf which would be better than nothing. We can't roll out any > >new rules at all right now. > >Then I would love to carefully redesign the scripts so they could be >run >hourly instead of daily and only run if there is an actual >change/commit >waiting to be tested. The new script would be more modular and share >functions/logic instead of each one doing nearly identical things but >slightly differently. The scripts would have proper logging with >meaningful return codes and not use "set -x" and "|| exit $?" which is >extremely difficult to troubleshoot. > >Right now with the current problem of commits messing up the masscheck >processing, it's taking days and weeks to troubleshoot problems as we >make changes to try to improve things. This makes the project seem >like >it's not doing anything when we are really trying hard to improve >things. > >We need to add new features and improve SA or rspamd is going to take >over and leave SA behind. I have some ideas and plans that I would >love >to implement over time to help the entire SA install base but we have >to >have a solid foundation first. > > Merijn, I am confused as to what is trying to be accomplished with the >patch on but 6400. Maybe I am misunderstanding something. It looks like >these MSPIKE scores need to remain in 50_scores.cf since they are tied to >a specific SA version check > 3.4.0. >>> >>> Normally, all rules which are in 50_scores.cf will never get >rescored >>> scores in 72_scores.cf. >>> >>> The script to which the patch applies does this, but due to a too >strict >>> regex, it doesn't see any score line if its prepended with >whitespace. >>> Do we need to remove them from 72_scores.cf, note that in the bug >then close it? >>> >>> Once the patch is applied, the mailspike rules should vanish from >>> 72_scores.cf by itself. >>> Dave >>> >>> >> >>
Re: mailspike in 72_scores
On 11/05/2017 10:26 AM, Merijn van den Kroonenberg wrote: Note to self: investigate why in do-nightly-rescore-example.sh is called like this: merge-scoresets $SCORESET With a $SCORESET param. It will cause all rules which are not in that scores-set$SCORESET file to be ignored and not written to the 72_scores.cf. The dos version of the script: http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/dos/new-rule-score-gen/do-nightly-rescore-example Also has the $SCORESET param, but it does not seem to be set anywhere so it might have functioned as if no param was given. To me it sounds illogical to ignore the net rules, they are just masschecked less often, but should still be taken into consideration, right? And I just noticed: scores-set1 is still truncated at MILLION_USD, but maybe because the masscheck is only once a week? Not sure either. I would like to get something to work even if it's not perfect so we can get sa-update updating again with "good enough" 72_scores.cf which would be better than nothing. We can't roll out any new rules at all right now. Then I would love to carefully redesign the scripts so they could be run hourly instead of daily and only run if there is an actual change/commit waiting to be tested. The new script would be more modular and share functions/logic instead of each one doing nearly identical things but slightly differently. The scripts would have proper logging with meaningful return codes and not use "set -x" and "|| exit $?" which is extremely difficult to troubleshoot. Right now with the current problem of commits messing up the masscheck processing, it's taking days and weeks to troubleshoot problems as we make changes to try to improve things. This makes the project seem like it's not doing anything when we are really trying hard to improve things. We need to add new features and improve SA or rspamd is going to take over and leave SA behind. I have some ideas and plans that I would love to implement over time to help the entire SA install base but we have to have a solid foundation first. Merijn, I am confused as to what is trying to be accomplished with the patch on but 6400. Maybe I am misunderstanding something. It looks like these MSPIKE scores need to remain in 50_scores.cf since they are tied to a specific SA version check > 3.4.0. Normally, all rules which are in 50_scores.cf will never get rescored scores in 72_scores.cf. The script to which the patch applies does this, but due to a too strict regex, it doesn't see any score line if its prepended with whitespace. Do we need to remove them from 72_scores.cf, note that in the bug then close it? Once the patch is applied, the mailspike rules should vanish from 72_scores.cf by itself. Dave
Re: mailspike in 72_scores
Note to self: investigate why in do-nightly-rescore-example.sh is called like this: merge-scoresets $SCORESET With a $SCORESET param. It will cause all rules which are not in that scores-set$SCORESET file to be ignored and not written to the 72_scores.cf. The dos version of the script: http://svn.apache.org/repos/asf/spamassassin/trunk/rulesrc/sandbox/dos/new-rule-score-gen/do-nightly-rescore-example Also has the $SCORESET param, but it does not seem to be set anywhere so it might have functioned as if no param was given. To me it sounds illogical to ignore the net rules, they are just masschecked less often, but should still be taken into consideration, right? And I just noticed: scores-set1 is still truncated at MILLION_USD, but maybe because the masscheck is only once a week? >> Merijn, >> >> I am confused as to what is trying to be accomplished with the patch on >> but 6400. Maybe I am misunderstanding something. It looks like these >> MSPIKE scores need to remain in 50_scores.cf since they are tied to a >> specific SA version check > 3.4.0. > > Normally, all rules which are in 50_scores.cf will never get rescored > scores in 72_scores.cf. > > The script to which the patch applies does this, but due to a too strict > regex, it doesn't see any score line if its prepended with whitespace. > >> >> Do we need to remove them from 72_scores.cf, note that in the bug then >> close it? > > Once the patch is applied, the mailspike rules should vanish from > 72_scores.cf by itself. > >> >> Dave > >
Re: mailspike in 72_scores
> Merijn, > > I am confused as to what is trying to be accomplished with the patch on > but 6400. Maybe I am misunderstanding something. It looks like these > MSPIKE scores need to remain in 50_scores.cf since they are tied to a > specific SA version check > 3.4.0. Normally, all rules which are in 50_scores.cf will never get rescored scores in 72_scores.cf. The script to which the patch applies does this, but due to a too strict regex, it doesn't see any score line if its prepended with whitespace. > > Do we need to remove them from 72_scores.cf, note that in the bug then > close it? Once the patch is applied, the mailspike rules should vanish from 72_scores.cf by itself. > > Dave
Re: mailspike in 72_scores
On 11/02/2017 08:11 AM, Kevin A. McGrail wrote: On 11/2/2017 6:44 AM, Merijn van den Kroonenberg wrote: While debugging the score generation problem, I encountered another weird case in 72_scores.cf. It is about the mailspike rules which are in 50_scores so should not be in 72_scores. There was already an existing issue about this: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6400 I found the cause of this and attached a patch to bugzilla. but since its already a very old issue I wonder if someone will ever look it it anymore ;) Thanks! I did have a look at it but I don't want to interfere with David's work. David, could you make the commit for him on this bug? Regards, KAM Merijn, I am confused as to what is trying to be accomplished with the patch on but 6400. Maybe I am misunderstanding something. It looks like these MSPIKE scores need to remain in 50_scores.cf since they are tied to a specific SA version check > 3.4.0. Do we need to remove them from 72_scores.cf, note that in the bug then close it? Dave
Re: mailspike in 72_scores
On 11/02/2017 08:11 AM, Kevin A. McGrail wrote: On 11/2/2017 6:44 AM, Merijn van den Kroonenberg wrote: While debugging the score generation problem, I encountered another weird case in 72_scores.cf. It is about the mailspike rules which are in 50_scores so should not be in 72_scores. There was already an existing issue about this: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6400 I found the cause of this and attached a patch to bugzilla. but since its already a very old issue I wonder if someone will ever look it it anymore ;) Thanks! I did have a look at it but I don't want to interfere with David's work. David, could you make the commit for him on this bug? I certainly will. I would like to wait a couple of days if that is OK since committing now could throw off the masscheck validations that we need to get sa-updates started again. Dave Regards, KAM
Re: mailspike in 72_scores
On 11/2/2017 6:44 AM, Merijn van den Kroonenberg wrote: While debugging the score generation problem, I encountered another weird case in 72_scores.cf. It is about the mailspike rules which are in 50_scores so should not be in 72_scores. There was already an existing issue about this: https://bz.apache.org/SpamAssassin/show_bug.cgi?id=6400 I found the cause of this and attached a patch to bugzilla. but since its already a very old issue I wonder if someone will ever look it it anymore ;) Thanks! I did have a look at it but I don't want to interfere with David's work. David, could you make the commit for him on this bug? Regards, KAM