On 11/09/2017 08:21 AM, Merijn van den Kroonenberg wrote:
On 11/09/2017 07:27 AM, Merijn van den Kroonenberg wrote:
Well... My local patch is still in place. This is lines 195-206 of
the
current trunk/masses/rule-update-score-gen/generate-new-scores.sh:
What is the full path? The question is in which working copy did you
apply
this change (what is it used for). Because in the working directories
used
for score generation it isn't. Thats in
/usr/local/spamassassin/automc/tmp/generate-new-scores/trunk-new-rules-set0
(each set having its fully checked out working copy)
In generate-new-scores.sh the workingcopies are done each night
completely
from scratch (old one removed, new one checked out). So no uncommitted
code can be used in the score generation part.
/usr/local/spamassassin/automc/svn/trunk/build/mkupdates/do-stable-update-with-scores
should be calling
/usr/local/spamassassin/automc/svn/masses/rule-update-score-gen/do-nightly-rescore-example.sh
which finally calls the locally modified and not committed script that
is intended to create our 72_active_before_grep.cf but it's not some
something is definitely not as I expected
/usr/local/spamassassin/automc/svn/masses/rule-update-score-gen/generate-new-scores.sh
Very strange, no idea why it doesn't work and nothing gets logged in the
mail.
can you do a:
svn status /usr/local/spamassassin/automc/svn/masses
and
svn status /usr/local/spamassassin/automc/svn/trunk
Both commands return nothing. I need to investigate this closer for
sure.
Thats weird, when it returns nothing it means there are no local changes
in the checkout.
Something which could affect this is the svn:ignore property, which
contains quite a list, but I don't really spot something which could
affect our case.
Unless ofcource, your changes are really gone now?
Sorry. I accidentally copy/pasted the first command twice. I was
juggling an issue at work and not focused.
root@sa-vm1:~# svn status /usr/local/spamassassin/automc/svn/trunk
M /usr/local/spamassassin/automc/svn/trunk/build/mkupdates/run_nightly
M
/usr/local/spamassassin/automc/svn/trunk/masses/rule-update-score-gen/generate-new-scores.sh
? /usr/local/spamassassin/automc/svn/trunk/ruleqa.cache.1
? /usr/local/spamassassin/automc/svn/trunk/ruleqa.cache.2
? /usr/local/spamassassin/automc/svn/trunk/ruleqa.cache.3
? /usr/local/spamassassin/automc/svn/trunk/rules/active.list.new
root@sa-vm1:~# svn diff
/usr/local/spamassassin/automc/svn/trunk/masses/rule-update-score-gen/generate-new-scores.sh
Index:
/usr/local/spamassassin/automc/svn/trunk/masses/rule-update-score-gen/generate-new-scores.sh
===================================================================
---
/usr/local/spamassassin/automc/svn/trunk/masses/rule-update-score-gen/generate-new-scores.sh
(revision 1813241)
+++
/usr/local/spamassassin/automc/svn/trunk/masses/rule-update-score-gen/generate-new-scores.sh
(working copy)
@@ -175,7 +175,7 @@
svn up trunk-new-rules-set$SCORESET/masses/
svn up trunk-new-rules-set$SCORESET/build/
-set +x
+#set +x
# we need to patch the Makefile to get it to mangle some data for us
cd trunk-new-rules-set${SCORESET}/masses
@@ -198,6 +198,9 @@
perl Makefile.PL < /dev/null || exit $?
make > make.out 2>&1 || exit $?
+# temp debug copy (investigate truncation issue)
+cp rules/72_active.cf 72_active_before_grep.cf
+
# strip scores from new rules so that the garescorer can set them
grep -v ^score rules/72_active.cf > rules/72_active.cf-scoreless
mv -f rules/72_active.cf-scoreless rules/72_active.cf
My changes are in there still but they don't seem to be executed. Must
be a different version of the script being run from cron. Here are the
cron commands in /etc/cron.d/automc
root@sa-vm1:~# cat /etc/cron.d/automc
MAILTO=sysadmins@spamassassin.apache.org
#
24 2 * * * automc svn update ~/svn/trunk/build > /dev/null
25 2 * * * automc
~/svn/trunk/build/mkupdates/do-stable-update-with-scores | /usr/bin/tee
/usr/local/spamassassin/automc/tmp/do-stable-update-with-scores.log
#
29 8 * * * automc svn update ~/svn/trunk/build/mkupdates > /dev/null
30 8 * * * automc ~/svn/trunk/build/mkupdates/run_nightly |
/usr/bin/tee /var/www/automc.spamassassin.org/mkupdates/mkupdates.txt
#
33 8 * * * automc svn update ~/svn/nitemc > /dev/null
34 8 * * 0-5 automc ~/svn/nitemc/corpora_runs >>
~/rsync/corpus/nightly-versions.txt
36 8 * * 0-5 automc ~/svn/nitemc/extract_to_rsync_dir nightly
~/rsync/corpus/nightly-versions.txt >
/tmp/last_nitemc_extract_nightly.log 2>&1
34 8 * * 6 automc ~/svn/nitemc/corpora_runs >>
~/rsync/corpus/weekly-versions.txt
36 8 * * 6 automc ~/svn/nitemc/extract_to_rsync_dir weekly
~/rsync/corpus/weekly-versions.txt > /tmp/last_nitemc_extract_weekly.log
2>&1
#
5 0-7,9-23 * * * automc . /etc/profile;
/usr/local/bin/runRuleQArefresh.sh > /dev/null
Day job and family stuff is currently keeping me busy. It's not
easy to tshoot this when it takes a long time between runs. Can't stay
focused on it. I could run these scripts manually in the evenings US
time but it still takes about 40 minutes if I remember correctly. It
did this a lot back in April and May when I could stay on top of it to
make some progress fairly quickly.
If I would enroll into apache, would I get ssh access so I could look
around myself?
Another major problem we have to figure out is the rules promotion
script from the build/mkupdates/run_nightly. If we can't get any rules
promoted, then masscheck'ing is just spinning it's wheels since we need
and updates active.list to tell what the current 72_scores.cf should
contain.
Tomorrow I have a day of (they are installing something at home, which
should not require my full attention). So I can dive into this tomorrow.
+ promote_active_rules
+ pwd
/usr/local/spamassassin/automc/svn/trunk
+ /usr/bin/perl build/mkupdates/listpromotable
HTTP get: http://ruleqa.spamassassin.org/1-days-ago?xml=1
HTTP get: http://ruleqa.spamassassin.org/2-days-ago?xml=1
no 'mcviewing', 'mcsubmitters' microformats on day 2
URL: http://ruleqa.spamassassin.org/2-days-ago?xml=1
+ exit 25
All of those script should be checking out fresh copies of everything
they work with into
/usr/local/spamassassin/automc/tmp
