On 2017-11-16 10:06, Merijn van den Kroonenberg wrote:
On 11/15/2017 07:10 AM, Dave Jones wrote:
I got my SVN authentication issue figured out on my laptop and
committed
these. Fingers crossed for the run in about 5 hours.
I have been comparing last night's 72_scores.cf against the one from
march
and it looks *really* good now. That last commit pushed up the amount
of
lines right up to the amount as we had in march.
I also ran the compare-rulefiles script just like yesterday.
./compare-rulefiles -d 72_scores_20170315.cf 72_scores-1815405.cf >
deleted_rules.txt
./compare-rulefiles -r 0 -d deleted_rules.txt active-1815421.list >
deactivated_rules.txt
small mistake, I used a too-new active.list
./compare-rulefiles -r 0 -d deleted_rules.txt active-1815296.list >
deactivated_rules.txt
./compare-rulefiles -r 0 -a deactivated_rules.txt deleted_rules.txt >
disappeared_rules.txt
cat disappeared_rules.txt
ADVANCE_FEE_4_NEW
ADVANCE_FEE_5_NEW
CN_B2B_SPAMMER
URI_GOOGLE_PROXY
cat disappeared_rules.txt
ADVANCE_FEE_4_NEW
CN_B2B_SPAMMER
URI_GOOGLE_PROXY
So even less with correct active.list
So thats only 4 rules which are not in our new scores file but which
were
in the march one (discounting deactivated rules).
When looking at the changes between now and then, I see nothing
suspicious. i am now pretty confident the score generation is running
as
before in march.
Anything which is not right, probably wasn't right in march either ;)
I would say, lets get people testing!
Here are the full changes between now and march so you can see for
yourself:
./compare-rulefiles 72_scores_20170315.cf 72_scores-1815405.cf
Only in 1 (removed in 2)
ADVANCE_FEE_4_NEW
ADVANCE_FEE_5_NEW
AXB_XMAILER_MIMEOLE_OL_1ECD5
AXB_XM_FORGED_OL2600
BODY_EMPTY
CN_B2B_SPAMMER
FREEMAIL_DOC_PDF_BCC
FSL_HELO_BARE_IP_2
HDRS_LCASE
HK_SCAM_N15
LOTTO_AGENT
LOTTO_DEPT
MONEY_LOTTERY
MSGID_NOFQDN1
RP_MATCHES_RCVD
SHARE_50_50
TO_NO_BRKTS_FROM_MSSP
URI_GOOGLE_PROXY
Only in 2 (added in 2)
ADVANCE_FEE_4_NEW_MONEY
ADVANCE_FEE_5_NEW_FRM_MNY
ADVANCE_FEE_5_NEW_MONEY
APOSTROPHE_TOCC
AXB_X_AOL_SEZ_S
DEAR_BENEFICIARY
FROM_MISSP_DYNIP
FSL_HELO_FAKE
FSL_MIME_NO_TEXT
FUZZY_UNSUBSCRIBE
HDRS_MISSP
MANY_PILL_PRICE
MILLION_USD
MONEY_ATM_CARD
MONEY_FORM
MONEY_FORM_SHORT
MONEY_FROM_41
MONEY_FROM_MISSP
SERGIO_SUBJECT_VIAGRA01
SHORTENED_URL_SRC
SINGLETS_LOW_CONTRAST
SPOOFED_FREEM_REPTO_RUS
TO_NO_BRKTS_DYNIP
Changed
AC_HTML_NONSENSE_TAGS
1.000 0.001 1.000 0.001
1.000 1.000 1.000 1.000
ADVANCE_FEE_2_NEW_MONEY
1.997 0.001 1.997 0.001
0.001 0.020 0.001 0.020
ADVANCE_FEE_3_NEW
3.496 0.001 3.496 0.001
3.001 3.467 3.001 3.467
ADVANCE_FEE_3_NEW_MONEY
2.796 0.001 2.796 0.001
3.099 2.696 3.099 2.696
AXB_XMAILER_MIMEOLE_OL_024C2
0.367 0.001 0.367 0.001
1.816 0.006 1.816 0.006
BODY_URI_ONLY
0.998 0.001 0.998 0.001
1.000 0.999 1.000 0.999
BOGUS_MSM_HDRS
0.909 0.001 0.909 0.001
0.795 1.377 0.795 1.377
CANT_SEE_AD
2.996 0.500 2.996 0.500
1.000 1.000 1.000 1.000
CK_HELO_DYNAMIC_SPLIT_IP
1.350 0.001 1.350 0.001
1.500 0.107 1.500 0.107
CK_HELO_GENERIC
0.249 0.249 0.249 0.249
0.250 0.248 0.250 0.248
COMMENT_GIBBERISH
1.498 1.499 1.498 1.499
1.000 1.000 1.000 1.000
DATE_IN_FUTURE_96_Q
3.296 3.299 3.296 3.299
2.899 2.696 2.899 2.696
FBI_MONEY
0.696 0.001 0.696 0.001
1.000 1.000 1.000 1.000
FBI_SPOOF
1.999 1.999 1.999 1.999
1.000 1.000 1.000 1.000
FILL_THIS_FORM
2.748 0.001 2.748 0.001
0.113 1.488 0.113 1.488
FORM_FRAUD
0.998 0.001 0.998 0.001
1.000 0.998 1.000 0.998
FORM_FRAUD_3
2.696 0.001 2.696 0.001
2.899 0.999 2.899 0.999
FORM_FRAUD_5
0.209 0.001 0.209 0.001
3.499 1.594 3.499 1.594
FOUND_YOU
3.013 0.001 3.013 0.001
1.000 1.000 1.000 1.000
FREEMAIL_FORGED_FROMDOMAIN
0.001 0.199 0.001 0.199
0.001 0.001 0.001 0.001
FROM_IN_TO_AND_SUBJ
0.287 0.262 0.287 0.262
0.001 0.001 0.001 0.001
FROM_MISSP_FREEMAIL
3.595 0.001 3.595 0.001
2.213 1.781 2.213 1.781
FROM_MISSP_MSFT
0.001 0.001 0.001 0.001
1.097 1.596 1.097 1.596
FROM_MISSP_REPLYTO
0.001 0.001 0.001 0.001
2.443 0.001 2.443 0.001
FROM_MISSP_SPF_FAIL
0.001 1.000 0.001 1.000
0.001 0.001 0.001 0.001
FROM_MISSP_TO_UNDISC
1.438 0.001 1.438 0.001
1.472 0.448 1.472 0.448
FROM_MISSP_USER
0.001 0.001 0.001 0.001
3.316 1.188 3.316 1.188
FROM_MISSP_XPRIO
0.001 0.001 0.001 0.001
1.785 2.497 1.785 2.497
FROM_WORDY
2.497 0.001 2.497 0.001
2.500 2.498 2.500 2.498
FSL_CTYPE_WIN1251
0.001 0.001 0.001 0.001
3.515 3.080 3.515 3.080
FSL_NEW_HELO_USER
0.083 0.001 0.083 0.001
1.719 0.750 1.719 0.750
HELO_MISC_IP
0.248 0.250 0.248 0.250
0.250 0.249 0.250 0.249
HK_RANDOM_FROM
0.998 0.001 0.998 0.001
0.999 0.999 0.999 0.999
HK_SCAM_N2
3.249 0.001 3.249 0.001
1.498 2.696 1.498 2.696
IMG_DIRECT_TO_MX
2.397 2.400 2.397 2.400
3.599 1.744 3.599 1.744
LIST_PRTL_SAME_USER
0.001 0.286 0.001 0.286
1.000 1.000 1.000 1.000
LONG_HEX_URI
2.194 2.290 2.194 2.290
1.102 0.853 1.102 0.853
LONG_IMG_URI
0.553 0.100 0.553 0.100
0.554 1.000 0.554 1.000
LOTS_OF_MONEY
0.001 0.001 0.001 0.001
0.001 0.005 0.001 0.005
MIMEOLE_DIRECT_TO_MX
1.445 0.381 1.445 0.381
1.999 0.738 1.999 0.738
MIME_NO_TEXT
1.000 1.000 1.000 1.000
1.803 1.997 1.803 1.997
MONEY_FRAUD_3
2.896 0.001 2.896 0.001
3.099 0.263 3.099 0.263
MONEY_FRAUD_5
3.096 0.001 3.096 0.001
3.400 2.896 3.400 2.896
MONEY_FRAUD_8
2.548 0.001 2.548 0.001
2.938 2.600 2.938 2.600
MSM_PRIO_REPTO
2.497 0.180 2.497 0.180
1.000 1.000 1.000 1.000
NSL_RCVD_FROM_USER
0.548 0.001 0.548 0.001
1.622 1.864 1.622 1.864
NSL_RCVD_HELO_USER
1.273 0.001 1.273 0.001
0.876 0.742 0.876 0.742
PHP_ORIG_SCRIPT
0.502 2.499 0.502 2.499
1.000 1.000 1.000 1.000
PP_MIME_FAKE_ASCII_TEXT
0.429 0.001 0.429 0.001
1.000 0.001 1.000 0.001
RCVD_IN_MSPIKE_H2
0.001 -2.800 0.001 -2.800
0.001 -1.240 0.001 -1.240
RCVD_IN_MSPIKE_L3
0.001 0.001 0.001 0.001
0.001 1.284 0.001 1.284
RCVD_IN_MSPIKE_L4
0.001 0.001 0.001 0.001
0.001 0.149 0.001 0.149
RCVD_IN_MSPIKE_L5
0.001 0.001 0.001 0.001
0.001 2.284 0.001 2.284
RCVD_IN_MSPIKE_ZBI
0.001 0.001 0.001 0.001
0.001 3.496 0.001 3.496
SPOOFED_FREEM_REPTO
2.498 1.368 2.498 1.368
2.500 2.497 2.500 2.497
STATIC_XPRIO_OLE
1.997 0.001 1.997 0.001
1.999 0.638 1.999 0.638
STOCK_LOW_CONTRAST
2.030 2.347 2.030 2.347
1.000 1.000 1.000 1.000
STYLE_GIBBERISH
2.800 3.093 2.800 3.093
3.499 2.570 3.499 2.570
THIS_AD
0.596 2.200 0.596 2.200
0.001 0.001 0.001 0.001
TO_EQ_FM_DIRECT_MX
2.497 0.622 2.497 0.622
2.499 2.497 2.499 2.497
TO_IN_SUBJ
0.099 0.099 0.099 0.099
0.099 0.001 0.099 0.001
TO_NO_BRKTS_HTML_IMG
0.001 2.000 0.001 2.000
0.001 1.997 0.001 1.997
TO_NO_BRKTS_HTML_ONLY
1.997 0.001 1.997 0.001
0.001 0.263 0.001 0.263
TO_NO_BRKTS_MSFT
2.497 0.001 2.497 0.001
2.499 2.497 2.499 2.497
TO_NO_BRKTS_NORDNS_HTML
0.398 0.001 0.398 0.001
0.001 0.340 0.001 0.340
TO_NO_BRKTS_PCNT
2.497 0.001 2.497 0.001
2.499 2.020 2.499 2.020
TVD_SPACE_ENCODED
2.497 0.001 2.497 0.001
2.499 2.497 2.499 2.497
TVD_SPACE_ENC_FM_MIME
1.997 0.001 1.997 0.001
1.999 1.997 1.999 1.997
TVD_SPACE_RATIO_MINFP
2.497 0.001 2.497 0.001
2.500 2.497 2.500 2.497
URI_ONLY_MSGID_MALF
0.001 1.191 0.001 1.191
1.999 1.997 1.999 1.997
URI_PHISH
3.995 3.999 3.995 3.999
1.000 1.000 1.000 1.000
URI_TRY_3LD
0.195 0.001 0.195 0.001
0.001 0.225 0.001 0.225
URI_TRY_USME
0.001 0.001 0.001 0.001
3.299 2.896 3.299 2.896
URI_WPADMIN
3.396 3.014 3.396 3.014
2.899 2.497 2.899 2.497
URI_WP_DIRINDEX
1.000 1.000 1.000 1.000
3.499 3.496 3.499 3.496
URI_WP_HACKED
2.996 3.000 2.996 3.000
3.499 3.150 3.499 3.150
URI_WP_HACKED_2
1.187 1.764 1.187 1.764
1.488 2.497 1.488 2.497
XPRIO
2.248 2.249 2.248 2.249
0.536 0.140 0.536 0.140
Cheers, Merijn
Dave
