>we could have the option to have HMM operate first
this is still the case
>and Bayes only run if HMM has too few results to score the message
Sounds good - I'll try to find a way to implement such an option.
What about this one ? An off/on switch seems not to be really nice. But if
wanted, this could be defined with 0.5-0.5 in the option below.
config name:
BayesAfterHMM
default is empty - no behavior change on an upgrade
description:
This value is ignored if DoHMM is not enabled or set to monitor. The
Bayesian check will only run, if the spam/ham probability of the HMM check
is in a given value range or the HMM check has given too few results.
Leave this blank to run the Bayesian check every time, independend from
any HMM result (default). To set this value, define a probability value
range like 0.3-0.7 (eg.) .
Thomas
Von: "Colin Waring" <[email protected]>
An: "'ASSP development mailing list'"
<[email protected]>,
Datum: 31.01.2014 17:44
Betreff: Re: [Assp-test] Bayes mistake
Hi Thomas,
This is actually a separate email. The previous one that I was talking
about
seems to be ok now. This mistake is a different message altogether.
I was wondering if we could have the option to have HMM operate first and
Bayes only run if HMM has too few results to score the message. This would
make setting the scoring much easier as HMM can be used to outright block
and Bayes against any that HMM can't handle.
Do you think that would improve the efficiency of the system? It would
also
reduce the number of checks run per message and therefore load on the
system
as well.
All the best,
Colin Waring.
-----Original Message-----
From: Thomas Eckardt [mailto:[email protected]]
Sent: 31 January 2014 13:26
To: ASSP development mailing list
Subject: Re: [Assp-test] Bayes mistake
Collin,
back to start - I think this was the short mail with the UTF-8 BOM and the
single link.
put the following in bombDataRe (in a single line)
^\s*[\S\x80-\xFF]{0,3}\s*(?:<\/?(?:html\s*|head\s*|meta[^>]*)>)+\s*\<\s*body
\s*>\s*(?:ht|f)tps?:\/\/[\w\.\/\-\?\&\=]+<\s*\/\s*body[^>]*>\s*<\s*\/html\s*
>[
\t\f]*\s*.{0,10}$
switch off 'DoTransliterate' otherwise the regex will not match
Thomas
Von: "Colin Waring" <[email protected]>
An: "'ASSP development mailing list'"
<[email protected]>,
Datum: 31.01.2014 11:03
Betreff: Re: [Assp-test] Bayes mistake
Hi Thomas,
Turns out another one got through the spam filtering yesterday evening.
Again same message content.
We have it the way it is because HMM misses smaller messages, we can't put
either one to a higher weight otherwise we end up with more false
positives.
I'd love to turn off Bayes and just use HMM but it isn't worth it for the
complaints on the short messages spam that gets through.
All the best,
Colin Waring.
-----Original Message-----
From: Thomas Eckardt [mailto:[email protected]]
Sent: 31 January 2014 07:04
To: ASSP development mailing list
Subject: Re: [Assp-test] Bayes mistake
Two reasons:
>I hadn't reported the previous one as a false negative yet.
1) Another one has reported the same or similar mail. ASSP V2 recalculates
the Bayes and HMM database on the fly if a mail is reported
2) A rebuild was done.
>Is there any way to figure out why Bayes made a boob on the first one?
No - all checks are done on the current DB's - no chance to go back in the
past. But I think, after eliminating pairs of very low (ham) and very high
(spam) values, there was at least one very low value left.
If you use both HMM and Bayes - set the scoring so, that your trust on HMM
is higher. Bayes is fine but less exact - for this reason HMM was
implemented.
Thomas
Von: "Colin Waring" <[email protected]>
An: "'ASSP development mailing list'"
<[email protected]>,
Datum: 30.01.2014 21:15
Betreff: [Assp-test] Bayes mistake
Hi there,
I'm wondering what's the best way to troubleshoot a Bayes mistake. We get
tonnes of fake bank security alert emails and nearly all of them got
blocked.
Imagine my surprise to see one in my own inbox this morning from
[email protected] <mailto:[email protected]>
So I checked the logs. What I found was more surprising. The exact same
message with the exact same content (I compared the .eml files and only
the
headers were different) hit my server later on and was blocked by Bayes. I
hadn't reported the previous one as a false negative yet.
Is there any way to figure out why Bayes made a boob on the first one?
Cheers,
Colin.
2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 <[email protected]> to: [email protected] HMM
Check
[scoring] - Prob: 1.00000 => spam
2014-01-30 09:41:52 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 <[email protected]> to: [email protected]
Message-Score: added 20 for HMM Probability: 1.0000, total score for this
message is now 35
2014-01-30 09:41:53 m1-74904-00342 [Worker_4] [TLS-in] [TLS-out]
212.227.137.50 <[email protected]> to: [email protected]
Bayesian
Check [scoring] - Prob: 0.10750 => ham
2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
<[email protected]> to: [email protected] HMM Check [scoring] -
Prob: 1.00000 => spam
2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
<[email protected]> to: [email protected] Message-Score: added
20
for HMM Probability: 1.0000, total score for this message is now 40
2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
<[email protected]> to: [email protected] Bayesian Check
[scoring]
- Prob: 0.99597 => spam
2014-01-30 12:40:56 m1-85654-02281 [Worker_7] [TLS-out] 85.94.77.22
<[email protected]> to: [email protected] Message-Score: added
30
for Bayesian Probability: 0.99597, total score for this message is now 70
----------------------------------------------------------------------------
--
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
----------------------------------------------------------------------------
--
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
WatchGuard Dimension instantly turns raw network data into actionable
security intelligence. It gives you real-time visual feedback on key
security issues and trends. Skip the complicated setup - simply import
a virtual appliance and go from zero to informed in seconds.
http://pubads.g.doubleclick.net/gampad/clk?id=123612991&iu=/4140/ostg.clktrk
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test