>If we put the address on the DKIMNPAddress list, shouldn't it honor that
regardless of anything else?
REGARDLESS ???
No . this makes no sense.
The pre-DKIM check is skipped for (18103):
invalidSenderDomain (no valid TLD)
whitelisted
acceptAllMail
bounce mails
outgoing mails
RWL high trust
contentOnly
noprocessing (except noprocessing by size)
noDKIMAddresses
noDKIMIP
-----------------------
>don't know if a change from 0.001 to 0.005 would be significant or make
sense,
Just use the 'Bayes/HMM confidence' graph. This requires
'enableGraphStats' to be enabled.
>Then there's a ton of spam messages also advertising (fake) handbags and
they're often using the SAME domain and sometimes identical from address
as the legit mails.
ASSP has alot of features to indentify the correctness of the origin of an
email. And there is (IMHO) a big difference between maliciouse spam and
normal (not dangerous) mails you don't want to get. The later are often
problematic. Most times it is better to let them pass, than to block
important mails.
-----------------------
The next version will have an improvement for HMM and Bayesian.
An real problem may become disclaimers and privat and corporate signatues.
They are always added to outgoing mails, but also to spam reports. They
can be found in most of the answers to our mails. And for example, in my
case, they may be added by spammers to there spam mail. Nobody can say,
how the occurrence of such a disclaimer will affect the HMM and Bayesian
results. It may possible, that these results differs from day to day, or
block always good mails, or.let spam pass.
The only way to prevent such "wild" results is to remove the disclaimers,
before the rebuildspamdb task builds the spamdb and HMMdb. I use this code
for a month now and I'm really happy with the result.
Thomas
Von: "K Post" <nntp.p...@gmail.com>
An: "ASSP development mailing list" <assp-test@lists.sourceforge.net>
Datum: 16.04.2018 17:43
Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as
expected, but some messages still processed as spam?
I'm always correcting HMM/Bayes by reviewing the block report on a daily
basis and reporting. It's a horrible task that I dread, but it is worth
it. I have the score set to 50, and 50 as the threshold for rejection.
We've tried lower, but too much spam is only tagged solely due to a HMM
hit and slips through.
I never changed baysConf from the original 0.001 with baysprobablity to
0.6, but I do have a 1.000 corpus norm. It's certainly a mature
installation (15+ years). The issue I'm having is legit messages, say a
message advertising handbags from a reputable seller that one of our staff
buys from (over their lunch hour of course!!). Then there's a ton of spam
messages also advertising (fake) handbags and they're often using the SAME
domain and sometimes identical from address as the legit mails. HMM/Bayes
is rightfully biased against all handbag email. You're saying that
increasing baysConf will help the legit ones get through but still will
block the spam ones (with almost identical content)? I've read through
the gui for baysConf, but the problem is that which I understand HMM and
Bayes from a concept standpoint, the calculations aren't something I
understand, so I don't dare change the 0.0001 threshold without real
guidance from you. I understand 1 is the max, but don't know if increases
linearly or exponentially change levels, don't know if a change from 0.001
to 0.005 would be significant or make sense, etc.
My hopes with the dkim np was to let hmm spammy mail through if it's a
dkim match. Ignore all other results, if the DKIM is good, just let it
through was my thinking. I know handbag seller X sends ad mail that's
DKIM signed, but I don't know when IP they'll come from (or the IP is
mailing service that I don't want to blanket allow). This has been quite
successful with a whole lot of mail. I've become spoiled, now I want it
to work for all mail when there's a DKIM match to the NP list.
You raise another a good point about the 2 kind of DKIM checks, thanks for
the reminder. Does it make any sense to always have ASSP do the second
one and if it validates and matches dkimNPaddress or dkimWLadderss,
process solely based on that match? For example, HMM might hit before the
full body validation of DKIM, but so what? If we put the address on the
DKIMNPAddress list, shouldn't it honor that regardless of anything else?
NO processing, as I'm interpreting it, should mean, well, NONE, so if
other hits have happened, they're ignored because we said don't process.
I will temporarily change the logging level for a bit and see if I can
figure out why dkim isn't being done for these messages, but I'm guessing
that it's by design.
As always, thanks
Ken
On Mon, Apr 16, 2018 at 2:35 AM, Thomas Eckardt <
thomas.ecka...@thockar.com> wrote:
>I'm still seeing scenarios where analyze shows a DKIM NP match, but the
message is still going to spam based on score from HMM.
The best solution is to correct the corpus, to get better HMM results. You
may also decrease the scoring points for HMM and/or Bayesian. If the
corpus is corrected and the corpusnorm is ~ 1.0 , 'baysConf' will increase
detection correctness.
If assp receives a mail it acts as a statemachine. If and how a check is
done, depends on the previouse reached states.
Using the analyzer, assp acts procedural. Every check is done without any
state dependency. This is done, to be able to show every feature match.
The analyzer uses the current configuration, hashes, lists and databases.
So it may be normal to get different results compared to the real mail
processing loggings, if a mail is analyzed.
>DKIM NP match:
The analyzer checks DKIM without any dependency and shows all results.
But, if a mail is received, the DKIM check depends on several previouse
states
DKIM NP is a resulting state of the DKIM check. So - if any of the
previouse (DKIM depdency) states prevents the DKIM check, there will be no
DKIM (DKIM NP) result. The mail will be processed the same way, as it was
not DKIM signed.
Every state, that depends on DKIM NP will not be reached.
You should also remember, that assp use two DKIM checks. The full DKIM
check, which requires the full mail to be received - the results of this
check affects only the Plugin Level 2 (full mail) checks.
And the DKIM-Pre-Check - which is done after the MIME header is received
and if 'DKIMCacheInterval' is enabled. The results (states) of this check
affects most of the header checks and all body and full mail checks.
If you expect but miss a match for 'DKIMWLAddresses' or 'DKIMNPAddresses'
, you may increase the logging level (ValidateSenderLog, SessionLog,
ipmatchLogging, slmatchLogging).
Thomas
Von: "K Post" <nntp.p...@gmail.com>
An: "ASSP development mailing list" <
assp-test@lists.sourceforge.net>
Datum: 15.04.2018 23:21
Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as
expected, but some messages still processed as spam?
I'm still seeing scenarios where analyze shows a DKIM NP match, but the
message is still going to spam based on score from HMM.
On Mon, Apr 9, 2018 at 12:19 PM, K Post <nntp.p...@gmail.com> wrote:
cheers.
On Mon, Apr 9, 2018 at 3:36 AM, Thomas Eckardt <thomas.ecka...@thockar.com
> wrote:
If assp has modified the original mail header (changed foreign X-ASSP- or
removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress
and DKIMWLAddress.
The next version will try to check, if removed or changed headers are
protected by a DKIM signature and do the check, if this is not the case.
Thomas
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
