I am seeing this with a bunch of other addresses now. DKIM verifies. HMM
fails, dkim not processed when accepting mail. What do you think about
doing the DKIMNPAddress check even if hmm passes the threshold? I think
everyone would benefit from this.
On Tue, Apr 17, 2018 at 11:20 AM, K Post <nntp.p...@gmail.com> wrote:
> Okay, "regardless" was a bit much, I agree with the reasons you have
> listed for DKIM skipping and figured that was the case for all. What I'm
> seeing with 18103 is mail that scores with hmm/bayes only NOT being
> noprocessed due to a DKIMNPAddress hit. We know it's a good signature
> because analyze says it's a match, but for whatever reason ASSP isn't no
> processing it after a hmm or possibily other scoring threthold match.
> Unfortunately, I haven't seen an example since I made logging more verbose,
> or I'd have provide more info. Can you confirm that DKIMNPAddress is
> supposed to run and results be honored if a message's score already exceeds
> the reject threshold AND none of the exceptions you previously listed are
> true?
>
> I'll take a look at the graph to see what I can learn.
>
> You definitely peaked my interest when you mentioned the disclaimers file
> previously.
>
> About 70% of our staff uses signatures, generally unique to them. We're
> small enough that I guess I could start compiling a list for the
> disclaimers file. I assume that grabbing the signatures right from mail
> text files in the corpus would be okay? That would pick up the html markup
> they tend to use. Yes?
>
> At least 50% (guessing) of our legitimate *inbound* mail has signatures
> and disclaimers on them too. I can't see manually maintaining a
> disclaimers file with all of them in it. Which is better, having some
> inbound signatures in the file or NONE? I don't want to create a bias
> against signatures that aren't in the disclaimer file.
>
> On Tue, Apr 17, 2018 at 5:17 AM, Thomas Eckardt <
> thomas.ecka...@thockar.com> wrote:
>
>> >If we put the address on the DKIMNPAddress list, shouldn't it honor
>> that regardless of anything else?
>>
>> REGARDLESS ???
>>
>> No . this makes no sense.
>>
>> The pre-DKIM check is skipped for (18103):
>>
>> invalidSenderDomain (no valid TLD)
>> whitelisted
>> acceptAllMail
>> bounce mails
>> outgoing mails
>> RWL high trust
>> contentOnly
>> noprocessing (except noprocessing by size)
>> noDKIMAddresses
>> noDKIMIP
>>
>> -----------------------
>> >don't know if a change from 0.001 to 0.005 would be significant or make
>> sense,
>>
>> Just use the 'Bayes/HMM confidence' graph. This requires
>> 'enableGraphStats' to be enabled.
>>
>> >Then there's a ton of spam messages also advertising (fake) handbags
>> and they're often using the SAME domain and sometimes identical from
>> address as the legit mails.
>>
>> ASSP has alot of features to indentify the correctness of the origin of
>> an email. And there is (IMHO) a big difference between maliciouse spam and
>> normal (not dangerous) mails you don't want to get. The later are often
>> problematic. Most times it is better to let them pass, than to block
>> important mails.
>>
>> -----------------------
>>
>> The next version will have an improvement for HMM and Bayesian.
>> An real problem may become disclaimers and privat and corporate
>> signatues. They are always added to outgoing mails, but also to spam
>> reports. They can be found in most of the answers to our mails. And for
>> example, in my case, they may be added by spammers to there spam mail.
>> Nobody can say, how the occurrence of such a disclaimer will affect the HMM
>> and Bayesian results. It may possible, that these results differs from day
>> to day, or block always good mails, or.let spam pass.
>>
>> The only way to prevent such "wild" results is to remove the disclaimers,
>> before the rebuildspamdb task builds the spamdb and HMMdb. I use this code
>> for a month now and I'm really happy with the result.
>>
>> Thomas
>>
>>
>>
>>
>>
>> Von: "K Post" <nntp.p...@gmail.com>
>> An: "ASSP development mailing list" <assp-test@lists.sourceforge.n
>> et>
>> Datum: 16.04.2018 17:43
>> Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as
>> expected, but some messages still processed as spam?
>> ------------------------------
>>
>>
>>
>> I'm always correcting HMM/Bayes by reviewing the block report on a daily
>> basis and reporting. It's a horrible task that I dread, but it is worth
>> it. I have the score set to 50, and 50 as the threshold for rejection.
>> We've tried lower, but too much spam is only tagged *solely* due to a
>> HMM hit and slips through.
>>
>> I never changed baysConf from the original 0.001 with baysprobablity to
>> 0.6, but I do have a 1.000 corpus norm. It's certainly a mature
>> installation (15+ years). The issue I'm having is legit messages, say a
>> message advertising handbags from a reputable seller that one of our staff
>> buys from (over their lunch hour of course!!). Then there's a ton of spam
>> messages also advertising (fake) handbags and they're often using the SAME
>> domain and sometimes identical from address as the legit mails. HMM/Bayes
>> is rightfully biased against all handbag email. You're saying that
>> increasing baysConf will help the legit ones get through but still will
>> block the spam ones (with almost identical content)? I've read through
>> the gui for baysConf, but the problem is that which I understand HMM and
>> Bayes from a concept standpoint, the calculations aren't something I
>> understand, so I don't dare change the 0.0001 threshold without real
>> guidance from you. I understand 1 is the max, but don't know if increases
>> linearly or exponentially change levels, don't know if a change from 0.001
>> to 0.005 would be significant or make sense, etc.
>>
>> My hopes with the dkim np was to let hmm spammy mail through if it's a
>> dkim match. Ignore all other results, if the DKIM is good, just let it
>> through was my thinking. I know handbag seller X sends ad mail that's DKIM
>> signed, but I don't know when IP they'll come from (or the IP is mailing
>> service that I don't want to blanket allow). This has been quite
>> successful with a whole lot of mail. I've become spoiled, now I want it to
>> work for all mail when there's a DKIM match to the NP list.
>>
>> You raise another a good point about the 2 kind of DKIM checks, thanks
>> for the reminder. Does it make any sense to *always* have ASSP do the
>> second one and if it validates and matches dkimNPaddress or dkimWLadderss,
>> process solely based on that match? For example, HMM might hit before the
>> full body validation of DKIM, but so what? If we put the address on the
>> DKIMNPAddress list, shouldn't it honor that regardless of anything else?
>> NO processing, as I'm interpreting it, should mean, well, NONE, so if other
>> hits have happened, they're ignored because we said don't process.
>>
>> I will temporarily change the logging level for a bit and see if I can
>> figure out why dkim isn't being done for these messages, but I'm guessing
>> that it's by design.
>>
>> As always, thanks
>> Ken
>>
>>
>> On Mon, Apr 16, 2018 at 2:35 AM, Thomas Eckardt <
>> *thomas.ecka...@thockar.com* <thomas.ecka...@thockar.com>> wrote:
>> >I'm still seeing scenarios where analyze shows a DKIM NP match, but the
>> message is still going to spam based on score from HMM.
>>
>> The best solution is to correct the corpus, to get better HMM results.
>> You may also decrease the scoring points for HMM and/or Bayesian. If the
>> corpus is corrected and the corpusnorm is ~ 1.0 , 'baysConf' will increase
>> detection correctness.
>>
>>
>> If assp receives a mail it acts as a statemachine. If and how a check is
>> done, depends on the previouse reached states.
>>
>> Using the analyzer, assp acts procedural. Every check is done without any
>> state dependency. This is done, to be able to show every feature match.
>> The analyzer uses the current configuration, hashes, lists and databases.
>> So it may be normal to get different results compared to the real mail
>> processing loggings, if a mail is analyzed.
>>
>> >DKIM NP match:
>>
>> The analyzer checks DKIM without any dependency and shows all results.
>>
>> But, if a mail is received, the DKIM check depends on several previouse
>> states
>> DKIM NP is a resulting state of the DKIM check. So - if any of the
>> previouse (DKIM depdency) states prevents the DKIM check, there will be no
>> DKIM
>> (DKIM NP) result. The mail will be processed the same way, as it was not
>> DKIM signed.
>> Every state, that depends on DKIM NP will not be reached.
>>
>> You should also remember, that assp use two DKIM checks. The full DKIM
>> check, which requires the full mail to be received - the results of this
>> check affects only the Plugin Level 2 (full mail) checks.
>> And the DKIM-Pre-Check - which is done after the MIME header is received
>> and if 'DKIMCacheInterval' is enabled. The results (states) of this check
>> affects most of the header checks and all body and full mail checks.
>>
>> If you expect but miss a match for 'DKIMWLAddresses' or 'DKIMNPAddresses'
>> , you may increase the logging level (ValidateSenderLog, SessionLog,
>> ipmatchLogging, slmatchLogging).
>>
>> Thomas
>>
>>
>>
>>
>> Von: "K Post" <*nntp.p...@gmail.com* <nntp.p...@gmail.com>>
>> An: "ASSP development mailing list" <
>> *assp-test@lists.sourceforge.net* <assp-test@lists.sourceforge.net>>
>> Datum: 15.04.2018 23:21
>> Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as
>> expected, but some messages still processed as spam?
>> ------------------------------
>>
>>
>>
>> I'm still seeing scenarios where analyze shows a DKIM NP match, but the
>> message is still going to spam based on score from HMM.
>>
>> On Mon, Apr 9, 2018 at 12:19 PM, K Post <*nntp.p...@gmail.com*
>> <nntp.p...@gmail.com>> wrote:
>> cheers.
>>
>> On Mon, Apr 9, 2018 at 3:36 AM, Thomas Eckardt <
>> *thomas.ecka...@thockar.com* <thomas.ecka...@thockar.com>> wrote:
>> If assp has modified the original mail header (changed foreign X-ASSP- or
>> removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress and
>> DKIMWLAddress.
>>
>> The next version will try to check, if removed or changed headers are
>> protected by a DKIM signature and do the check, if this is not the case.
>>
>> Thomas
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! *http://sdm.link/slashdot*
>> <http://sdm.link/slashdot>
>> _______________________________________________
>> Assp-test mailing list
>> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
>> *https://lists.sourceforge.net/lists/listinfo/assp-test*
>> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! *http://sdm.link/slashdot*
>> <http://sdm.link/slashdot>
>> _______________________________________________
>> Assp-test mailing list
>> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
>> *https://lists.sourceforge.net/lists/listinfo/assp-test*
>> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>>
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! *http://sdm.link/slashdot*
>> <http://sdm.link/slashdot>
>> _______________________________________________
>> Assp-test mailing list
>> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
>> *https://lists.sourceforge.net/lists/listinfo/assp-test*
>> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>>
>>
>>
>> DISCLAIMER:
>> *******************************************************
>> This email and any files transmitted with it may be confidential, legally
>> privileged and protected in law and are intended solely for the use of the
>> individual to whom it is addressed.
>> This email was multiple times scanned for viruses. There should be no
>> known virus in this email!
>> *******************************************************
>>
>>
>> ------------------------------------------------------------
>> ------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Assp-test mailing list
>> Assp-test@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/assp-test
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
