Okay, "regardless" was a bit much, I agree with the reasons you have listed
for DKIM skipping and figured that was the case for all. What I'm seeing
with 18103 is mail that scores with hmm/bayes only NOT being noprocessed
due to a DKIMNPAddress hit. We know it's a good signature because analyze
says it's a match, but for whatever reason ASSP isn't no processing it
after a hmm or possibily other scoring threthold match. Unfortunately, I
haven't seen an example since I made logging more verbose, or I'd have
provide more info. Can you confirm that DKIMNPAddress is supposed to run
and results be honored if a message's score already exceeds the reject
threshold AND none of the exceptions you previously listed are true?
I'll take a look at the graph to see what I can learn.
You definitely peaked my interest when you mentioned the disclaimers file
previously.
About 70% of our staff uses signatures, generally unique to them. We're
small enough that I guess I could start compiling a list for the
disclaimers file. I assume that grabbing the signatures right from mail
text files in the corpus would be okay? That would pick up the html markup
they tend to use. Yes?
At least 50% (guessing) of our legitimate *inbound* mail has signatures and
disclaimers on them too. I can't see manually maintaining a disclaimers
file with all of them in it. Which is better, having some inbound
signatures in the file or NONE? I don't want to create a bias against
signatures that aren't in the disclaimer file.
On Tue, Apr 17, 2018 at 5:17 AM, Thomas Eckardt <thomas.ecka...@thockar.com>
wrote:
> >If we put the address on the DKIMNPAddress list, shouldn't it honor that
> regardless of anything else?
>
> REGARDLESS ???
>
> No . this makes no sense.
>
> The pre-DKIM check is skipped for (18103):
>
> invalidSenderDomain (no valid TLD)
> whitelisted
> acceptAllMail
> bounce mails
> outgoing mails
> RWL high trust
> contentOnly
> noprocessing (except noprocessing by size)
> noDKIMAddresses
> noDKIMIP
>
> -----------------------
> >don't know if a change from 0.001 to 0.005 would be significant or make
> sense,
>
> Just use the 'Bayes/HMM confidence' graph. This requires
> 'enableGraphStats' to be enabled.
>
> >Then there's a ton of spam messages also advertising (fake) handbags and
> they're often using the SAME domain and sometimes identical from address as
> the legit mails.
>
> ASSP has alot of features to indentify the correctness of the origin of an
> email. And there is (IMHO) a big difference between maliciouse spam and
> normal (not dangerous) mails you don't want to get. The later are often
> problematic. Most times it is better to let them pass, than to block
> important mails.
>
> -----------------------
>
> The next version will have an improvement for HMM and Bayesian.
> An real problem may become disclaimers and privat and corporate signatues.
> They are always added to outgoing mails, but also to spam reports. They can
> be found in most of the answers to our mails. And for example, in my case,
> they may be added by spammers to there spam mail. Nobody can say, how the
> occurrence of such a disclaimer will affect the HMM and Bayesian results.
> It may possible, that these results differs from day to day, or block
> always good mails, or.let spam pass.
>
> The only way to prevent such "wild" results is to remove the disclaimers,
> before the rebuildspamdb task builds the spamdb and HMMdb. I use this code
> for a month now and I'm really happy with the result.
>
> Thomas
>
>
>
>
>
> Von: "K Post" <nntp.p...@gmail.com>
> An: "ASSP development mailing list" <assp-test@lists.sourceforge.
> net>
> Datum: 16.04.2018 17:43
> Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as
> expected, but some messages still processed as spam?
> ------------------------------
>
>
>
> I'm always correcting HMM/Bayes by reviewing the block report on a daily
> basis and reporting. It's a horrible task that I dread, but it is worth
> it. I have the score set to 50, and 50 as the threshold for rejection.
> We've tried lower, but too much spam is only tagged *solely* due to a HMM
> hit and slips through.
>
> I never changed baysConf from the original 0.001 with baysprobablity to
> 0.6, but I do have a 1.000 corpus norm. It's certainly a mature
> installation (15+ years). The issue I'm having is legit messages, say a
> message advertising handbags from a reputable seller that one of our staff
> buys from (over their lunch hour of course!!). Then there's a ton of spam
> messages also advertising (fake) handbags and they're often using the SAME
> domain and sometimes identical from address as the legit mails. HMM/Bayes
> is rightfully biased against all handbag email. You're saying that
> increasing baysConf will help the legit ones get through but still will
> block the spam ones (with almost identical content)? I've read through
> the gui for baysConf, but the problem is that which I understand HMM and
> Bayes from a concept standpoint, the calculations aren't something I
> understand, so I don't dare change the 0.0001 threshold without real
> guidance from you. I understand 1 is the max, but don't know if increases
> linearly or exponentially change levels, don't know if a change from 0.001
> to 0.005 would be significant or make sense, etc.
>
> My hopes with the dkim np was to let hmm spammy mail through if it's a
> dkim match. Ignore all other results, if the DKIM is good, just let it
> through was my thinking. I know handbag seller X sends ad mail that's DKIM
> signed, but I don't know when IP they'll come from (or the IP is mailing
> service that I don't want to blanket allow). This has been quite
> successful with a whole lot of mail. I've become spoiled, now I want it to
> work for all mail when there's a DKIM match to the NP list.
>
> You raise another a good point about the 2 kind of DKIM checks, thanks for
> the reminder. Does it make any sense to *always* have ASSP do the second
> one and if it validates and matches dkimNPaddress or dkimWLadderss, process
> solely based on that match? For example, HMM might hit before the full
> body validation of DKIM, but so what? If we put the address on the
> DKIMNPAddress list, shouldn't it honor that regardless of anything else?
> NO processing, as I'm interpreting it, should mean, well, NONE, so if other
> hits have happened, they're ignored because we said don't process.
>
> I will temporarily change the logging level for a bit and see if I can
> figure out why dkim isn't being done for these messages, but I'm guessing
> that it's by design.
>
> As always, thanks
> Ken
>
>
> On Mon, Apr 16, 2018 at 2:35 AM, Thomas Eckardt <
> *thomas.ecka...@thockar.com* <thomas.ecka...@thockar.com>> wrote:
> >I'm still seeing scenarios where analyze shows a DKIM NP match, but the
> message is still going to spam based on score from HMM.
>
> The best solution is to correct the corpus, to get better HMM results. You
> may also decrease the scoring points for HMM and/or Bayesian. If the corpus
> is corrected and the corpusnorm is ~ 1.0 , 'baysConf' will increase
> detection correctness.
>
>
> If assp receives a mail it acts as a statemachine. If and how a check is
> done, depends on the previouse reached states.
>
> Using the analyzer, assp acts procedural. Every check is done without any
> state dependency. This is done, to be able to show every feature match.
> The analyzer uses the current configuration, hashes, lists and databases.
> So it may be normal to get different results compared to the real mail
> processing loggings, if a mail is analyzed.
>
> >DKIM NP match:
>
> The analyzer checks DKIM without any dependency and shows all results.
>
> But, if a mail is received, the DKIM check depends on several previouse
> states
> DKIM NP is a resulting state of the DKIM check. So - if any of the
> previouse (DKIM depdency) states prevents the DKIM check, there will be no
> DKIM
> (DKIM NP) result. The mail will be processed the same way, as it was not
> DKIM signed.
> Every state, that depends on DKIM NP will not be reached.
>
> You should also remember, that assp use two DKIM checks. The full DKIM
> check, which requires the full mail to be received - the results of this
> check affects only the Plugin Level 2 (full mail) checks.
> And the DKIM-Pre-Check - which is done after the MIME header is received
> and if 'DKIMCacheInterval' is enabled. The results (states) of this check
> affects most of the header checks and all body and full mail checks.
>
> If you expect but miss a match for 'DKIMWLAddresses' or 'DKIMNPAddresses'
> , you may increase the logging level (ValidateSenderLog, SessionLog,
> ipmatchLogging, slmatchLogging).
>
> Thomas
>
>
>
>
> Von: "K Post" <*nntp.p...@gmail.com* <nntp.p...@gmail.com>>
> An: "ASSP development mailing list" <
> *assp-test@lists.sourceforge.net* <assp-test@lists.sourceforge.net>>
> Datum: 15.04.2018 23:21
> Betreff: Re: [Assp-test] Analyze shows DKIMNPAddress match as
> expected, but some messages still processed as spam?
> ------------------------------
>
>
>
> I'm still seeing scenarios where analyze shows a DKIM NP match, but the
> message is still going to spam based on score from HMM.
>
> On Mon, Apr 9, 2018 at 12:19 PM, K Post <*nntp.p...@gmail.com*
> <nntp.p...@gmail.com>> wrote:
> cheers.
>
> On Mon, Apr 9, 2018 at 3:36 AM, Thomas Eckardt <
> *thomas.ecka...@thockar.com* <thomas.ecka...@thockar.com>> wrote:
> If assp has modified the original mail header (changed foreign X-ASSP- or
> removed cc,bcc or ...) the DKIM check is skipped and also DKIMNPAddress and
> DKIMWLAddress.
>
> The next version will try to check, if removed or changed headers are
> protected by a DKIM signature and do the check, if this is not the case.
>
> Thomas
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! *http://sdm.link/slashdot*
> <http://sdm.link/slashdot>
> _______________________________________________
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! *http://sdm.link/slashdot*
> <http://sdm.link/slashdot>
> _______________________________________________
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! *http://sdm.link/slashdot*
> <http://sdm.link/slashdot>
> _______________________________________________
> Assp-test mailing list
> *Assp-test@lists.sourceforge.net* <Assp-test@lists.sourceforge.net>
> *https://lists.sourceforge.net/lists/listinfo/assp-test*
> <https://lists.sourceforge.net/lists/listinfo/assp-test>
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
>
>
>
> DISCLAIMER:
> *******************************************************
> This email and any files transmitted with it may be confidential, legally
> privileged and protected in law and are intended solely for the use of the
> individual to whom it is addressed.
> This email was multiple times scanned for viruses. There should be no
> known virus in this email!
> *******************************************************
>
>
> ------------------------------------------------------------
> ------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Assp-test mailing list
> Assp-test@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/assp-test
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Assp-test mailing list
Assp-test@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/assp-test
