Thanks for all this detail, Tero! I will have to digest it and reply
further later.
Barry
On Tue, Jun 13, 2023 at 5:34 PM Tero Kivinen <kivi...@iki.fi> wrote:
>
> Barry Leiba writes:
> > > DKIM only: ~99.5%
> > > DKIM + SPF: ~100%
> > > SPF only: ~100%
> >
> > That's interesting and disturbing if it remains consistent.
>
> The statistics I have are quite different. The failure rate is much
> bigger both in DKIM and SPF.
>
> Following statistics is random subset of emails going through iki.fi
> system, from last 30 days, consisting bit less than 4 million emails.
> Iki.fi is email forwarding service, so about 90% of those emails will
> fail SPF checks after iki.fi sends them forward. DKIM will go through
> unmodified, and we do not modify normal messages (spam messages might
> get tagged as spam depending on the members configuration), so 85.75%
> of emails will still have valid DKIM signature after passing iki.
>
> We do graylisting of blacklisted ip-addresses, thus spammers that do
> not work around graylisting are not part of the statistics.
>
> There is significant amount of mailing lists going through iki, and
> quickly checking that 1.58% of emails going through has spf-errors,
> dkim signers or similar with domain name in form of list.domain or
> lists.domain, so that will cause some of the SPF and DKIM failures.
> Note, that this only counts cases where the domain name was used in
> the verification and printed in the logs i.e., only in error cases.
>
> As we are using ARC, and we add ARC-Authentication-Results header to
> all emails as first step when they come in, and I used those headers
> to generate these statistics.
>
> First some generic statistics:
>
> Number of ARC-header levels
> ===========================
> 95.61% 3811208 1
> 3.83% 152487 2
> 0.44% 17711 3
> 0.09% 3586 4
> 0.01% 460 5
> 0.01% 349 6
> 0.01% 207 7
> 0.00% 36 8
> 0.00% 15 9
> 0.00% 1 10
>
> Mailer
> ======================
> 91.96% 3665744 MTA-v4
> 8.04% 320315 MTA-v6
> 0.00% 1 MSA
>
> So 3.83% of emails already had one ARC header, and 0.56% had more than
> one arc header, with exactly one email having 10
> ARC-Authentication-Results headers. Most of the emails do not have ARC
> headers.
>
> 92% of traffic came in using IPv4..
>
> Then lets compare DKIM, SPF, DMARC and ARC results
>
> DKIM summary results
> =========================
> 85.75% 3417541 pass
> 13.11% 522367 none
> 1.12% 44604 fail
> 0.02% 893 temperror
>
> SPF results
> =========================
> 86.50% 3447577 pass
> 8.78% 349947 none
> 1.89% 75137 softfail
> 1.18% 46913 permerror
> 1.12% 44553 fail
> 0.49% 19536 neutral
> 0.05% 2037 temperror
>
> DMARC results
> =========================
> 62.82% 1243393 pass
> 30.99% 613478 none
> 6.05% 119800 fail
> 0.08% 1485 temperror
> 0.06% 1244 permerror
>
> ARC results
> =========================
> 91.66% 160268 pass
> 8.34% 14584 reject
>
> As you can see 85.75% of incoming email was already signed by DKIM,
> and 86.5% of emails had SPF records that passed. So they both have
> about same amount if usage coming in to our servers.
>
> The difference is that only 1.14% of emails had errors (fail, or
> temperror) in their DKIM signatures (most of those were because the
> email was from the mailing list that modified the body, but did not
> generate new DKIM header), compared to the 4.24% of emails having SPF
> failures (softfail, permerror, fail or temperror). Meaning there were
> much more emails that failed SPF than DKIM. Even if we ignore the
> softfails, we still have about double the emails failing (2.35%).
>
> Note, that the dmarc and arc statistics are not from all of the
> emails, it only includes those which actually had DMARC or ARC
> information. For dmarc this was about 50%, and for ARC it was only
> 4.3% of all emails.
>
> Here are some statistics abut the DKIM processing and the error cases.
> 76.75% had one DKIM signature, and over 20% had more than one
> signature. Here is number of DKIM signatures and their results, i.e.,
> 22.22% of emails had two DKIM signatures both passing, and 0.34% had
> one signature that passed, and another that failed etc:
>
> DKIM results
> =======================================
> 62.67% 2497633 pass
> 22.22% 885372 pass,pass
> 13.06% 520332 none
> 1.04% 41477 fail
> 0.34% 13353 pass,fail
> 0.19% 7506 none,pass
> 0.15% 5910 pass,none
> 0.07% 2635 fail,fail
> 0.06% 2235 pass,pass,pass
> 0.05% 2034 none,none
> 0.03% 1296 pass,pass,pass,pass
> 0.03% 1026 pass,pass,fail
> 0.03% 1002 fail,pass
> 0.02% 892 temperror
> 0.02% 631 pass,fail,fail
> 0.01% 583 pass,none,none
> 0.01% 369 fail,fail,fail
> 0.01% 356 fail,fail,pass
> 0.01% 335 pass,pass,none
> 0.00% 86 pass,fail,fail,fail
> 0.00% 69 none,fail
> 0.00% 67 pass,fail,pass
> 0.00% 48 pass,pass,fail,fail
> 0.00% 27 temperror,pass
> 0.00% 26 fail,fail,none
> 0.00% 22 pass,temperror
> 0.00% 15 pass,pass,none,none
> 0.00% 10 none,pass,pass
> 0.00% 9 fail,fail,fail,fail
> 0.00% 7 pass,fail,none
> 0.00% 7 none,fail,fail
> 0.00% 7 fail,fail,fail,fail,none
> 0.00% 4 pass,none,pass
> 0.00% 4 fail,none
> 0.00% 4 pass,fail,fail,fail,fail
> 0.00% 3 fail,pass,pass
> 0.00% 2 pass,pass,pass,pass,pass,pass
> 0.00% 2 pass,none,fail
> 0.00% 2 pass,pass,pass,fail
> 0.00% 2 none,fail,pass
> 0.00% 1 temperror,temperror
> 0.00% 1 pass,pass,pass,pass,fail
> 0.00% 1 fail,fail,temperror
> 0.00% 1 pass,temperror,pass
> 0.00% 1 none,none,none
>
> The none,none,none cases etc are where it had 3 DKIM signatures but it
> could not find any DKIM records from the DNS, and was not able to
> verify the signatures.
>
> And here are reasons why dkim signature checking failed. The Invalid
> DKIM record actually results the dkim result to be none, but other
> errors result to the final result to be fail. As you can see there is
> significant part where the body hash did not verify (most likely
> because this is coming from mailing list). This only includes those
> emails where there was no passing DKIM signature at all.
>
> DKIM failures
> ================================================================
> 36.34% 26619 invalid DKIM record
> 36.28% 26577 body hash did not verify
> 20.34% 14900 headers rsa verify failed
> 2.78% 2034 invalid DKIM record,invalid DKIM record
> 1.62% 1186 headers rsa verify failed,headers rsa verify
> failed
> 1.62% 1185 body hash did not verify,body hash did not
> verify
> 0.49% 360 body hash did not verify,body hash did not
> verify,body hash did not verify
> 0.30% 218 headers rsa verify failed,headers eddsa verify
> failed
> 0.09% 65 invalid DKIM record,body hash did not verify
> 0.05% 37 headers rsa verify failed,body hash did not
> verify
> 0.04% 26 body hash did not verify,body hash did not
> verify,invalid DKIM record
> 0.01% 9 headers eddsa verify failed,headers rsa verify
> failed
> 0.01% 9 body hash did not verify,body hash did not
> verify,body hash did not verify,body hash did
> not verify
> 0.01% 7 body hash did not verify,body hash did not
> verify,body hash did not verify,body hash did
> not verify,invalid DKIM record
> 0.01% 6 invalid DKIM record,body hash did not
> verify,body hash did not verify
> 0.01% 4 headers rsa verify failed,headers rsa verify
> failed,body hash did not verify
> 0.01% 4 invalid DKIM record,headers rsa verify failed
> 0.00% 3 headers rsa verify failed,headers rsa verify
> failed,headers rsa verify failed
> 0.00% 2 headers rsa verify failed,invalid DKIM record
> 0.00% 2 headers rsa verify failed,body hash did not
> verify,body hash did not verify
> 0.00% 2 body hash did not verify,invalid DKIM record
> 0.00% 1 invalid DKIM record,invalid DKIM
> record,invalid DKIM record
> 0.00% 1 body hash did not verify,headers rsa verify
> failed
> 0.00% 1 invalid DKIM record,headers rsa verify
> failed,headers rsa verify failed
>
> SPF failures show that it is not that big difference whether you use
> IPv4, or IPv6, as this matches the generic use of IP protocols for
> these incoming emails:
>
> SPF failures
> ==============================================================
> 92.71% 41307 MTA-v4: domain of x@y does not designate ipxxx
> as permitted sender
> 7.29% 3246 MTA-v6: domain of x@y does not designate ipxxx
> as permitted sender
>
> For DMARC failures there is quite a large number of those which do not
> have SPF or DKIM. I do not really known what I should interpret from
> those other errors for DMARC.
>
> DMARC failures
> ============================================================
> 52.53% 62925 No valid SPF, No valid DKIM
> 32.97% 39504 SPF not aligned (relaxed), DKIM not aligned (relaxed)
> 5.41% 6486 SPF not aligned (relaxed), No valid DKIM
> 3.49% 4186 No valid SPF
> 2.68% 3213 SPF not aligned (relaxed)
> 2.07% 2484 No valid SPF, DKIM not aligned (relaxed)
> 0.25% 297 SPF not aligned (strict), DKIM not aligned (strict)
> 0.21% 256 SPF not aligned (relaxed), DKIM not aligned (strict)
> 0.17% 207 SPF not aligned (strict)
> 0.09% 106 SPF not aligned (strict), No valid DKIM
> 0.08% 100 SPF not aligned (strict), DKIM not aligned (relaxed)
> 0.03% 36 No valid SPF, DKIM not aligned (strict)
>
> For ARC there is quite big number of signature check failures, I am
> not actually sure whether that is because there is no key to be found
> or what is the issue.
>
> ARC failures
> ===========================================================
> 80.36% 11720 "signature check failed: fail, {[1] = sig:xxx:reject}"
> 6.37% 929 "cv is fail on i=4"
> 6.31% 920 "cv is fail on i=2"
> 3.73% 544 "seal check failed: fail, {[1] = sig:xxx:reject}"
> 1.89% 275 "cv is none on i=2"
> 0.80% 116 "signature check failed: fail, {[1] =
> sig:xxx:dns request to xxx
> 0.52% 76 "cv is fail on i=3"
> 0.02% 3 "seal check failed: fail, {[1] = sig:xxx:dns
> request to xxx
> 0.01% 1 unknown
>
>
> Summary: Looking at the data there is much more SPF related failures
> than DKIM related failures, and as I said 90% of these emails WILL
> FAIL SPF checks when iki.fi will forward them to their final
> destination (only those that say +all or do not publish SPF record
> will survive), while the DKIM records still are correct.
>
> We have several cases where final email domain where the user asks us
> to forward his email is only using SPF, thus we simply ask them to
> switch to someone who does email properly and uses DKIM too...
>
> If the DMARCv2 would mandate support of DKIM and would get rid of the
> SPF checks completely then hopefully more people would actually start
> using DKIM also in the verification. It is quite widely already used
> in the generation of the messages.
>
> Of course this is selected data-set as if out user find out he can't
> use his iki.fi address for certain service as it does not do DKIM, and
> his/her final destination checks SPF, he/she will not use his iki.fi
> address in those places or he/she changes his email mailbox provider
> (which is easy to do if all your emails go through iki, you simply
> change the forward to go to your new address, and hour later
> all your emails go there :-)
> --
> kivi...@iki.fi
_______________________________________________
dmarc mailing list
dmarc@ietf.org
https://www.ietf.org/mailman/listinfo/dmarc
