Re: [dmarc-ietf] DMARC2 & SPF Dependency Removal
This topic raised a question, at least in my mind, whether DKIM signing algorithms are subject to random failures. If random failures occur, they could be blamed on either the sender algorithm or the receiver algorithm. The question can be assessed on incoming messages, using authentication results, or on outgoing messages, using aggregate reports. My environment has a single MX performing DKIM checking, with essentially zero mailing list traffic. I also have a single MTA performing DKIM signing. The signing server is a different software implementation than the checking server. For outgoing messages, I see no evidence of random failures. All of the reported DKIM failures appear to have explanations. For incoming messages, I defined a unique configuration as the tuple of ( Helo Domain, MailFrom domain, and From domain ). Then I looked at verification percentages for signed messages from each tuple.The results were interesting: - 92% of tuples, sending 89% of all signed messages, had 100% verification rates - 5% of tuples, sending 2% of all signed messages, had 0% verification rates - 3% of tuples, sending 9% of all signed messages, had a verification rate between these limits. - Of all messages with DKIM failures, 85% were authenticated using aligned SPF PASS. These statistics are based on 100% of incoming messages, regardless of subsequent disposition. My conclusions: - SPF is a necessary supplement to DKIM. - DKIM can be reliable: Most senders and receivers have 100% reliable DKIM implementations. - Senders with 100% failure rates appear to be playing games with DKIM. Some of these may already be on my block list. The rest should be reviewed to see if they should be added to my block list. - The 5% with inconsistent results need further investigation. Perhaps a server farm has one server that is generating wrong signatures. Doug Foster On Tue, Jun 13, 2023 at 5:34 PM Tero Kivinen wrote: > Barry Leiba writes: > > > DKIM only: ~99.5% > > > DKIM + SPF: ~100% > > > SPF only: ~100% > > > > That's interesting and disturbing if it remains consistent. > > The statistics I have are quite different. The failure rate is much > bigger both in DKIM and SPF. > > Following statistics is random subset of emails going through iki.fi > system, from last 30 days, consisting bit less than 4 million emails. > Iki.fi is email forwarding service, so about 90% of those emails will > fail SPF checks after iki.fi sends them forward. DKIM will go through > unmodified, and we do not modify normal messages (spam messages might > get tagged as spam depending on the members configuration), so 85.75% > of emails will still have valid DKIM signature after passing iki. > > We do graylisting of blacklisted ip-addresses, thus spammers that do > not work around graylisting are not part of the statistics. > > There is significant amount of mailing lists going through iki, and > quickly checking that 1.58% of emails going through has spf-errors, > dkim signers or similar with domain name in form of list.domain or > lists.domain, so that will cause some of the SPF and DKIM failures. > Note, that this only counts cases where the domain name was used in > the verification and printed in the logs i.e., only in error cases. > > As we are using ARC, and we add ARC-Authentication-Results header to > all emails as first step when they come in, and I used those headers > to generate these statistics. > > First some generic statistics: > > Number of ARC-header levels > === > 95.61% 3811208 1 > 3.83% 152487 2 > 0.44% 17711 3 > 0.09% 35864 > 0.01% 460 5 > 0.01% 349 6 > 0.01% 207 7 > 0.00% 36 8 > 0.00% 15 9 > 0.00% 1 10 > > Mailer > == > 91.96% 3665744 MTA-v4 > 8.04% 320315 MTA-v6 > 0.00% 1 MSA > > So 3.83% of emails already had one ARC header, and 0.56% had more than > one arc header, with exactly one email having 10 > ARC-Authentication-Results headers. Most of the emails do not have ARC > headers. > > 92% of traffic came in using IPv4.. > > Then lets compare DKIM, SPF, DMARC and ARC results > > DKIM summary results > = > 85.75% 3417541 pass > 13.11% 522367 none > 1.12% 44604 fail > 0.02% 893 temperror > > SPF results > = > 86.50% 3447577 pass > 8.78% 349947 none > 1.89% 75137 softfail > 1.18% 46913 permerror > 1.12% 44553 fail > 0.49% 19536 neutral > 0.05% 2037temperror > > DMARC results > = > 62.82% 1243393 pass > 30.99% 613478 none > 6.05% 119800 fail >
Re: [dmarc-ietf] DMARC2 & SPF Dependency Removal
Thanks for all this detail, Tero! I will have to digest it and reply further later. Barry On Tue, Jun 13, 2023 at 5:34 PM Tero Kivinen wrote: > > Barry Leiba writes: > > > DKIM only: ~99.5% > > > DKIM + SPF: ~100% > > > SPF only: ~100% > > > > That's interesting and disturbing if it remains consistent. > > The statistics I have are quite different. The failure rate is much > bigger both in DKIM and SPF. > > Following statistics is random subset of emails going through iki.fi > system, from last 30 days, consisting bit less than 4 million emails. > Iki.fi is email forwarding service, so about 90% of those emails will > fail SPF checks after iki.fi sends them forward. DKIM will go through > unmodified, and we do not modify normal messages (spam messages might > get tagged as spam depending on the members configuration), so 85.75% > of emails will still have valid DKIM signature after passing iki. > > We do graylisting of blacklisted ip-addresses, thus spammers that do > not work around graylisting are not part of the statistics. > > There is significant amount of mailing lists going through iki, and > quickly checking that 1.58% of emails going through has spf-errors, > dkim signers or similar with domain name in form of list.domain or > lists.domain, so that will cause some of the SPF and DKIM failures. > Note, that this only counts cases where the domain name was used in > the verification and printed in the logs i.e., only in error cases. > > As we are using ARC, and we add ARC-Authentication-Results header to > all emails as first step when they come in, and I used those headers > to generate these statistics. > > First some generic statistics: > > Number of ARC-header levels > === > 95.61% 3811208 1 > 3.83% 152487 2 > 0.44% 17711 3 > 0.09% 35864 > 0.01% 460 5 > 0.01% 349 6 > 0.01% 207 7 > 0.00% 36 8 > 0.00% 15 9 > 0.00% 1 10 > > Mailer > == > 91.96% 3665744 MTA-v4 > 8.04% 320315 MTA-v6 > 0.00% 1 MSA > > So 3.83% of emails already had one ARC header, and 0.56% had more than > one arc header, with exactly one email having 10 > ARC-Authentication-Results headers. Most of the emails do not have ARC > headers. > > 92% of traffic came in using IPv4.. > > Then lets compare DKIM, SPF, DMARC and ARC results > > DKIM summary results > = > 85.75% 3417541 pass > 13.11% 522367 none > 1.12% 44604 fail > 0.02% 893 temperror > > SPF results > = > 86.50% 3447577 pass > 8.78% 349947 none > 1.89% 75137 softfail > 1.18% 46913 permerror > 1.12% 44553 fail > 0.49% 19536 neutral > 0.05% 2037temperror > > DMARC results > = > 62.82% 1243393 pass > 30.99% 613478 none > 6.05% 119800 fail > 0.08% 1485temperror > 0.06% 1244permerror > > ARC results > = > 91.66% 160268 pass > 8.34% 14584 reject > > As you can see 85.75% of incoming email was already signed by DKIM, > and 86.5% of emails had SPF records that passed. So they both have > about same amount if usage coming in to our servers. > > The difference is that only 1.14% of emails had errors (fail, or > temperror) in their DKIM signatures (most of those were because the > email was from the mailing list that modified the body, but did not > generate new DKIM header), compared to the 4.24% of emails having SPF > failures (softfail, permerror, fail or temperror). Meaning there were > much more emails that failed SPF than DKIM. Even if we ignore the > softfails, we still have about double the emails failing (2.35%). > > Note, that the dmarc and arc statistics are not from all of the > emails, it only includes those which actually had DMARC or ARC > information. For dmarc this was about 50%, and for ARC it was only > 4.3% of all emails. > > Here are some statistics abut the DKIM processing and the error cases. > 76.75% had one DKIM signature, and over 20% had more than one > signature. Here is number of DKIM signatures and their results, i.e., > 22.22% of emails had two DKIM signatures both passing, and 0.34% had > one signature that passed, and another that failed etc: > > DKIM results > === > 62.67% 2497633 pass > 22.22% 885372 pass,pass > 13.06% 520332 none > 1.04% 41477 fail > 0.34% 13353 pass,fail > 0.19% 7506none,pass > 0.15% 5910pass,none > 0.07% 2635fail,fail > 0.06% 2235pass,pass,pass > 0.05% 2034
Re: [dmarc-ietf] DMARC2 & SPF Dependency Removal
Barry Leiba writes: > > DKIM only: ~99.5% > > DKIM + SPF: ~100% > > SPF only: ~100% > > That's interesting and disturbing if it remains consistent. The statistics I have are quite different. The failure rate is much bigger both in DKIM and SPF. Following statistics is random subset of emails going through iki.fi system, from last 30 days, consisting bit less than 4 million emails. Iki.fi is email forwarding service, so about 90% of those emails will fail SPF checks after iki.fi sends them forward. DKIM will go through unmodified, and we do not modify normal messages (spam messages might get tagged as spam depending on the members configuration), so 85.75% of emails will still have valid DKIM signature after passing iki. We do graylisting of blacklisted ip-addresses, thus spammers that do not work around graylisting are not part of the statistics. There is significant amount of mailing lists going through iki, and quickly checking that 1.58% of emails going through has spf-errors, dkim signers or similar with domain name in form of list.domain or lists.domain, so that will cause some of the SPF and DKIM failures. Note, that this only counts cases where the domain name was used in the verification and printed in the logs i.e., only in error cases. As we are using ARC, and we add ARC-Authentication-Results header to all emails as first step when they come in, and I used those headers to generate these statistics. First some generic statistics: Number of ARC-header levels === 95.61% 3811208 1 3.83% 152487 2 0.44% 17711 3 0.09% 35864 0.01% 460 5 0.01% 349 6 0.01% 207 7 0.00% 36 8 0.00% 15 9 0.00% 1 10 Mailer == 91.96% 3665744 MTA-v4 8.04% 320315 MTA-v6 0.00% 1 MSA So 3.83% of emails already had one ARC header, and 0.56% had more than one arc header, with exactly one email having 10 ARC-Authentication-Results headers. Most of the emails do not have ARC headers. 92% of traffic came in using IPv4.. Then lets compare DKIM, SPF, DMARC and ARC results DKIM summary results = 85.75% 3417541 pass 13.11% 522367 none 1.12% 44604 fail 0.02% 893 temperror SPF results = 86.50% 3447577 pass 8.78% 349947 none 1.89% 75137 softfail 1.18% 46913 permerror 1.12% 44553 fail 0.49% 19536 neutral 0.05% 2037temperror DMARC results = 62.82% 1243393 pass 30.99% 613478 none 6.05% 119800 fail 0.08% 1485temperror 0.06% 1244permerror ARC results = 91.66% 160268 pass 8.34% 14584 reject As you can see 85.75% of incoming email was already signed by DKIM, and 86.5% of emails had SPF records that passed. So they both have about same amount if usage coming in to our servers. The difference is that only 1.14% of emails had errors (fail, or temperror) in their DKIM signatures (most of those were because the email was from the mailing list that modified the body, but did not generate new DKIM header), compared to the 4.24% of emails having SPF failures (softfail, permerror, fail or temperror). Meaning there were much more emails that failed SPF than DKIM. Even if we ignore the softfails, we still have about double the emails failing (2.35%). Note, that the dmarc and arc statistics are not from all of the emails, it only includes those which actually had DMARC or ARC information. For dmarc this was about 50%, and for ARC it was only 4.3% of all emails. Here are some statistics abut the DKIM processing and the error cases. 76.75% had one DKIM signature, and over 20% had more than one signature. Here is number of DKIM signatures and their results, i.e., 22.22% of emails had two DKIM signatures both passing, and 0.34% had one signature that passed, and another that failed etc: DKIM results === 62.67% 2497633 pass 22.22% 885372 pass,pass 13.06% 520332 none 1.04% 41477 fail 0.34% 13353 pass,fail 0.19% 7506none,pass 0.15% 5910pass,none 0.07% 2635fail,fail 0.06% 2235pass,pass,pass 0.05% 2034none,none 0.03% 1296pass,pass,pass,pass 0.03% 1026pass,pass,fail 0.03% 1002fail,pass 0.02% 892 temperror 0.02% 631 pass,fail,fail 0.01% 583 pass,none,none 0.01% 369 fail,fail,fail 0.01% 356 fail,fail,pass 0.01% 335
