Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Sun 16/Jun/2024 16:38:48 +0200 Tobias Fiebig via mailop wrote: You'd need several domains, all having a rua= pointing to you. I'd donate a (sub) domain to that effort. I'm donating a couple of domains to Project Honey Pot. Unlike that project, however, in this case donated domains will have to actively send replies. Actually LUA records with powerdns should suffice; Similar to what is already being done for the DNS tests: dig MX sometext.uniq.measurement.email-security-scans.org \ @dns.measurement.email-security-scans.org So, creating something like _dmarc..dmarcfail.measurement.email-security-scans.org, and only sending the mails after at least N mails for the test have been successfully received. In theory, that's correct. However, we'd need both domains matching the PSL as well as domains matching tree walks. I'm not familiar with PowerDNS, but clients will query their usual DNS servers and resolve. Setting up domains correctly won't be easy. _dmarc.sometext.uniq.measurement.email-security-scans.org -> v=spf1 mx ip4:195.191.197.88 ip6:2a06:d1c0:dead:3::88 -all _dmarc.uniq.measurement.email-security-scans.org -> v=spf1 mx ip4:195.191.197.88 ip6:2a06:d1c0:dead:3::88 -all _dmarc.measurement.email-security-scans.org -> v=spf1 mx ip4:195.191.197.88 ip6:2a06:d1c0:dead:3::88 -all _dmarc.email-security-scans.org -> v=DMARC1; p=reject; rua=mailto:dm...@aperture-labs.org There will also be confirmation RRs for rua= at external domains (some will have to not be confirmed, to check for that check). Some subdomains will have DMARC records, some not. Perhaps, some mails can be sent from real IPs, if their owners are not afraid to be blacklisted. I agree the same effect can be obtained by creating lots of subdomains, but that wont work for filters still using the PSL. In addition, having domain donors might boost cooperation. Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, > You'd need several domains, all having a rua= pointing to you. I'd > donate a (sub) domain to that effort. I'm donating a couple of > domains to Project Honey Pot. Unlike that project, however, in this > case donated domains will have to actively send replies. Actually LUA records with powerdns should suffice; Similar to what is already being done for the DNS tests: dig MX sometext.uniq.measurement.email-security-scans.org \ @dns.measurement.email-security-scans.org So, creating something like _dmarc..dmarcfail.measurement.email-security-scans.org, and only sending the mails after at least N mails for the test have been successfully received. > I'm tempted, although Python is not my forté. No worries. :-) With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Sat 15/Jun/2024 18:27:15 +0200 Tobias Fiebig via mailop wrote: Do reports received at dm...@aperture-labs.org contribute to the output of email-security-scans? No, of course not; esec.o is tests-are-atomic. Technically I _could_ (or rather: should) try to implement something similar to what I am already doing for the TLS-RPT test for DMARC _sending_ as well (currently, I am only testing deliverability of RUA/RUF). TLS-RPT reports seem to be more useful than DMARC ones. I, for one, forward them to a daily-seen folder when they contain failed connections, which doesn't happen every day. (In some cases, I remove the blocked IP from the firewall.) DMARC reports have a plethora of failures every day, due to mailing lists. Sporadically, I take a look at them, but not always, and never sum them up. However, I skipped on that initially, because: - It is more about receiving than sending (and esec.o was initially sending focused) - It is difficult to fill in an identifier there; Technically, I could, e.g., send from unique domains (difficult, as some large domains are now blocked for the startup mail and have a web-only-flow; Also, deliverability for that is likely low(er)), or add something where you can request the DMARC test in addition when you submitted the some test results. Sending DKIM invalid mails for the test should further reduce the noise (while still triggering reports). However, that would have to be implemented, and I am currently struggling with the very stupid idea somebody had some when that a day should just have 24h. Some hold DKIM reports are to be delivered just around midnight. You'd need several domains, all having a rua= pointing to you. I'd donate a (sub) domain to that effort. I'm donating a couple of domains to Project Honey Pot. Unlike that project, however, in this case donated domains will have to actively send replies. Similarly, it would kind of make sense to maybe tie in the internet.nl suite and display/integrate those results as well. But again, time. So, somewhat related: If somebody suffers from an abundance of time, is kind of good with python, mail, and PHP... and would like to work on what is objectively likely some of the worst code they have ever seen... drop me a line. ;-) I'm tempted, although Python is not my forté. Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, > There is a /human_result/ in DMARC reports. It would be a good idea > to check whether there are (new) messages there, and bring it to > human attention in case. Not doing so deprives the field of its > /human/ attribute. Ah, makes sense; Still, just noting that most analysis software does not display it. > just bespeaks the quality of report consumer software. Not arguing with that. > Do reports received at dm...@aperture-labs.org contribute to the > output of email-security-scans? No, of course not; esec.o is tests-are-atomic. Technically I _could_ (or rather: should) try to implement something similar to what I am already doing for the TLS-RPT test for DMARC _sending_ as well (currently, I am only testing deliverability of RUA/RUF). However, I skipped on that initially, because: - It is more about receiving than sending (and esec.o was initially sending focused) - It is difficult to fill in an identifier there; Technically, I could, e.g., send from unique domains (difficult, as some large domains are now blocked for the startup mail and have a web-only-flow; Also, deliverability for that is likely low(er)), or add something where you can request the DMARC test in addition when you submitted the some test results. Sending DKIM invalid mails for the test should further reduce the noise (while still triggering reports). However, that would have to be implemented, and I am currently struggling with the very stupid idea somebody had some when that a day should just have 24h. Similarly, it would kind of make sense to maybe tie in the internet.nl suite and display/integrate those results as well. But again, time. So, somewhat related: If somebody suffers from an abundance of time, is kind of good with python, mail, and PHP... and would like to work on what is objectively likely some of the worst code they have ever seen... drop me a line. ;-) With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Thu 13/Jun/2024 11:14:04 +0200 Tobias Fiebig via mailop wrote: What if spamd, while authenticating a malformed signature, notifies the error in DMARC report? Would that "fix" spamd? Would be somewhat pointless, though; From the top of my head I am not sure whether there is a fitting field; And apart from that, I doubt anyone would read those mails (or, rather, parse it into the DMARC webif in use and display it in a way anyone would notice). Also, there should be enough implementations out there that do not verify DKIM there; So you _should_ already see the spike in non-passing DKIM messages, incl. from your own senders. There is a /human_result/ in DMARC reports. It would be a good idea to check whether there are (new) messages there, and bring it to human attention in case. Not doing so deprives the field of its /human/ attribute. Yes, bounces are certainly more noticeable. However, they only happen on p=reject. Otherwise, failure spikes being more noticeable than human results just bespeaks the quality of report consumer software. Do reports received at dm...@aperture-labs.org contribute to the output of email-security-scans? Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, > What if spamd, while authenticating a malformed signature, notifies > the error in DMARC report? Would that "fix" spamd? Would be somewhat pointless, though; From the top of my head I am not sure whether there is a fitting field; And apart from that, I doubt anyone would read those mails (or, rather, parse it into the DMARC webif in use and display it in a way anyone would notice). Also, there should be enough implementations out there that do not verify DKIM there; So you _should_ already see the spike in non-passing DKIM messages, incl. from your own senders. With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Sun 09/Jun/2024 02:43:38 +0200 Tobias Fiebig via mailop wrote: On Sat, 2024-06-08 at 12:36 +0200, Alessandro Vesely via mailop wrote: ... (unless bugs in email-security-scans are just decorative.) Which ones exactly? What if spamd, while authenticating a malformed signature, notifies the error in DMARC report? Would that "fix" spamd? Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Sat, 2024-06-08 at 12:36 +0200, Alessandro Vesely via mailop wrote: > ... (unless bugs in email-security-scans are just decorative.) Which ones exactly? With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Sat 08/Jun/2024 13:28:30 +0200 Slavko via mailop wrote: Dňa 8. júna 2024 10:36:44 UTC používateľ Alessandro Vesely via mailop napísal: Yes, as it seems Tobias is going to file a bug against rspamd, I presume you are going to somehow fix it (unless bugs in email-security-scans are just decorative.) I have nothing to do with email-security-scans, and very little to do with rspamd (i only use it)... Thus no, i will not (be able to) fix anything about this ;-) Oops, I must have meant Vsevolod. Sorry, these Slavic names confused me... Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Dňa 8. júna 2024 10:36:44 UTC používateľ Alessandro Vesely via mailop napísal: >Yes, as it seems Tobias is going to file a bug against rspamd, I presume you >are going to somehow fix it (unless bugs in email-security-scans are just >decorative.) I have nothing to do with email-security-scans, and very little to do with rspamd (i only use it)... Thus no, i will not (be able to) fix anything about this ;-) regards -- Slavko https://www.slavino.sk/ ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Fri 07/Jun/2024 17:03:00 +0200 Slavko via mailop wrote: Dňa 7. júna 2024 14:37:24 UTC používateľ Alessandro Vesely via mailop napísal: If I were Slavko I'd fix rspamd by adding bug reporting (if it's not already there) rather than removing 2047-decoding. Are you sure, that you did mean me? Yes, as it seems Tobias is going to file a bug against rspamd, I presume you are going to somehow fix it (unless bugs in email-security-scans are just decorative.) The fix I propose requires they to also consider DMARC reports, which would be cool. I was just curious about IDNA syntax in this case... Just got this: Authentication-Results: mx.google.com; dkim=pass header.i=@foà.it header.s=gamma header.b=Akp+6zYW; spf=pass (google.com: domain of postmaster@foà.it designates 94.198.96.74 as permitted sender) Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
It appears that Tobias Fiebig via mailop said: > >Moin, > >On Thu, 2024-06-06 at 12:44 +1000, Viktor Dukhovni via mailop wrote: >> The distinction is essential, because it would be a terrible mistake >> to, for example, RFC2047-encode the "mailbox" construct in "From", >> "To", ... headers. An RFC2047-ignorant MUA or MTA can still >> correctly decode the addresses in those headers without caring about >> the display name encoding. > >Kind of a point; However, if I got John correctly, an SMTPUTF8 mail >having to go through a system that does not support it should simply >bounce? Right. That's the 5.6.9 extended status message. R's, John ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Dňa 7. júna 2024 14:37:24 UTC používateľ Alessandro Vesely via mailop napísal: >If I were Slavko I'd fix rspamd by adding bug reporting (if it's not already >there) rather than removing 2047-decoding. Are you sure, that you did mean me? I was just curious about IDNA syntax in this case... regards -- Slavko https://www.slavino.sk/ ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Wed 05/Jun/2024 11:58:53 +0200 John Levine via mailop wrote: It appears that Tobias Fiebig via mailop said: Well, that would then be rspamd and the python email parser; Question is whether that would qualify as a bug, i.e., 'should not validate'; My understanding would be more in a 'be liberal in what you accept and conservative and what you send'-sense, though; I.e., even though not technically allowed no harm in validating. That's a common misunderstanding of the robustness principle. You should be liberal in what you accept *when the spec is ambiguous.* Other than that you should be prepared for people to send you any arbitrary garbage so you can reject it. In this case, if DKIM validators correctly rejected the invalid signatures, this mistake would have been caught and fixed more quickly. Would it? That certainly depends on the ability of the signer to understand the reason a message bounced (assuming that a "fail" would have triggered a bounce.) Unlikely. There is a field in DMARC report where a generator can put a human readable sentence to describe DKIM verification results. If I were Slavko I'd fix rspamd by adding bug reporting (if it's not already there) rather than removing 2047-decoding. Still, I wonder whether any report consumer highlights messages containing (new) human readable fields... Best Ale -- ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
In message , Tobias Fiebig via mailop writes >My point here is, that 'deliverability' is often more of a priority for >ESPs than 'following all documents to the letter'. Granted, for the >bigger ones, usually more like 'outbound deliverability' than 'inbound >deliverability', though. > >Hence, I am not completely sure if the stance is outright unreasonable, >even if it disagrees with the documents. If you send an encoded DKIM header field then it is rather unlikely that large mailbox providers will parse it correctly (I don't think they run rpamd) or indeed at all. Failure to get a "DKIM pass" may well mean that "no auth no entry" (recall that 1 Feb was a while back) will kick in and your deliverability will be zero. The likelihood of getting an error message that illuminates the (rather obscure) cause of the deliverability failure is zero. Recall that the first thing Postel said was "be conservative in what you send". BTW: RFC2047 encoding the bit in the angle brackets in the RFC5322 From (which a non-zero number of senders were doing, last I looked at $DAYJOB$ data) will also mean no deliverability (albeit some of the error messages you get for that may lead you to your formatting problem relatively quickly). -- richard Richard Clayton Those who would give up essential Liberty, to purchase a little temporary Safety, deserve neither Liberty nor Safety. Benjamin Franklin 11 Nov 1755 signature.asc Description: PGP signature ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Thu, Jun 06, 2024 at 10:23:28AM +0200, Tobias Fiebig via mailop wrote: > > To a degree, but not to the point of accepting total garbage > > (RFC2047-encoded DKIM-Signature headers), or especially, generating > > total garbage (producing RFC2047-encoded DKIM-Signature headers). > > Just to clarify here: rspamd creates correct DKIM signatures when > working as a signer; The broken DKIM sig was somebody else's > accomplishment. rspamd ingested it and was ably to verify it. That level of bending over backwards to tolerate garbage is generally counterproductive. Instead treat the message as not signed (by any domains in mangled DKIM-Signature headers). -- Viktor. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, On Thu, 2024-06-06 at 17:53 +1000, Viktor Dukhovni via mailop wrote: > It is also, IIRC, a DKIM *signer* and verifier. DKIM signing and > verification needs to interoperate. > ... > To a degree, but not to the point of accepting total garbage > (RFC2047-encoded DKIM-Signature headers), or especially, generating > total garbage (producing RFC2047-encoded DKIM-Signature headers). Just to clarify here: rspamd creates correct DKIM signatures when working as a signer; The broken DKIM sig was somebody else's accomplishment. rspamd ingested it and was ably to verify it. With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Thu, Jun 06, 2024 at 08:38:48AM +0100, Vsevolod Stakhov via mailop wrote: > > Such willful disregard of essential interoperability requirements in > > "rspamd" means I will not use it unless you back off from your current > > position, and will strongly discourage others (e.g. postfix-users list > > readers) from using it. I've heard "rspamd" otherwise has some > > appealing features, but this is show-stopper. :-( > > > > I'm sorry but I do not accept the interoperability argument in this context. > Rspamd is not an MTA - it is a spam filtering system. It is also, IIRC, a DKIM *signer* and verifier. DKIM signing and verification needs to interoperate. > Hence, it has to work *somehow* with broken and non-conformant emails. > I've seen so many messages with bad mime structure, bad headers > encoding, broken received traces etc. To a degree, but not to the point of accepting total garbage (RFC2047-encoded DKIM-Signature headers), or especially, generating total garbage (producing RFC2047-encoded DKIM-Signature headers). > What I wanted to say by this message is that complications in the standards > lead to ambiguity in the implementations (as they , which, in turn, lead to > the possibilities for spammers to exploit those implementation issues. I'm > not even talking about the standard that are ambiguous by design, e.g. DKIM > simple canonicalization. I am well aware of the issues. I wrong a MIME-normaliser circa Y2K that too questionable MIME in, and produced unambiguous MIME out. But unambiguous did not mean bending over backwards to accommodate total garbage. Rather, in many cases, just neuter it so that downstream tools are not going to be confused. And certainly, not to produce output that violates the specs. -- Viktor. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, On Thu, 2024-06-06 at 12:44 +1000, Viktor Dukhovni via mailop wrote: > The distinction is essential, because it would be a terrible mistake > to, for example, RFC2047-encode the "mailbox" construct in "From", > "To", ... headers. An RFC2047-ignorant MUA or MTA can still > correctly decode the addresses in those headers without caring about > the display name encoding. Kind of a point; However, if I got John correctly, an SMTPUTF8 mail having to go through a system that does not support it should simply bounce? On Wed, 2024-06-05 at 11:00 +0200, John R Levine via mailop wrote: > Nope. You cannot downgrade a SMTUTF8 message to an ASCII message. > The experimental versions of EAI tried to do that and it never worked > so they took it out of the standards track EAI RFCs. To be fair... if my students working on 'normal' security stuff complain about things being to confusing, inconsistent, and complex... I threaten them with the proposition that they could _also_ be working on DNS. If those working on DNS complain, I suggest they could also do BGP instead. Now, if the ones working on BGP complain, I threaten them with the prospect of working on mail. And if the ones working on mail complain... I usually just say "I'm sorry; At least look at the bright side: It can't get worse." ;-) (This is a joke.) On Thu, 2024-06-06 at 12:44 +1000, Viktor Dukhovni via mailop wrote: > It seems to me, that you may not have thought through the issues > deeply enough. I would, indeed, argue that this issue has a few more facets that need to be discussed; With some of those potentially being somewhat supportive of Vsevolod's point. On Thu, 2024-06-06 at 12:44 +1000, Viktor Dukhovni via mailop wrote: > On Wed, 2024-06-05 at 17:29 +0100, Vsevolod Stakhov via mailop wrote: > > As Rspamd author, I will not change the existing logic, as it works > > with headers as with black boxes making the following steps: unfold > > -> rfc2047 decode -> process specific data. > > This, IMNSHO, is not a reasonable stance to take... I am not so sure, at least IMSSANSMBLAHSIDTO (in my somewhat-stupid- and-not-so-my-but-looking-at-how-some-implementations-do-this opinion) the sketched approach is also how, e.g., Python handles RFC2047 headers. In fact, it was somewhat messy to convince python to rewrite an RFC2047-ized DKIM header to not mime encoded text, as the email module would read the header correctly, print it in ASCII/UTF8, but writes it back mime encoded (this is for email-security-scans.org; So a specific case sometimes requiring doing weird things not advisable anywhere else, i.e., here the rewriting.). On Thu, 2024-06-06 at 12:44 +1000, Viktor Dukhovni via mailop wrote: > Such willful disregard of essential interoperability requirements... To be fair, rspamd was the _more_ interoperable mail handling software in this case. > I've heard "rspamd" otherwise has some appealing features, but this > is show-stopper. :-( It does. Especially when it comes to DKIM (ed25519 support, not injecting funny whitespaces breaking signatures), ARC (well, it _does_ ARC, contrary to many other implementations), DMARC reporting (which also tends to be more slim in other implementations) etcetc. And, contrary to many other milters, it is really scalable. So, personally, big fan of rspamd. So, the stance, which I would abbreviate as 'in case of doubt, I do anything to get an email reliably processed and delivered' is not only Vsevolod's; In fact, Gmail--for example--also tends to be 'a bit more lenient' to get an (outbound) email delivered. Including, up until recently, 'ignore DNSSEC being broken' (even though that _does_ seem to have changed, at least when testing this morning; Or it is attached to a timer for a domain, i.e., if a domain has been seen with working DNSSEC for $time, it starts to get enforced; Anyway, tangential.). My point here is, that 'deliverability' is often more of a priority for ESPs than 'following all documents to the letter'. Granted, for the bigger ones, usually more like 'outbound deliverability' than 'inbound deliverability', though. Hence, I am not completely sure if the stance is outright unreasonable, even if it disagrees with the documents. After all, the 'drift' between documents and 'what is being done on the wire' has been getting larger for some time already. And if I have the choice between 'no commits for six years' (OpenDKIM; Good candidate for next xz, imho, btw.), 'something involving python' (dkimpy), and rspamd, I am somewhat inclined to use rspamd, i.e., there is not that many alternatives to recommend, when recommending people not to use rspamd. (The last comment is observational concerning that drift, and not a suggestion to outright change the documents, a call for rspamd to change, or a statement in support for either direction; I am marveling the problem, suggesting that this drift might need some discussion (again), and ultimately, some form of approach to fix it.) With best
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On 06/06/2024 03:44, Viktor Dukhovni via mailop wrote: On Wed, Jun 05, 2024 at 05:29:16PM +0100, Vsevolod Stakhov via mailop wrote: In fact, the original distinction between structured and unstructured headers defined in the RFC2047 just makes parsing extremely complicated and I personally consider it as an example of a standard being accepted with a clear violation of KISS principle for no good reason. The distinction is essential, because it would be a terrible mistake to, for example, RFC2047-encode the "mailbox" construct in "From", "To", ... headers. An RFC2047-ignorant MUA or MTA can still correctly decode the addresses in those headers without caring about the display name encoding. Unfortunately, SMTPUTF8 makes it even worse as instead of following something that works (e.g. punycode) it creates a completely different state machine for parsing messages otherwise indistinguishable from generic ASCII compatible emails. It seems to me, that you may not have thought through the issues deeply enough. As Rspamd author, I will not change the existing logic, as it works with headers as with black boxes making the following steps: unfold -> rfc2047 decode -> process specific data. This, IMNSHO, is not a reasonable stance to take... Such willful disregard of essential interoperability requirements in "rspamd" means I will not use it unless you back off from your current position, and will strongly discourage others (e.g. postfix-users list readers) from using it. I've heard "rspamd" otherwise has some appealing features, but this is show-stopper. :-( I'm sorry but I do not accept the interoperability argument in this context. Rspamd is not an MTA - it is a spam filtering system. Hence, it has to work *somehow* with broken and non-conformant emails. I've seen so many messages with bad mime structure, bad headers encoding, broken received traces etc. And in all the cases MUAs were able somehow to show that apparent spam/phishing to the end users. For example, one spam campaign has used '\0' character in messages, as this character is silently removed and ignored by Microsoft Outlook. What I wanted to say by this message is that complications in the standards lead to ambiguity in the implementations (as they , which, in turn, lead to the possibilities for spammers to exploit those implementation issues. I'm not even talking about the standard that are ambiguous by design, e.g. DKIM simple canonicalization. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Wed, Jun 05, 2024 at 05:29:16PM +0100, Vsevolod Stakhov via mailop wrote: > In fact, the original distinction between structured and unstructured > headers defined in the RFC2047 just makes parsing extremely complicated and > I personally consider it as an example of a standard being accepted with a > clear violation of KISS principle for no good reason. The distinction is essential, because it would be a terrible mistake to, for example, RFC2047-encode the "mailbox" construct in "From", "To", ... headers. An RFC2047-ignorant MUA or MTA can still correctly decode the addresses in those headers without caring about the display name encoding. > Unfortunately, SMTPUTF8 makes it even worse as instead of following > something that works (e.g. punycode) it creates a completely different state > machine for parsing messages otherwise indistinguishable from generic ASCII > compatible emails. It seems to me, that you may not have thought through the issues deeply enough. > As Rspamd author, I will not change the existing logic, as it works with > headers as with black boxes making the following steps: unfold -> rfc2047 > decode -> process specific data. This, IMNSHO, is not a reasonable stance to take... Such willful disregard of essential interoperability requirements in "rspamd" means I will not use it unless you back off from your current position, and will strongly discourage others (e.g. postfix-users list readers) from using it. I've heard "rspamd" otherwise has some appealing features, but this is show-stopper. :-( -- Viktor. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On 05/06/2024 10:25, Viktor Dukhovni via mailop wrote: On Wed, Jun 05, 2024 at 11:08:31AM +0200, Tobias Fiebig via mailop wrote: Yeah, I misread 8616 there, then; My brain somewhat autoclicked to "well, if there can be UTF8 you must be able to mime encode." No, RFC2047 encoding of headers applies only to header parts that are an ABNF *phrase* in an *unstructured* message header. The primary examples of this are the Display-Name in "From:", "To:" or "Cc:" and the message Subject. In fact, the original distinction between structured and unstructured headers defined in the RFC2047 just makes parsing extremely complicated and I personally consider it as an example of a standard being accepted with a clear violation of KISS principle for no good reason. Unfortunately, SMTPUTF8 makes it even worse as instead of following something that works (e.g. punycode) it creates a completely different state machine for parsing messages otherwise indistinguishable from generic ASCII compatible emails. I know that I might be opening a can of worms but I just want to express my own opinion here. As Rspamd author, I will not change the existing logic, as it works with headers as with black boxes making the following steps: unfold -> rfc2047 decode -> process specific data. Side comment: I have tried once to build spec-compatible state machine for the full RFC822 compatible ABNF in Ragel. The resulting machine was around 65k thousands of lines of C code and clang was unable to compile it without crashing. GCC managed to compile that code but the machine was extremely slow (probably because of ICache pollution/locality issues). That's probably a good example why do we have that many *slightly* incompatible parsers in the world. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
It appears that Tobias Fiebig via mailop said: >Moin, >> In this case, if DKIM validators correctly rejected the invalid >> signatures, this mistake would have been caught and fixed more >> quickly. >So, back to the initial question: Would it make sense if i'd file a bug >against rspamd? Sure. See Viktor's note about where MIME encoding is allowed. R's, John ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, > In this case, if DKIM validators correctly rejected the invalid > signatures, this mistake would have been caught and fixed more > quickly. So, back to the initial question: Would it make sense if i'd file a bug against rspamd? With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
It appears that Tobias Fiebig via mailop said: >Well, that would then be rspamd and the python email parser; Question >is whether that would qualify as a bug, i.e., 'should not validate'; My >understanding would be more in a 'be liberal in what you accept and >conservative and what you send'-sense, though; I.e., even though not >technically allowed no harm in validating. That's a common misunderstanding of the robustness principle. You should be liberal in what you accept *when the spec is ambiguous.* Other than that you should be prepared for people to send you any arbitrary garbage so you can reject it. In this case, if DKIM validators correctly rejected the invalid signatures, this mistake would have been caught and fixed more quickly. R's, John ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Wed, Jun 05, 2024 at 11:08:31AM +0200, Tobias Fiebig via mailop wrote: > Yeah, I misread 8616 there, then; My brain somewhat autoclicked to > "well, if there can be UTF8 you must be able to mime encode." No, RFC2047 encoding of headers applies only to header parts that are an ABNF *phrase* in an *unstructured* message header. The primary examples of this are the Display-Name in "From:", "To:" or "Cc:" and the message Subject. In the same headers, there is no RFC2047 encoding of email addreses. Nor is RFC2047 encoding applicable to parts of MIME Content-* (structured) headers, including "filename" in Content-Disposition which has its own dedicated encoding (https://datatracker.ietf.org/doc/html/rfc2231), but this is one case where in practice the spec was broadly violated (Microsoft was a prominent guilty party), so mail user agents typically accept RFC2047 encoding there, despite the specification. The EAI specification is generally sound, apart from one major blunder, it allowed non-identity encodings of message/global, sadly I've not had the energy to try to reverse this in a subsequent spec... -- Viktor. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, > I wouldn't verify that either. It's just wrong. You're not allowed > to MIME encode strings in a DKIM-Signature header.* Yeah, I misread 8616 there, then; My brain somewhat autoclicked to "well, if there can be UTF8 you must be able to mime encode." > * - I'm pretty sure that if you asked the author of RFC 8616, he'd > say the same thing. Yes, thank you. I am currently discussing with the author and he gave valuable input on where I misread 8616. Based on his feedback, I'll change the evaluation for such headers to 'error' instead of 'warning' in the mailtest system. > Unfortunately there is a lot of badly written mail processing code > that tries to be helpful by MIME encoding headers without checking > whether the headers allow it. Well, that would then be rspamd and the python email parser; Question is whether that would qualify as a bug, i.e., 'should not validate'; My understanding would be more in a 'be liberal in what you accept and conservative and what you send'-sense, though; I.e., even though not technically allowed no harm in validating. With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
On Wed, 5 Jun 2024, Tobias Fiebig wrote: If you're not sending SMTPUTF8 mail, the DKIM signature headers should be ASCII with no encoding needed. But if you are ending SMTPUTF8 mail, you can put UTF-8 directly in the header and it doesn't need any futher encoding either. Yeah, even more odd, the actual data did not contain any UTF-8 anyway. Meta now also fixed this. Can you give an example of the signature headers that caused a problem? They just sound wrong. See attached. dkimpy/dkimverify failed on the original mail with: I wouldn't verify that either. It's just wrong. You're not allowed to MIME encode strings in a DKIM-Signature header.* Unfortunately there is a lot of badly written mail processing code that tries to be helpful by MIME encoding headers without checking whether the headers allow it. My understanding, though, is that encoding _should_ be permissible here, as it would be needed, e.g., when receiving a message from a server with SMTPUTF8 which then must be forwarded via a server that does not support it. Nope. You cannot downgrade a SMTUTF8 message to an ASCII message. The experimental versions of EAI tried to do that and it never worked so they took it out of the standards track EAI RFCs. You can wrap one as a message/global MIME part and send it as an attachment, but you can't "translate" the message.. R's, John * - I'm pretty sure that if you asked the author of RFC 8616, he'd say the same thing. ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Hello John, > If you're not sending SMTPUTF8 mail, the DKIM signature headers > should be ASCII with no encoding needed. But if you are ending > SMTPUTF8 mail, you can put UTF-8 directly in the header and it > doesn't need any futher encoding either. Yeah, even more odd, the actual data did not contain any UTF-8 anyway. Meta now also fixed this. > Can you give an example of the signature headers that caused a > problem? They just sound wrong. See attached. dkimpy/dkimverify failed on the original mail with: dkim.MessageFormatError: b'=?UTF-8?Q?v=3D1' However, when fixing the header manually, the mail correctly validates: % cat ascii.mbox|dkimverify signature ok rspamc is also able to verify the original mail: # cat 36751.mbox| rspamc Results for file: stdin (2.36 seconds) [Metric: default] Action: no action Spam: false Score: 1.10 / 15.00 ... Symbol: DKIM_TRACE (0.00)[meta.com:+] Symbol: R_DKIM_ALLOW (-0.20)[meta.com:s=s2048-2021-q4] ... My understanding, though, is that encoding _should_ be permissible here, as it would be needed, e.g., when receiving a message from a server with SMTPUTF8 which then must be forwarded via a server that does not support it. Would be nice to know, though, as that influences how I should score such headers being there (error vs. warning) for the test-website. With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl --- ascii.mbox 2024-06-04 18:52:56.213538681 +0200 +++ 36751.mbox 2024-06-03 23:44:21.342406024 +0200 @@ -13,7 +13,16 @@ Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 453LCtYI010400; Mon, 3 Jun 2024 14:24:06 -0700 -DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=meta.com; h=content-type:date:from:in-reply-to:message-id:mime-version:references:subject:to; s=s2048-2021-q4; bh=+u8G0t/3N3vkyvfnVsE3LmLsnsfDjQZbrcpl03e4/lw=; b=b/+Or8DXaJuDThOsxHzXoSEPgs595iO+fF50NpcmYH0/FyZ7YbDAASqsssKGp3A2P0+H P9Ed7gY8QrH4kb5PydaPnO7aHOd/ggqaDszuxXqzTpwN6WYNpEsjG0Rci4J+CaMClpR4 MmDT9lkvErV1LzKBr2CeVOmjlcoY9DDXV58W7LF8ChcBolXb5jphrn/kjsT3wUNAADjC qXuiaK8XjckdThchqrAhaaqmjTrym8iFMcwmYcnxtlxdMc0vDAcj1L9OwaTJF2orIBEu gqyoY4Kp5wmXZMEMKbb4n8AphX4fbWJBSENovqYfaBAzFJZUghkfrlW+4pN7qYxf7i6d 5w== +DKIM-Signature: =?UTF-8?Q?v=3D1;_a=3Drsa-sha256;_c=3Drelaxed/relaxed;_d=3Dmeta.com;_h=3Dc?= + =?UTF-8?Q?ontent-type:date:from:in-reply-to:message-id:mime-version:refer?= + =?UTF-8?Q?ences:subject:to;_s=3Ds2048-2021-q4;_bh=3D+u8G0t/3N3vkyvfnVsE3L?= + =?UTF-8?Q?mLsnsfDjQZbrcpl03e4/lw=3D;_b=3Db/+Or8DXaJuDThOsxHzXoSEPgs595iO+?= + =?UTF-8?Q?fF50NpcmYH0/FyZ7YbDAASqsssKGp3A2P0+H_P9Ed7gY8QrH4kb5PydaPnO7aHO?= + =?UTF-8?Q?d/ggqaDszuxXqzTpwN6WYNpEsjG0Rci4J+CaMClpR4_MmDT9lkvErV1LzKBr2Ce?= + =?UTF-8?Q?VOmjlcoY9DDXV58W7LF8ChcBolXb5jphrn/kjsT3wUNAADjC_qXuiaK8XjckdTh?= + =?UTF-8?Q?chqrAhaaqmjTrym8iFMcwmYcnxtlxdMc0vDAcj1L9OwaTJF2orIBEu_gqyoY4Kp?= + =?UTF-8?Q?5wmXZMEMKbb4n8AphX4fbWJBSENovqYfaBAzFJZUghkfrlW+4pN7qYxf7i6d_5w?= + =?UTF-8?Q?=3D=3D_?= Received: from nam12-dm6-obe.outbound.protection.outlook.com (mail-dm6nam12lp2169.outbound.protection.outlook.com [104.47.59.169]) by mx0a-00082601.pphosted.com (PPS) with ESMTPS id 3yhg8jtq5u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); DKIM-Signature: =?UTF-8?Q?v=3D1;_a=3Drsa-sha256;_c=3Drelaxed/relaxed;_d=3Dmeta.com;_h=3Dc?= =?UTF-8?Q?ontent-type:date:from:in-reply-to:message-id:mime-version:refer?= =?UTF-8?Q?ences:subject:to;_s=3Ds2048-2021-q4;_bh=3D+u8G0t/3N3vkyvfnVsE3L?= =?UTF-8?Q?mLsnsfDjQZbrcpl03e4/lw=3D;_b=3Db/+Or8DXaJuDThOsxHzXoSEPgs595iO+?= =?UTF-8?Q?fF50NpcmYH0/FyZ7YbDAASqsssKGp3A2P0+H_P9Ed7gY8QrH4kb5PydaPnO7aHO?= =?UTF-8?Q?d/ggqaDszuxXqzTpwN6WYNpEsjG0Rci4J+CaMClpR4_MmDT9lkvErV1LzKBr2Ce?= =?UTF-8?Q?VOmjlcoY9DDXV58W7LF8ChcBolXb5jphrn/kjsT3wUNAADjC_qXuiaK8XjckdTh?= =?UTF-8?Q?chqrAhaaqmjTrym8iFMcwmYcnxtlxdMc0vDAcj1L9OwaTJF2orIBEu_gqyoY4Kp?= =?UTF-8?Q?5wmXZMEMKbb4n8AphX4fbWJBSENovqYfaBAzFJZUghkfrlW+4pN7qYxf7i6d_5w?= =?UTF-8?Q?=3D=3D_?= ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
According to Tobias Fiebig via mailop : >Moin, > >to share some closure on this with the rest of the list; What was >ultimately the issue was an RFC8616 based DKIM-Signature header, i.e., >containing UTF-8 encoded fields (in this case, even though there were >no non-ascii characters in them). ... If you're not sending SMTPUTF8 mail, the DKIM signature headers should be ASCII with no encoding needed. But if you are ending SMTPUTF8 mail, you can put UTF-8 directly in the header and it doesn't need any futher encoding either. Can you give an example of the signature headers that caused a problem? They just sound wrong. R's, John ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, to share some closure on this with the rest of the list; What was ultimately the issue was an RFC8616 based DKIM-Signature header, i.e., containing UTF-8 encoded fields (in this case, even though there were no non-ascii characters in them). While rspamd on my machine did not have an issue, forwarding recipients were unable to parse this. Incidentally, it seems like dkimpy also is unable to parse such headers, even though the email modules decodes them just fine. I added support to test for this to email-security-scans.org, i.e., if your DKIM header contains UTF8 encoded fields, it now shows a warning. With best regards, Tobias On Mon, 2024-06-03 at 23:11 +0200, Tobias Fiebig via mailop wrote: > Moin, > got it, thx. > > With best regards, > Tobias > > On Mon, 2024-06-03 at 21:36 +0200, Tobias Fiebig via mailop wrote: > > Moin, > > > > tl;dr: Could someone do https://email-security-scans.org from > > meta.com, > > storing mails on the server and sharing the link with me to help me > > debug a deliverability issue? > > > > I just got a report in that a user's mail bounced when writing from > > '@meta.com' to an alias on a domain I operate, which forwards to a > > third party hosting on zoho.com. > > > > The NDR is for a DMARC reject; In my logs, I see that: > > - ARC verification already failed on inbound with a bh mismatch > > - DKIM seems to have passed, though, at least according to the logs > > (with a selector hinting at it being for Q4 2021) > > > > zoho.com then rejects with "550 5.7.1 Email rejected per DMARC > > policy" > > > > Given that SPF obviously fails, the question is why DMARC does no > > longer validate when hitting zoho.com; I currently suspect that > > there > > was either a tmp-error for the lookup of the DKIM key, or that > > meta/outlook.com signs some headers that may be affected during a > > normal forward. Also, there are no issues with other DKIM signing > > p=reject domains being forwarded via the setup. > > > > To help me debug this; Is there anyone from meta / with an account > > under meta.com on the list who could do a test on > > https://email-security-scans.org (ideally checking the 'store my > > mails' > > checkbox)? > > > > (Already asked into the direction of the user as well, but this is > > a > > multi-hop conversation through a user of mine; And it somewhat bugs > > me > > and i'd like to resolve this asap. ;-)) > > > > With best regards, > > Tobias > > > -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
Re: [mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, got it, thx. With best regards, Tobias On Mon, 2024-06-03 at 21:36 +0200, Tobias Fiebig via mailop wrote: > Moin, > > tl;dr: Could someone do https://email-security-scans.org from > meta.com, > storing mails on the server and sharing the link with me to help me > debug a deliverability issue? > > I just got a report in that a user's mail bounced when writing from > '@meta.com' to an alias on a domain I operate, which forwards to a > third party hosting on zoho.com. > > The NDR is for a DMARC reject; In my logs, I see that: > - ARC verification already failed on inbound with a bh mismatch > - DKIM seems to have passed, though, at least according to the logs > (with a selector hinting at it being for Q4 2021) > > zoho.com then rejects with "550 5.7.1 Email rejected per DMARC > policy" > > Given that SPF obviously fails, the question is why DMARC does no > longer validate when hitting zoho.com; I currently suspect that there > was either a tmp-error for the lookup of the DKIM key, or that > meta/outlook.com signs some headers that may be affected during a > normal forward. Also, there are no issues with other DKIM signing > p=reject domains being forwarded via the setup. > > To help me debug this; Is there anyone from meta / with an account > under meta.com on the list who could do a test on > https://email-security-scans.org (ideally checking the 'store my > mails' > checkbox)? > > (Already asked into the direction of the user as well, but this is a > multi-hop conversation through a user of mine; And it somewhat bugs > me > and i'd like to resolve this asap. ;-)) > > With best regards, > Tobias > -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
[mailop] Debugging fwd issue meta.com to zoho.com (Help from user under meta.com needed)
Moin, tl;dr: Could someone do https://email-security-scans.org from meta.com, storing mails on the server and sharing the link with me to help me debug a deliverability issue? I just got a report in that a user's mail bounced when writing from '@meta.com' to an alias on a domain I operate, which forwards to a third party hosting on zoho.com. The NDR is for a DMARC reject; In my logs, I see that: - ARC verification already failed on inbound with a bh mismatch - DKIM seems to have passed, though, at least according to the logs (with a selector hinting at it being for Q4 2021) zoho.com then rejects with "550 5.7.1 Email rejected per DMARC policy" Given that SPF obviously fails, the question is why DMARC does no longer validate when hitting zoho.com; I currently suspect that there was either a tmp-error for the lookup of the DKIM key, or that meta/outlook.com signs some headers that may be affected during a normal forward. Also, there are no issues with other DKIM signing p=reject domains being forwarded via the setup. To help me debug this; Is there anyone from meta / with an account under meta.com on the list who could do a test on https://email-security-scans.org (ideally checking the 'store my mails' checkbox)? (Already asked into the direction of the user as well, but this is a multi-hop conversation through a user of mine; And it somewhat bugs me and i'd like to resolve this asap. ;-)) With best regards, Tobias -- Dr.-Ing. Tobias Fiebig T +31 616 80 98 99 M tob...@fiebig.nl ___ mailop mailing list mailop@mailop.org https://list.mailop.org/listinfo/mailop
