Re: Pigeonhole redirect is adding a message-id header when it already exists

[A. Schulze]

Am 02.10.22 um 11:37 schrieb Emmanuel Fusté:

Le 02/10/2022 à 06:35, Sébastien Riccio a écrit :

- What options could we have to resolve this?

a) Having dovecot core to remove the Message-ID header line from the mail if it 
is not going to consider it valid ? (So there is no dupe headers when 
pigeonhole adds one?)
b) Having pigeonhole check, when adding a new valid Message-ID, if there is 
already one existing, and remove the bogus one ?


Hello,
In my opinion, an option c) should be considered.
Bogus or not, if a Message-ID is present in the header you should not touch it.


I agree. Pigeonhole should not modify content on forwarding.

Andreas

Re: Pigeonhole redirect is adding a message-id header when it already exists

[John Stoffel]

> "Michael" == Michael Peddemors  writes:

> This should almost be an RFC discussion, rather than a dovecot 
> discussion, for clarity on what to do with a malformed Message-Id.

Seems simple, just rename it to "Message-Id-Orig: ..." and insert your
own.  Fixes the problem, still let's you trace the message, etc. 



> For the record, if you start modifying it by deleting the bad message 
> id, and adding your own, you can start breaking other things, such as 
> DKIM signing etc..

> IMHO, Dovecot should simply refuse to accept or deliver a message with a 
> 'bad' message id, so that the sending system can identify and correct 
> the problem.

> That way Dovecot doesn't need to address/modify the email message.

>   -- Michael --

> On 2022-10-01 21:35, Sébastien Riccio wrote:
>> Hi,
>> 
>> After reading a bit the code and trying to understand it, here is what I 
>> think happens here:
>> 
>> Given a bogus Message-ID, for example (notice it's missing angle 
>> brackets < >:
>> 
>> Message-ID: 
>> 1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7
>> 
>> Dovecot is parsing Message-ID headers and is looking for the angle 
>> bracket as the begining of the Message-ID:
>> https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75
>>  
>> 
>> 
>> As none is found it will act as if there was no Message-ID header in the 
>> mail (even that the header is present).
>> 
>> Then, pigeonhole's redirect function  is told to generate a new 
>> Message-ID if none was previously detected:
>> https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569
>>  
>> 
>> 
>> The result is the mail being forwarded, in this case, is now having dual 
>> Message-ID and is not RFC 5322 compliant anymore and can be rejected for 
>> this reason (hi, gmail?)
>> https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html
>>  
>> 
>> 
>> Some thoughts:
>> 
>> - First, to be honest, I'm not sure gmail would accept the original mail 
>> with the bogus Message-ID sent directly to their servers, but if it was 
>> refused, I would assume that these senders would have fixed the issue on 
>> their side so their message are delivered (unless there is some 
>> whitelisting going on?)
>> 
>> - What options could we have to resolve this?
>> 
>> a) Having dovecot core to remove the Message-ID header line from the 
>> mail if it is not going to consider it valid ? (So there is no dupe 
>> headers when pigeonhole adds one?)
>> b) Having pigeonhole check, when adding a new valid Message-ID, if there 
>> is already one existing, and remove the bogus one ?
>> For now, to workaround this, I'm trying to find a way in the mail flow 
>> on our servers to keep only the top most Message-ID when more than one 
>> exists.
>> Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks 
>>  but I'm not 
>> sure how to achieve it yet or even if it's possible.
>> 
>> Kind regards
>> 
>> *Sébastien RICCIO*
>> 
>> *SYSTEM ADMINISTRATOR*
>> 
>> *P*  +41 840 888 888
>> 
>> *F***+41 840 888 000
>> 
>> *Msric...@swisscenter.com *
>> 
>> *
>> *
>> 
>> 
>> 
>> -- Message d'origine --
>> De "michael.z...@feierfighter.de " 
>> mailto:michael.z...@feierfighter.de>>
>> À "dovecot@dovecot.org " 
>> mailto:dovecot@dovecot.org>>
>> Date 01.10.2022 14:49:13
>> Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when 
>> it already exists
>> 
>>> Hi there,
>>> I can confirm this behavior. A few months ago I introduced a milter 
>>> which is checking for multiple headers when the RFC says that there 
>>> just should be one of them For example "Message-Id".
>>> I found the described problem in an email coming from Alibaba, which 
>>> had an invalid "Message-Id" header. It didn't contain an "@" sign or 
>>> similar. It was RFC-invalid.
>>> This email was sent from Alibaba to a German email provider. There was 
>>> a redirect at that email provider, pointing to my mailserver.
>>> My server rejected the email because there were 2 "Message-Id" 
>>> headers: The original invalid "Message-Id" header from Alibaba, and a 
>>> new "Message-Id" header from the German provider, which seems to have 
>>> been added during the redirect. There were "Dovecot-sieve" headers in 
>>> that mail, so my guess was that it happened because of 
>>> Dovecot-sieve/pigeonhole implementation.
>>> I contacted the email provider, asking for help. Asking if it really 

Re: Pigeonhole redirect is adding a message-id header when it already exists

[Michael Peddemors]

This should almost be an RFC discussion, rather than a dovecot 
discussion, for clarity on what to do with a malformed Message-Id.


For the record, if you start modifying it by deleting the bad message 
id, and adding your own, you can start breaking other things, such as 
DKIM signing etc..


IMHO, Dovecot should simply refuse to accept or deliver a message with a 
'bad' message id, so that the sending system can identify and correct 
the problem.


That way Dovecot doesn't need to address/modify the email message.

-- Michael --

On 2022-10-01 21:35, Sébastien Riccio wrote:

Hi,

After reading a bit the code and trying to understand it, here is what I 
think happens here:


Given a bogus Message-ID, for example (notice it's missing angle 
brackets < >:


Message-ID: 
1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7


Dovecot is parsing Message-ID headers and is looking for the angle 
bracket as the begining of the Message-ID:
https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75 



As none is found it will act as if there was no Message-ID header in the 
mail (even that the header is present).


Then, pigeonhole's redirect function  is told to generate a new 
Message-ID if none was previously detected:
https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569 



The result is the mail being forwarded, in this case, is now having dual 
Message-ID and is not RFC 5322 compliant anymore and can be rejected for 
this reason (hi, gmail?)
https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html 



Some thoughts:

- First, to be honest, I'm not sure gmail would accept the original mail 
with the bogus Message-ID sent directly to their servers, but if it was 
refused, I would assume that these senders would have fixed the issue on 
their side so their message are delivered (unless there is some 
whitelisting going on?)


- What options could we have to resolve this?

a) Having dovecot core to remove the Message-ID header line from the 
mail if it is not going to consider it valid ? (So there is no dupe 
headers when pigeonhole adds one?)
b) Having pigeonhole check, when adding a new valid Message-ID, if there 
is already one existing, and remove the bogus one ?
For now, to workaround this, I'm trying to find a way in the mail flow 
on our servers to keep only the top most Message-ID when more than one 
exists.
Maybe using: https://www.postfix.org/postconf.5.html#smtp_header_checks 
 but I'm not 
sure how to achieve it yet or even if it's possible.


Kind regards

*Sébastien RICCIO*

*SYSTEM ADMINISTRATOR*

*P*  +41 840 888 888

*F***+41 840 888 000

*Msric...@swisscenter.com *

*
*



-- Message d'origine --
De "michael.z...@feierfighter.de " 
mailto:michael.z...@feierfighter.de>>
À "dovecot@dovecot.org " 
mailto:dovecot@dovecot.org>>

Date 01.10.2022 14:49:13
Objet Re: Re[6]: Pigeonhole redirect is adding a message-id header when 
it already exists



Hi there,
I can confirm this behavior. A few months ago I introduced a milter 
which is checking for multiple headers when the RFC says that there 
just should be one of them For example "Message-Id".
I found the described problem in an email coming from Alibaba, which 
had an invalid "Message-Id" header. It didn't contain an "@" sign or 
similar. It was RFC-invalid.
This email was sent from Alibaba to a German email provider. There was 
a redirect at that email provider, pointing to my mailserver.
My server rejected the email because there were 2 "Message-Id" 
headers: The original invalid "Message-Id" header from Alibaba, and a 
new "Message-Id" header from the German provider, which seems to have 
been added during the redirect. There were "Dovecot-sieve" headers in 
that mail, so my guess was that it happened because of 
Dovecot-sieve/pigeonhole implementation.
I contacted the email provider, asking for help. Asking if it really 
is a bug in pigeonhole (or maybe some other system at that provider, 
who knows). And I contacted Alibaba, so they fix the invalid 
"Message-Id". I got responses from both, but until now, as far as I 
can see, it has not been fixed.
The best fix would be (if it really is a bug in pigeonhole), if 
pigeonhole fixes the problem, then it's fixed for all users of 
Dovecot. I guess Alibaba is not the only sender with an invalid 
"Message-ID" header, but that's the only one I saw.

Michael


Am 

Re: Pigeonhole redirect is adding a message-id header when it already exists

[Emmanuel Fusté]

Le 02/10/2022 à 06:35, Sébastien Riccio a écrit :

Hi,

After reading a bit the code and trying to understand it, here is what 
I think happens here:


Given a bogus Message-ID, for example (notice it's missing angle 
brackets < >:


Message-ID: 
1883biz_pay_after_purchase:0:0_572392900$ae7ed6e4d53b424c84aaf83b30c507e7


Dovecot is parsing Message-ID headers and is looking for the angle 
bracket as the begining of the Message-ID:

https://github.com/dovecot/core/blob/d2ff32792ac052610cea7d65f30de1ee139cb55c/src/lib-mail/message-id.c#L75

As none is found it will act as if there was no Message-ID header in 
the mail (even that the header is present).


Then, pigeonhole's redirect function  is told to generate a new 
Message-ID if none was previously detected:

https://github.com/dovecot/pigeonhole/blob/5a3f4bd672cc2fb9e755a4b09c4753ac86e15f99/src/lib-sieve/cmd-redirect.c#L569

The result is the mail being forwarded, in this case, is now having 
dual Message-ID and is not RFC 5322 compliant anymore and can be 
rejected for this reason (hi, gmail?)

https://www.spamresource.com/2022/08/gmail-weird-rfc-5322-bounces-and-what.html

Some thoughts:

- First, to be honest, I'm not sure gmail would accept the original 
mail with the bogus Message-ID sent directly to their servers, but if 
it was refused, I would assume that these senders would have fixed the 
issue on their side so their message are delivered (unless there is 
some whitelisting going on?)


- What options could we have to resolve this?

a) Having dovecot core to remove the Message-ID header line from the 
mail if it is not going to consider it valid ? (So there is no dupe 
headers when pigeonhole adds one?)
b) Having pigeonhole check, when adding a new valid Message-ID, if 
there is already one existing, and remove the bogus one ?


Hello,
In my opinion, an option c) should be considered.
Bogus or not, if a Message-ID is present in the header you should not 
touch it.
For it own use, dovecot should continue to not use it as it is bogus.The 
only thing to do is to warn in logs in the dovecot core.
In a forwarding scenario, you should not alter the existing Message-ID, 
you should not take the responsibility to fix someone else problem.
The bogus message ID key is ok for duplicate detection. The only risk is 
false trigger by another equal bogus Message-ID header. Bogus messages 
are bogus so ...
For the present case, it means that the check for the header presence 
should be done by pigeonhole directly, not relying on the dovecot core 
Message-ID validity check.
In the missing/not implemented/not drafted redirect-as-new (inline or 
attached) case which exiting in some systems, you will generate a 
completely new message with a fresh/compliant Message-ID.


Emmanuel.

Re: Pigeonhole redirect is adding a message-id header when it already exists

[hi]

On 2022-10-01 22:59, michael.z...@feierfighter.de wrote:

Hi Zakaria,

I'm very happy about your email to the mailing list, I thought I'm
alone with my problem, and it's nice to see that the problem might be
a bug/problem in pigeonhole. I was not sure about it because I didn't
get a technical answer from that German provider, so I didn't know it
it's a bug in pigeonhole or not. I didn't have time to investigate,
thanks for your time to investigate and pinpoint to the line of code
where the problem might be!

Back then, a few months ago, I disabled the "Message-Id duplicate
check" because of this problem.

It's just a problem of <0.001% of the emails, most systems generate
valid Message-IDs. Maybe Alibaba/Aliexpress is the biggest one who
generates invalid Message-Ids...

It would be awesome if you could also try to contact
Alibaba/Aliexpress, and tell them the problem. Maybe they fix it if
multiple people report it. But I doubt it.

Let's see what the Dovecot guys say to this problem. Maybe the best
solution is to remove the invalid Message-Id before adding a new one.

Michael

Am 01-Oct-2022 18:24:05 +0200 schrieb hi@zakaria.website:


On 2022-10-01 12:49, michael.z...@feierfighter.de wrote:

Hi there,

I can confirm this behavior. A few months ago I introduced a

milter

which is checking for multiple headers when the RFC says that

there

just should be one of them For example "Message-Id".

I found the described problem in an email coming from Alibaba,

which

had an invalid "Message-Id" header. It didn't contain an "@" sign

or

similar. It was RFC-invalid.

This email was sent from Alibaba to a German email provider. There

was

a redirect at that email provider, pointing to my mailserver.

My server rejected the email because there were 2 "Message-Id"
headers: The original invalid "Message-Id" header from Alibaba,

and a

new "Message-Id" header from the German provider, which seems to

have

been added during the redirect. There were "Dovecot-sieve" headers

in

that mail, so my guess was that it happened because of
Dovecot-sieve/pigeonhole implementation.

I contacted the email provider, asking for help. Asking if it

really

is a bug in pigeonhole (or maybe some other system at that

provider,

who knows). And I contacted Alibaba, so they fix the invalid
"Message-Id". I got responses from both, but until now, as far as

I

can see, it has not been fixed.

The best fix would be (if it really is a bug in pigeonhole), if
pigeonhole fixes the problem, then it's fixed for all users of
Dovecot. I guess Alibaba is not the only sender with an invalid
"Message-ID" header, but that's the only one I saw.

Michael

Am 01-Oct-2022 14:00:45 +0200 schrieb sric...@swisscenter.com:


You wrote in the original email the message was rejected. Sorry

I

don't have login access to my gmail test account anymore since

the

google @#$%@#$% wanted to have me add a phone number.

In my original post I said that gmail was rejecting the forwards
because
of duplicate headers, and that the duplicate header seems to be a
Message-ID added by pigeonhole when it's "not happy" with the
original
mail Message-ID.

I probably failed to explain the issue clearly and sorry for

that.


Thank you anyway for trying to help :)


Hi Michael,

I just wanted to say sorry, that I sent this empty email by mistake
intending nothing else.

Anyhow, thanks so much for your valuable input, it's very much
appreciated.

Zakaria.


Good Morning Michael,

No problem. Although, to be clear I didnt point out anything, but its 
Sébastien and the credit shall go for him.


Also, by the way in EXIM MTA, there is verify syntax condition in data 
acl, which checks all email headers and validate that they are standards 
compliant and reject sending any email beforehand to your issue which 
has non-compliant header and offers the ability to remove such headers 
with condition handling e.g. if contained any Sieve header, to request 
removing message id headers with headers_remove and add compliant one 
with add_header, and I guess this will mitigate any mail service 
provider rejection issue.


If you are postfix user and considering switching, let me know, I might 
be able to help.


Zakaria.

RE: Pigeonhole redirect is adding a message-id header when it already exists

[Marc]

 
> 
> 
> We are using sieve filter to redirect incoming mails to an external
> mailbox (gmail) an we are seeing some forwarded e-mails being rejected
> by gmail because of duplicate headers.
> 

How are you redirecting, like this[1]? I tested this ages ago. I am mostly 
testing with own gmail account, so  can't imagine this not working. It was an 
older 2.2 server I was testing this on.


[1]
if false # true
{
 redirect :copy "s...@gmail.com";
}