empty message id

[Amira Othman]

Hi all

I found in postfix log empty message id for email received. Is that mean I
am receiving spam ? and how can I handle that?

Regards

 

empty message-ID

[natan]

Hi
I have probably to trivial questions about message-ID

Why sometimes, some user have empty message-id=<>

example:
Nov 23 13:13:53 smtp1 postfix/submission/smtpd[29867]: 4CfmKF1CSDz5MwK:
.domain.ltd[xxx.xxx.xxx.xxx], sasl_method=login,
sasl_username=bi...@domain2.ltd
Nov 23 13:13:53 smtp1 postfix/cleanup[46909]: 4CfmKF1CSDz5MwK: message-id=<>
Nov 23 13:13:53 smtp1 postfix/qmgr[25287]: 4CfmKF1CSDz5MwK:
from=, size=94874, nrcpt=3 (queue active)
.



--

Re: empty message id

[Ralf Hildebrandt]

* Amira Othman :
> Hi all
> 
> I found in postfix log empty message id for email received. 

Yep, seen those too

> Is that mean I am receiving spam ?

No, it means you're receiving mails with an empty message-id :)

> and how can I handle that?

What is there to handle? Something not working?

Dec  7 11:40:23 mail postfix/cleanup[26180]: 3SyxDR6fnNzFvmL: message-id=<>
Dec  7 11:41:09 mail postfix/cleanup[26708]: 3SyxFK1S5TzFvml: message-id=<>
Dec  7 11:44:51 mail postfix/cleanup[27202]: 3SyxKb3H7vzFvn0: message-id=<>
Dec  7 11:45:06 mail postfix/cleanup[27719]: 3SyxKt0tmgzFvn0: message-id=<>
Dec  7 11:45:24 mail postfix/cleanup[27837]: 3SyxLD2N7TzFvjb: message-id=<>
Dec  7 11:45:24 mail postfix/cleanup[27719]: 3SyxLD2ThwzFvjl: message-id=<>
Dec  7 11:45:32 mail postfix/cleanup[27719]: 3SyxLN5g79zFvjb: message-id=<>
Dec  7 11:46:49 mail postfix/cleanup[27837]: 3SyxMs6YQKzFvjb: message-id=<>
Dec  7 11:46:49 mail postfix/cleanup[28188]: 3SyxMs6svGzFvmL: message-id=<>


-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: empty message id

[Benny Pedersen]

On Wed, 7 Dec 2011 12:39:25 +0200, Amira Othman wrote:


I found in postfix log empty message id for email received. Is that
mean I am receiving spam ? and how can I handle that?


spamassassin hits on MSGID_MULTIPLE_AT from this mail, so related ?

RE: empty message id

[Amira Othman]

No everything is working fine. But I thought that every email is sent 
associated with unique message id. And I am using it in a script that parse log 
file to insert it to database. But now I have duplication in message id because 
of empty one . does every mail sent or received must have queue id so I can use 
it instead of message id? And is it unique ?

* Amira Othman :
> Hi all
> 
> I found in postfix log empty message id for email received. 

Yep, seen those too

> Is that mean I am receiving spam ?

No, it means you're receiving mails with an empty message-id :)

> and how can I handle that?

What is there to handle? Something not working?

Dec  7 11:40:23 mail postfix/cleanup[26180]: 3SyxDR6fnNzFvmL: message-id=<>
Dec  7 11:41:09 mail postfix/cleanup[26708]: 3SyxFK1S5TzFvml: message-id=<>
Dec  7 11:44:51 mail postfix/cleanup[27202]: 3SyxKb3H7vzFvn0: message-id=<>
Dec  7 11:45:06 mail postfix/cleanup[27719]: 3SyxKt0tmgzFvn0: message-id=<>
Dec  7 11:45:24 mail postfix/cleanup[27837]: 3SyxLD2N7TzFvjb: message-id=<>
Dec  7 11:45:24 mail postfix/cleanup[27719]: 3SyxLD2ThwzFvjl: message-id=<>
Dec  7 11:45:32 mail postfix/cleanup[27719]: 3SyxLN5g79zFvjb: message-id=<>
Dec  7 11:46:49 mail postfix/cleanup[27837]: 3SyxMs6YQKzFvjb: message-id=<>
Dec  7 11:46:49 mail postfix/cleanup[28188]: 3SyxMs6svGzFvmL: message-id=<>


-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: empty message id

[Ralf Hildebrandt]

* Amira Othman :

> No everything is working fine. But I thought that every email is sent
> associated with unique message id.

Theory and praxis :)

> And I am using it in a script that parse log file to insert it to
> database. But now I have duplication in message id because of empty one
> . does every mail sent or received must have queue id so I can use it
> instead of message id? And is it unique ?

If you use long queue filenames (enable_long_queue_ids = yes) they're
unique.

-- 
Ralf Hildebrandt
  Geschäftsbereich IT | Abteilung Netzwerk
  Charité - Universitätsmedizin Berlin
  Campus Benjamin Franklin
  Hindenburgdamm 30 | D-12203 Berlin
  Tel. +49 30 450 570 155 | Fax: +49 30 450 570 962
  ralf.hildebra...@charite.de | http://www.charite.de

Re: empty message id

[Wietse Venema]

Amira Othman:
> Hi all
> 
> I found in postfix log empty message id for email received. Is that mean I
> am receiving spam ? and how can I handle that?

Postfix logs an empty message-id when the message does not have one.
According to RFC 822 and its successors, Message-ID is not required. 

Wietse 

Re: empty message-ID

[Matus UHLAR - fantomas]

On 23.11.20 14:35, natan wrote:

I have probably to trivial questions about message-ID

Why sometimes, some user have empty message-id=<>

example:
Nov 23 13:13:53 smtp1 postfix/submission/smtpd[29867]: 4CfmKF1CSDz5MwK:
.domain.ltd[xxx.xxx.xxx.xxx], sasl_method=login,
sasl_username=bi...@domain2.ltd
Nov 23 13:13:53 smtp1 postfix/cleanup[46909]: 4CfmKF1CSDz5MwK: message-id=<>
Nov 23 13:13:53 smtp1 postfix/qmgr[25287]: 4CfmKF1CSDz5MwK:
from=, size=94874, nrcpt=3 (queue active)
.


client's MUA apparently does not generate Message-Id: header.

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Boost your system's speed by 500% - DEL C:\WINDOWS\*.*

Re: empty message-ID

[maciejm]

Hi
Thanks for replay I found "RFC 822 Message-ID is not required"
Probably "problem" is in configurations in some clients.

On 23.11.2020 14:39, Matus UHLAR - fantomas wrote:
> On 23.11.20 14:35, natan wrote:
>> I have probably to trivial questions about message-ID
>>
>> Why sometimes, some user have empty message-id=<>
>>
>> example:
>> Nov 23 13:13:53 smtp1 postfix/submission/smtpd[29867]: 4CfmKF1CSDz5MwK:
>> .domain.ltd[xxx.xxx.xxx.xxx], sasl_method=login,
>> sasl_username=bi...@domain2.ltd
>> Nov 23 13:13:53 smtp1 postfix/cleanup[46909]: 4CfmKF1CSDz5MwK:
>> message-id=<>
>> Nov 23 13:13:53 smtp1 postfix/qmgr[25287]: 4CfmKF1CSDz5MwK:
>> from=, size=94874, nrcpt=3 (queue active)
>> .
>
> client's MUA apparently does not generate Message-Id: header.
>

Re: empty message-ID

[D'Arcy Cain]

On 11/23/20 9:49 AM, maciejm wrote:

Hi
Thanks for replay I found "RFC 822 Message-ID is not required"
Probably "problem" is in configurations in some clients.


I used to have a client who was not getting emails from one of his friends. 
 Turned out that the friend's client/MUA was not adding the message ID. 
After the first message was accepted all of the rest were silently dropped 
as duplicates due to a very standard procmail recipe:


:0 Wh: msgid.lock
| formail -D 65536 $HOME/.msgid.cache

In other words, the message ID "" was considered a duplicate after the first 
one.


--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:da...@vybenetworks.com
VoIP: sip:da...@vex.net

Re: empty message-ID

[Jaroslaw Rafa]

Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
> 
> I used to have a client who was not getting emails from one of his
> friends.  Turned out that the friend's client/MUA was not adding the
> message ID.

Doesn't Postfix automatically add Message-Id: header upon sending a message
if none is present?

> After the first message was accepted all of the rest
> were silently dropped as duplicates due to a very standard procmail
> recipe:
> 
> :0 Wh: msgid.lock
> | formail -D 65536 $HOME/.msgid.cache

Who uses that? It's not normal to get email duplicates and it usually
means that mail system is not functioning properly. They should find the
cause of the duplicates and eliminate it instead of hiding symptoms...
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: empty message-ID

[Phil Stracchino]

On 11/23/20 9:18 AM, D'Arcy Cain wrote:
> On 11/23/20 9:49 AM, maciejm wrote:
>> Hi
>> Thanks for replay I found "RFC 822 Message-ID is not required"
>> Probably "problem" is in configurations in some clients.
> 
> I used to have a client who was not getting emails from one of his friends. 
>   Turned out that the friend's client/MUA was not adding the message ID. 
> After the first message was accepted all of the rest were silently dropped 
> as duplicates due to a very standard procmail recipe:
> 
> :0 Wh: msgid.lock
> | formail -D 65536 $HOME/.msgid.cache
> 
> In other words, the message ID "" was considered a duplicate after the first 
> one.


The lesson from which is, don't make message delivery dependent upon an
optional header.


-- 
  Phil Stracchino
  Babylon Communications
  ph...@caerllewys.net
  p...@co.ordinate.org
  Landline: +1.603.293.8485
  Mobile:   +1.603.998.6958

Re: empty message-ID

[Wietse Venema]

Jaroslaw Rafa:
> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
> > 
> > I used to have a client who was not getting emails from one of his
> > friends.  Turned out that the friend's client/MUA was not adding the
> > message ID.
> 
> Doesn't Postfix automatically add Message-Id: header upon sending a message
> if none is present?

For the last 17 years, Message-ID is added to "local" email only.

http://www.postfix.org/postconf.5.html#always_add_missing_headers

> > After the first message was accepted all of the rest
> > were silently dropped as duplicates due to a very standard procmail
> > recipe:
> > 
> > :0 Wh: msgid.lock
> > | formail -D 65536 $HOME/.msgid.cache

They should skip that when email has no Message-Id: header.

Wietse

Re: empty message-ID

[D'Arcy Cain]

On 11/23/20 10:44 AM, Jaroslaw Rafa wrote:

Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:


I used to have a client who was not getting emails from one of his
friends.  Turned out that the friend's client/MUA was not adding the
message ID.


Doesn't Postfix automatically add Message-Id: header upon sending a message
if none is present?


I guess they weren't using Postfix.


After the first message was accepted all of the rest
were silently dropped as duplicates due to a very standard procmail
recipe:

:0 Wh: msgid.lock
| formail -D 65536 $HOME/.msgid.cache


Who uses that? It's not normal to get email duplicates and it usually
means that mail system is not functioning properly. They should find the
cause of the duplicates and eliminate it instead of hiding symptoms...


If someone replies to a mailing list and copies the sender then that person 
gets two copies.  The above recipe avoids that.


People also send to every alias that someone has.  Example, billing@, 
admin@, support@, joe@, etc.


--
D'Arcy J.M. Cain
System Administrator, Vex.Net
http://www.Vex.Net/ IM:da...@vybenetworks.com
VoIP: sip:da...@vex.net

Re: empty message-ID

[Benny Pedersen]

D'Arcy Cain skrev den 2020-11-23 15:18:


:0 Wh: msgid.lock
| formail -D 65536 $HOME/.msgid.cache

In other words, the message ID "" was considered a duplicate after the
first one.


if you use postfix there would be uniq msgid always, eq postfix ensures 
there is always fqdn in msgid aswell, many mua's does not ensure the 
fqdn part

Re: empty message-ID

[Wietse Venema]

Benny Pedersen:
> D'Arcy Cain skrev den 2020-11-23 15:18:
> 
> > :0 Wh: msgid.lock
> > | formail -D 65536 $HOME/.msgid.cache
> > 
> > In other words, the message ID "" was considered a duplicate after the
> > first one.
> 
> if you use postfix there would be uniq msgid always, eq postfix ensures 
> there is always fqdn in msgid aswell, many mua's does not ensure the 
> fqdn part

Postfix 2.6 and later don't add a Message-ID header, unless the
message comes from a "local" source. That header is a combination
of a time stamp and the Postfix $myhostname value, so it is unique
as long as both values are unique.

Wietse

Re: empty message-ID

[Benny Pedersen]

Wietse Venema skrev den 2020-11-23 17:10:


Postfix 2.6 and later don't add a Message-ID header, unless the
message comes from a "local" source. That header is a combination
of a time stamp and the Postfix $myhostname value, so it is unique
as long as both values are unique.


okay, what if msgid miss the @ charter ?

i remember postfix have configs for when @ is not part of msgid it can 
add @invalid.example.org so its diffrent if not sent local or remote

Re: empty message-ID

[Wietse Venema]

Benny Pedersen:
> Wietse Venema skrev den 2020-11-23 17:10:
> 
> > Postfix 2.6 and later don't add a Message-ID header, unless the
> > message comes from a "local" source. That header is a combination
> > of a time stamp and the Postfix $myhostname value, so it is unique
> > as long as both values are unique.
> 
> okay, what if msgid miss the @ charter ?

A message-id is not an email address.

> i remember postfix have configs for when @ is not part of msgid it can 
> add @invalid.example.org so its diffrent if not sent local or remote

Postfix can be configured to 'fix' an email address.

Wietse

remote_header_rewrite_domain (default: empty)
   Don't rewrite message headers from remote  clients  at  all  when  this
   parameter  is  empty; otherwise, rewrite message headers and append the
   specified domain name to incomplete  addresses.   The  local_header_re-
   write_clients parameter controls what clients Postfix considers local.

   Examples:

   The   safe   setting:  append  "domain.invalid"  to  incomplete  header
   addresses from remote SMTP clients, so that those addresses  cannot  be
   confused with local addresses.

   remote_header_rewrite_domain = domain.invalid

   The default, purist, setting: don't rewrite headers from remote clients
   at all.

   remote_header_rewrite_domain =


Wietse

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 06:49, maciejm  wrote:
> "RFC 822 Message-ID is not required"

RFC 822 has been obsoleted several times. 

RFC 5322 states:

   Though listed as optional in the table in section 3.6, every message
   SHOULD have a "Message-ID:" field.  Furthermore, reply messages
   SHOULD have "In-Reply-To:" and "References:" fields as appropriate
   and as described below.

And:

RFC 2119
SHOULD  This word, or the adjective "RECOMMENDED", mean that there
may exist valid reasons in particular circumstances to ignore a
particular item, but the full implications must be understood and
carefully weighed before choosing a different course.

So SHOULD is much stronger than "it's a good idea" and much more like "You 
better have a really good reason for ignoring this".

I would feel comfortable rejecting messages without a Message-ID.

-- 
Imagine all the people Sharing all the world

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 07:44, Jaroslaw Rafa  wrote:
> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
>> 
>> :0 Wh: msgid.lock
>> | formail -D 65536 $HOME/.msgid.cache
> 
> Who uses that?

Everyone who ever used procmail? Nearly everyone who ever used procmail?

It's even in the procmail man page.

>If you are subscribed to several mailinglists and people cross-post to
>some of them, you usually receive several duplicate mails (one from
>every list).  The following simple recipe eliminates duplicate mails.
>It tells formail to keep an 8KB cache file in which it will store the
>Message-IDs of the most recent mails you received.  Since Message-IDs
>are guaranteed to be unique for every new mail, they are ideally suited
>to weed out duplicate mails.  Simply put the following recipe at the
>top of your rcfile, and no duplicate mail will get past it.
> 
>   :0 Wh: msgid.lock
>   | formail -D 8192 msgid.cache


-- 
All our loves are first loves

Re: empty message-ID

[Bob Proulx]

Jaroslaw Rafa wrote:
> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
> > After the first message was accepted all of the rest
> > were silently dropped as duplicates due to a very standard procmail
> > recipe:
> > 
> > :0 Wh: msgid.lock
> > | formail -D 65536 $HOME/.msgid.cache
> 
> Who uses that? It's not normal to get email duplicates and it usually
> means that mail system is not functioning properly. They should find the
> cause of the duplicates and eliminate it instead of hiding symptoms...

Although I have been using procmail since the inception of it I have
always found this type rule problematic.  Because for me it keeps the
wrong message.  If I am sent a direct copy and a mailing list copy
then the copy I want is the mailing list copy.

But so many people use Gmail these days that they have gotten used to
the way Gmail does things.  And Gmail de-duplicates and saves the
first message with any particular message-id that arrives.  And then
displays a "mailbox" showing a view of the current tag being
displayed.  It's a very different paradigm from having separate
mailbox folders for different topics.  Gmail has one mailbox for
everything and multiple tags are possible on each message and only
displays the current display tag view of the mailbox.  And since it is
one mailbox it de-duplicates by only showing the first message-id.
And people have gotten used to that paradigm.  But it does cause some
odd behavior when dealing with mailing lists.

Bob

Re: empty message-ID

[Erwan David]

Le 23/11/2020 à 20:16, @lbutlr a écrit :
> On 23 Nov 2020, at 06:49, maciejm  wrote:
>> "RFC 822 Message-ID is not required"
> RFC 822 has been obsoleted several times. 
>
> RFC 5322 states:
>
>Though listed as optional in the table in section 3.6, every message
>SHOULD have a "Message-ID:" field.  Furthermore, reply messages
>SHOULD have "In-Reply-To:" and "References:" fields as appropriate
>and as described below.
>
> And:
>
> RFC 2119
> SHOULDThis word, or the adjective "RECOMMENDED", mean that there
>   may exist valid reasons in particular circumstances to ignore a
>   particular item, but the full implications must be understood and
>   carefully weighed before choosing a different course.
>
> So SHOULD is much stronger than "it's a good idea" and much more like "You 
> better have a really good reason for ignoring this".
>
> I would feel comfortable rejecting messages without a Message-ID.
>

Maybe on smtp, but not on submission. FOr me policy there is completeley
different

Re: empty message-ID

[Richard Damon]

On 11/23/20 3:34 PM, Erwan David wrote:
> Le 23/11/2020 à 20:16, @lbutlr a écrit :
>> On 23 Nov 2020, at 06:49, maciejm  wrote:
>>> "RFC 822 Message-ID is not required"
>> RFC 822 has been obsoleted several times. 
>>
>> RFC 5322 states:
>>
>>Though listed as optional in the table in section 3.6, every message
>>SHOULD have a "Message-ID:" field.  Furthermore, reply messages
>>SHOULD have "In-Reply-To:" and "References:" fields as appropriate
>>and as described below.
>>
>> And:
>>
>> RFC 2119
>> SHOULD   This word, or the adjective "RECOMMENDED", mean that there
>>  may exist valid reasons in particular circumstances to ignore a
>>  particular item, but the full implications must be understood and
>>  carefully weighed before choosing a different course.
>>
>> So SHOULD is much stronger than "it's a good idea" and much more like "You 
>> better have a really good reason for ignoring this".
>>
>> I would feel comfortable rejecting messages without a Message-ID.
>>
> Maybe on smtp, but not on submission. FOr me policy there is completeley
> different

I thought one strategy to handle this was that submission would detect
lack of the message-id header and add one with a proper message-id.

-- 
Richard Damon

Re: empty message-ID

[Jaroslaw Rafa]

Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
> 
> If someone replies to a mailing list and copies the sender then that
> person gets two copies.  The above recipe avoids that.

If someone gets two copies - a direct one and the mailing list one - then
he/she knows that the sender has replied both to author and to the list and
can instruct the sender not to do it. With the above recipe, the recipient
doesn't even know about that.

Moreover, it breaks the continuity of threads on mailing lists, because it's
unpredictable which copy will arrive first, and if only the direct copy is
left, the reply will go only to the sender and not to the mailing list. Thus
some messages are missing from lists.

> People also send to every alias that someone has.  Example,
> billing@, admin@, support@, joe@, etc.

But it's usually one message with multiple recipients, and if all these
recipients "resolve" to the same final destination, the receiving MTA
usually avoids creating duplicates. At least that was always the case for me
as the recipient, no matter if I was using sendmail, Exim or Postfix for my
mail.


Dnia 23.11.2020 o godz. 12:22:37 @lbutlr pisze:
> 
> Everyone who ever used procmail? Nearly everyone who ever used procmail?
> 
> It's even in the procmail man page.

Yes it is, but I never saw any real use case for this.


Dnia 23.11.2020 o godz. 13:24:13 Bob Proulx pisze:
> 
> Although I have been using procmail since the inception of it I have
> always found this type rule problematic.  Because for me it keeps the
> wrong message.  If I am sent a direct copy and a mailing list copy
> then the copy I want is the mailing list copy.
> 
> But so many people use Gmail these days that they have gotten used to
> the way Gmail does things.
[...]

+1 ;)
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: empty message-ID

[Richard Damon]

On 11/23/20 5:27 PM, Jaroslaw Rafa wrote:
> Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
>> If someone replies to a mailing list and copies the sender then that
>> person gets two copies.  The above recipe avoids that.
> If someone gets two copies - a direct one and the mailing list one - then
> he/she knows that the sender has replied both to author and to the list and
> can instruct the sender not to do it. With the above recipe, the recipient
> doesn't even know about that.
>
> Moreover, it breaks the continuity of threads on mailing lists, because it's
> unpredictable which copy will arrive first, and if only the direct copy is
> left, the reply will go only to the sender and not to the mailing list. Thus
> some messages are missing from lists.

You CAN still reply to the list from the private copy, you won't have a
'Reply-to-List' opiton, because of the lack of list headers, but
'Reply-All' will still work.

It just becomes a bit harder to reply back JUST to the list. Your need
Reply-All and then editing the list of recipients.



-- 
Richard Damon

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 13:34, Erwan David  wrote:
> Le 23/11/2020 à 20:16, @lbutlr a écrit :
>> I would feel comfortable rejecting messages without a Message-ID.

> Maybe on smtp, but not on submission. FOr me policy there is completeley
> different

On submission postfix adds the message ID as is proper if the MUA hasn't added 
it.

-- 
'Now what?' it said. IT'S UP TO YOU. IT'S ALWAYS UP TO YOU.
--Maskerade

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 13:24, Bob Proulx  wrote:
> Jaroslaw Rafa wrote:
>> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
>>> After the first message was accepted all of the rest
>>> were silently dropped as duplicates due to a very standard procmail
>>> recipe:
>>> 
>>> :0 Wh: msgid.lock
>>> | formail -D 65536 $HOME/.msgid.cache
>> 
>> Who uses that? It's not normal to get email duplicates and it usually
>> means that mail system is not functioning properly. They should find the
>> cause of the duplicates and eliminate it instead of hiding symptoms...
> 
> Although I have been using procmail since the inception of it I have
> always found this type rule problematic.  Because for me it keeps the
> wrong message.  If I am sent a direct copy and a mailing list copy
> then the copy I want is the mailing list copy.

Since my list filters looked at the to and CC headers this wasn't an issue.

> But so many people use Gmail these days that they have gotten used to
> the way Gmail does things.  And Gmail de-duplicates and saves the
> first message with any particular message-id that arrives.  And then
> displays a "mailbox" showing a view of the current tag being
> displayed.  It's a very different paradigm from having separate
> mailbox folders for different topics.

I don't use Gmail much, but my primary sort criteria now is a series of smart 
searches in Mail.app.

List mail al goes into a single "List" mailbox, then a smart search shows me, 
for example, all the postfix-users messages as a virtual mailbox.

The means that I rarely see the messages sent directly to me instead of the 
list because I don't generally read the "list" mailbox directly. Every now and 
then I archive all the list messages which takes them out of the smart 
mailboxes. (I could automate that action, but I haven't for reasons).

-- 
'Witches just aren't like that,' said Magrat. 'We live in harmony
with the great cycles of Nature, and do no harm to anyone, and
it's wicked of them to say we don't. We ought to fill their bones
with hot lead.'

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 15:27, Jaroslaw Rafa  wrote:
> Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
>> 
>> If someone replies to a mailing list and copies the sender then that
>> person gets two copies.  The above recipe avoids that.

> Moreover, it breaks the continuity of threads on mailing lists, because it's
> unpredictable which copy will arrive first, and if only the direct copy is
> left, the reply will go only to the sender and not to the mailing list. Thus
> some messages are missing from lists.

This is not accurate. First, the direct message almost certainly arrives first. 
Second of all, that message still has headers indicating it was sent to the 
mailing list. Third, whether your client gets confused about threading is a 
client issue, not an issue of the mail message. (My client does not get 
confused and the messages end up in the same virtual mailbox regardless).

>> People also send to every alias that someone has.  Example,
>> billing@, admin@, support@, joe@, etc.

> But it's usually one message with multiple recipients, and if all these
> recipients "resolve" to the same final destination, the receiving MTA
> usually avoids creating duplicates. At least that was always the case for me
> as the recipient, no matter if I was using sendmail, Exim or Postfix for my
> mail.

We are talking about duplicated messages with the same message-id. That is one 
message with multiple recipients. If they were separate messages, they would 
have unique message-id headers.

-- 
"Are you pondering what I'm pondering?"
"I think so, Brain, but why would anyone want a depressed tongue?"

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 15:40, Richard Damon  wrote:
> On 11/23/20 5:27 PM, Jaroslaw Rafa wrote:
>> Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
>>> If someone replies to a mailing list and copies the sender then that
>>> person gets two copies.  The above recipe avoids that.
>> If someone gets two copies - a direct one and the mailing list one - then
>> he/she knows that the sender has replied both to author and to the list and
>> can instruct the sender not to do it. With the above recipe, the recipient
>> doesn't even know about that.
>> 
>> Moreover, it breaks the continuity of threads on mailing lists, because it's
>> unpredictable which copy will arrive first, and if only the direct copy is
>> left, the reply will go only to the sender and not to the mailing list. Thus
>> some messages are missing from lists.
> 
> You CAN still reply to the list from the private copy, you won't have a
> 'Reply-to-List' opiton, because of the lack of list headers, but
> 'Reply-All' will still work.
> 
> It just becomes a bit harder to reply back JUST to the list. Your need
> Reply-All and then editing the list of recipients.

Or you use procmail/Sieve to add a reply-to header to messages that have the 
mailing list email in the headers.


-- 
"A common mistake people make when trying to design something
completely foolproof is to underestimate the ingenuity of
complete fools.: - Douglas Adams

Re: empty message-ID

[Bob Proulx]

@lbutlr wrote:
> On 23 Nov 2020, at 15:27, Jaroslaw Rafa  wrote:
> > Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
> >> 
> >> If someone replies to a mailing list and copies the sender then that
> >> person gets two copies.  The above recipe avoids that.
> 
> > Moreover, it breaks the continuity of threads on mailing lists, because it's
> > unpredictable which copy will arrive first, and if only the direct copy is
> > left, the reply will go only to the sender and not to the mailing list. Thus
> > some messages are missing from lists.
> 
> This is not accurate. First, the direct message almost certainly
> arrives first. Second of all, that message still has headers
> indicating it was sent to the mailing list.

That is not accurate.  The direct message never went through the
mailing list.  The direct copy is missing the mailing list headers.
For this list the direct copy is missing these headers.
 
Sender: owner-postfix-us...@postfix.org
Precedence: bulk
List-Id: Postfix users 
List-Post: 
List-Help: 
List-Unsubscribe: 
List-Subscribe: 

The most important of those are List-Id and List-Post without which
the message will not be filed correctly and most mailers will not be
able to list reply correctly.  This puts the onus upon the receiver to
manually take corrective action with the message.  That is something
that I and probably most readers of this list can do.  But for most
random people today they do not understand email and most people today
do not have the skill to do this correctly.  For them it is simply
completely broken.

> >> People also send to every alias that someone has.  Example,
> >> billing@, admin@, support@, joe@, etc.
> 
> > But it's usually one message with multiple recipients, and if all these
> > recipients "resolve" to the same final destination, the receiving MTA
> > usually avoids creating duplicates. At least that was always the case for me
> > as the recipient, no matter if I was using sendmail, Exim or Postfix for my
> > mail.
> 
> We are talking about duplicated messages with the same
> message-id. That is one message with multiple recipients. If they
> were separate messages, they would have unique message-id headers.

That is not accurate.  A single message to multiple recipients will
have one Message-Id.  If you receive it by being the target of some of
those multiple recipients then you will receive multiple copies of the
message and all of the copies will have the same Message-Id.

Bob

Re: empty message-ID

[Bob Proulx]

@lbutlr wrote:
> Bob Proulx wrote:
> > But so many people use Gmail these days that they have gotten used to
> > the way Gmail does things.  And Gmail de-duplicates and saves the
> > first message with any particular message-id that arrives.  And then
> > displays a "mailbox" showing a view of the current tag being
> > displayed.  It's a very different paradigm from having separate
> > mailbox folders for different topics.
> 
> I don't use Gmail much, but my primary sort criteria now is a series
> of smart searches in Mail.app.

I don't use Gmail at all.  But the people who do not use Gmail are in
the minority.  It's the world we live in now.  I have to know how
Gmail works even though I don't use it because almost everyone else
around me uses it.

Bob

Re: empty message-ID

[Jaroslaw Rafa]

Dnia 23.11.2020 o godz. 16:29:13 @lbutlr pisze:
> 
> This is not accurate. First, the direct message almost certainly arrives
> first.

Unless you use greylisting :) - in that case list message usually arrives
first, as the direct message is generally from previously unknown sender and
has to wait. It happened for me multiple times.

> Third, whether your client gets confused about
> threading is a client issue, not an issue of the mail message.  (My client
> does not get confused and the messages end up in the same virtual mailbox
> regardless).

I'm not talking about client getting confused about threading. I'm talking
about the fact that when you get only an off-list message and reply to it,
the reply goes only to the original sender and not to the list, thus
messages are missing from thread on the list (thread understood as an
abstract concept, not a particular implementation in this or that client).
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: empty message-ID

[Richard Damon]

On 11/24/20 1:51 AM, Bob Proulx wrote:
> @lbutlr wrote:
>> On 23 Nov 2020, at 15:27, Jaroslaw Rafa  wrote:
>>> Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
 If someone replies to a mailing list and copies the sender then that
 person gets two copies.  The above recipe avoids that.
>>> Moreover, it breaks the continuity of threads on mailing lists, because it's
>>> unpredictable which copy will arrive first, and if only the direct copy is
>>> left, the reply will go only to the sender and not to the mailing list. Thus
>>> some messages are missing from lists.
>> This is not accurate. First, the direct message almost certainly
>> arrives first. Second of all, that message still has headers
>> indicating it was sent to the mailing list.
> That is not accurate.  The direct message never went through the
> mailing list.  The direct copy is missing the mailing list headers.
> For this list the direct copy is missing these headers.
>  
> Sender: owner-postfix-us...@postfix.org
> Precedence: bulk
> List-Id: Postfix users 
> List-Post: 
> List-Help: 
> List-Unsubscribe: 
> List-Subscribe: 
>
> The most important of those are List-Id and List-Post without which
> the message will not be filed correctly and most mailers will not be
> able to list reply correctly.  This puts the onus upon the receiver to
> manually take corrective action with the message.  That is something
> that I and probably most readers of this list can do.  But for most
> random people today they do not understand email and most people today
> do not have the skill to do this correctly.  For them it is simply
> completely broken.

Slight details, those headers indicate that it is FROM the mailing list,
the direct message will still have the To: or Cc: header indicating that
it was TO the mailing list.

That says that a Reply-All will still go to the list (and send a direct
copy back to the person who sent this to you, which might be what they
want).

How the message get filed will depend on how you have your filters setup
for the messages, and the difference is perhaps useful. After all, a
reply to your posting might be considered of higher importance than just
a run of the mill posting to the list, so not filing it the same might
be correct.

One way to make a personal reply less likely would be for you to add a
Reply-To: header pointing to the list submission address, that way if
someone is using a MUA that doesn't support the 'Reply-To-List' function
and does a Reply-All, it is likely that it will redirect the reply to
the list. (More broken MUAs might still send you a copy, if the ignore
or mishandle Reply-All.)

-- 
Richard Damon

Re: empty message-ID

[Matus UHLAR - fantomas]

Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:

If someone replies to a mailing list and copies the sender then that
person gets two copies.  The above recipe avoids that.



On 11/23/20 5:27 PM, Jaroslaw Rafa wrote:

If someone gets two copies - a direct one and the mailing list one - then
he/she knows that the sender has replied both to author and to the list and
can instruct the sender not to do it. With the above recipe, the recipient
doesn't even know about that.

Moreover, it breaks the continuity of threads on mailing lists, because it's
unpredictable which copy will arrive first, and if only the direct copy is
left, the reply will go only to the sender and not to the mailing list. Thus
some messages are missing from lists.



On 23 Nov 2020, at 15:40, Richard Damon  wrote:

You CAN still reply to the list from the private copy, you won't have a
'Reply-to-List' opiton, because of the lack of list headers, but
'Reply-All' will still work.

It just becomes a bit harder to reply back JUST to the list. Your need
Reply-All and then editing the list of recipients.


On 23.11.20 16:30, @lbutlr wrote:

Or you use procmail/Sieve to add a reply-to header to messages that have
the mailing list email in the headers.


It's silly to set up procmail rule to remove "duplicate" message and then
set up rule to add headers to the mail that are missing in one of the
duplicates.

note that it's possible to Bcc: message to mailing list so it does not
contain list address in To:/Cc:

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
"The box said 'Requires Windows 95 or better', so I bought a Macintosh".

Re: empty message-ID

[Matus UHLAR - fantomas]

On 24.11.20 13:50, Matus UHLAR - fantomas wrote:

note that it's possible to Bcc: message to mailing list so it does not
contain list address in To:/Cc:


... as this message clearly shows.

I set mailing lists not to avoid duplicate messages (and usually drop direct
mail).


However, this thread is off-topic. We should close it with conclusion:

don't avoid duplicate mail based on Message-Id:


--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Depression is merely anger without enthusiasm.

Re: empty message-ID

[@lbutlr]

On 24 Nov 2020, at 05:50, Matus UHLAR - fantomas  wrote:
> On 23.11.20 16:30, @lbutlr wrote:
>> Or you use procmail/Sieve to add a reply-to header to messages that have
>> the mailing list email in the headers.

> It's silly to set up procmail rule to remove "duplicate" message and then
> set up rule to add headers to the mail that are missing in one of the
> duplicates.

If you say so. I do not find it silly at all. The first eliminates unwanted and 
unneeded duplicate messages, the second puts messages where they belong. I see 
no reason to store duplicate messages, but I do want messages to the mailing 
list to be show with the other messages to the mailing list, and I often have 
to fix mailing lists so that the reply goes to the list anyway, so it's not 
even an extra step.

> note that it's possible to Bcc: message to mailing list so it does not
> contain list address in To:/Cc:

Depends on the mailing list, many (most? Some?) mailing lists reject messages 
if the list address is not in the To or CC.

-- 
A sadder and a wiser man he rose the morrow morn.

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 23:51, Bob Proulx  wrote:
> @lbutlr wrote:
>> On 23 Nov 2020, at 15:27, Jaroslaw Rafa  wrote:
>>> Dnia 23.11.2020 o godz. 11:49:39 D'Arcy Cain pisze:
 
 If someone replies to a mailing list and copies the sender then that
 person gets two copies.  The above recipe avoids that.
>> 
>>> Moreover, it breaks the continuity of threads on mailing lists, because it's
>>> unpredictable which copy will arrive first, and if only the direct copy is
>>> left, the reply will go only to the sender and not to the mailing list. Thus
>>> some messages are missing from lists.
>> 
>> This is not accurate. First, the direct message almost certainly
>> arrives first. Second of all, that message still has headers
>> indicating it was sent to the mailing list.
> 
> That is not accurate.

Of course it is accurate.

It will have the list address in the To or CC field.

> The most important of those are List-Id and List-Post without which
> the message will not be filed correctly

That is certainly not the case. It is COMMON to check list-id, but it is 
trivial to match messages based on the To and/or CC. ESPECIALLY if you have an 
email address that is used only for mailing lists, as I do. I never said the 
CCed message would cantina the list headers, only that it "still has headers 
indicating it was sent to the mailing list."

> and most mailers will not be able to list reply correctly.

Again, not at all accurate. This is all managed automatically by sieve and it 
used to be managed automatically by procmail.

> This puts the onus upon the receiver to manually take corrective action with 
> the message.

Not at all.

>  That is something
> that I and probably most readers of this list can do.  But for most
> random people today they do not understand email and most people today
> do not have the skill to do this correctly.  For them it is simply
> completely broken.

When did we start talking about most people? We were talking about procmail 
recipes to eliminate duplicate messages. That already narrows the field to a 
tiny fraction of a percent of the people getting email.

>> We are talking about duplicated messages with the same
>> message-id. That is one message with multiple recipients. If they
>> were separate messages, they would have unique message-id headers.
> 
> That is not accurate.

Yes it is accurate.

>  A single message to multiple recipients will
> have one Message-Id.  If you receive it by being the target of some of
> those multiple recipients then you will receive multiple copies of the
> message and all of the copies will have the same Message-Id.

Yes. That is EXACTLY what I said: "We are talking about duplicated messages 
with the same message-id. That is one message with multiple recipients."

-- 
*** AgentSmith sets mode: +m

Re: empty message-ID

[@lbutlr]

On 23 Nov 2020, at 23:55, Bob Proulx  wrote:
> @lbutlr wrote:
>> Bob Proulx wrote:
>>> But so many people use Gmail these days that they have gotten used to
>>> the way Gmail does things.  And Gmail de-duplicates and saves the
>>> first message with any particular message-id that arrives.  And then
>>> displays a "mailbox" showing a view of the current tag being
>>> displayed.  It's a very different paradigm from having separate
>>> mailbox folders for different topics.
>> 
>> I don't use Gmail much, but my primary sort criteria now is a series
>> of smart searches in Mail.app.
> 
> I don't use Gmail at all.  But the people who do not use Gmail are in
> the minority.  It's the world we live in now.  I have to know how
> Gmail works even though I don't use it because almost everyone else
> around me uses it.

That's OK, most the people using gmail don't know how it works either.

"Why is this message in my Inbox when I tagged it as being 'Bank mail'?"

And that's an advanced gmail user who uses tags.

-- 
You are in my inappropriate thoughts

Re: empty message-ID

[@lbutlr]

On 24 Nov 2020, at 02:44, Jaroslaw Rafa  wrote:
> Dnia 23.11.2020 o godz. 16:29:13 @lbutlr pisze:
>> 
>> This is not accurate. First, the direct message almost certainly arrives
>> first.
> 
> Unless you use greylisting

That's a whole different issue and anyone using greylisting now I would never 
use for email. But it doesn’t matter which message arrives at my server first, 
the duplicate gets trashed and the first one goes where it should.

> I'm not talking about client getting confused about threading. I'm talking
> about the fact that when you get only an off-list message and reply to it,
> the reply goes only to the original sender and not to the list,

No, that is not the case. When I get a message from bob that was also sent to 
postfix-user several things happen.

First, the message has a reply-to: header added for the postfix list. All 
mailing lists get a reply-to header added with the list address.

Second, the message has a "X-List-name: postfix" added to it regardless of who 
sent it.

Then, on my client, all the messages that are "X-List-name: Postfix" and are 
not archived are shown in a smart mailbox.

> thus messages are missing from thread on the list (thread understood as an
> abstract concept, not a particular implementation in this or that client).

That's not how it works here, and not how it's worked here since I started 
using procmail in the late 90s, though now I use sieve.

-- 
"I can't see the point in the theatre. All that sex and violence. I
get enough of that at home. Apart from the sex, of course." -
Baldrick

Re: empty message-ID

[Jaroslaw Rafa]

Dnia 24.11.2020 o godz. 12:20:23 @lbutlr pisze:
> > I'm not talking about client getting confused about threading. I'm talking
> > about the fact that when you get only an off-list message and reply to it,
> > the reply goes only to the original sender and not to the list,
> 
> No, that is not the case. When I get a message from bob that was also sent
> to postfix-user several things happen.
> 
> First, the message has a reply-to: header added for the postfix list. All
> mailing lists get a reply-to header added with the list address.
> 
> Second, the message has a "X-List-name: postfix" added to it regardless of
> who sent it.

Only the copy that went through the mailing list has those, because they are
added by the list management software and not by sender's email client. So
the copy that was sent off-list directly to you doesn't have any of them
(unless you add them yourself on your side uning eg. some filtering rule). 
So, if the list copy comes first, the direct copy is deleted and everything
is ok. But if the direct copy comes first (and thus list copy is deleted),
it doesn't have those headers. So if you reply to it, the copy goes only to
Bob and not to list.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: empty message-ID

[@lbutlr]

On 24 Nov 2020, at 13:57, Jaroslaw Rafa  wrote:
> Dnia 24.11.2020 o godz. 12:20:23 @lbutlr pisze:
>>> I'm not talking about client getting confused about threading. I'm talking
>>> about the fact that when you get only an off-list message and reply to it,
>>> the reply goes only to the original sender and not to the list,
>> 
>> No, that is not the case. When I get a message from bob that was also sent
>> to postfix-user several things happen.
>> 
>> First, the message has a reply-to: header added for the postfix list. All
>> mailing lists get a reply-to header added with the list address.
>> 
>> Second, the message has a "X-List-name: postfix" added to it regardless of
>> who sent it.
> 
> Only the copy that went through the mailing list has those,

No.

Please re-read what I wrote.

-- 
"A synonym is a word you use when you can't spell the word you first
thought of." - Burt Bacharach

Re: empty message-ID

[Jaroslaw Rafa]

Dnia 24.11.2020 o godz. 14:37:13 @lbutlr pisze:
> > Only the copy that went through the mailing list has those,
> 
> No.
> 
> Please re-read what I wrote.

So, I looked through my archives and found an actual message from this
mailing list, that someone sent both to me and to list. Below are the actual
headers in their entirety - sorry for posting long uninteresting stuff to
the list, but seems there's no other way to demonstrate what I mean. :(

First the message that went through mailing list:

Return-Path: 
X-Original-To: r...@rafa.eu.org
Delivered-To: r...@rafa.eu.org
Received: from russian-caravan.cloud9.net (russian-caravan.cloud9.net 
[168.100.1.4])
(using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits))
(No client certificate requested)
by rafa.eu.org (Postfix) with ESMTPS id AAECD40F06
for ; Sat,  6 Jun 2020 19:28:24 +0200 (CEST)
Received: by russian-caravan.cloud9.net (Postfix)
id 7A5FC3426E0; Sat,  6 Jun 2020 13:27:45 -0400 (EDT)
Delivered-To: postfix-users-outgo...@cloud9.net
Received: from localhost (localhost [127.0.0.1])
by russian-caravan.cloud9.net (Postfix) with ESMTP id 76C793426DA
for ; Sat,  6 Jun 2020 13:27:45 
-0400 (EDT)
X-Virus-Scanned: amavisd-new at cloud9.net
Received: from russian-caravan.cloud9.net ([127.0.0.1])
by localhost (russian-caravan.cloud9.net [127.0.0.1]) (amavisd-new, 
port 10024)
with ESMTP id vL9E6T48R-KU for ;
Sat,  6 Jun 2020 13:27:45 -0400 (EDT)
Received: by russian-caravan.cloud9.net (Postfix, from userid 54)
id 543BE3426E2; Sat,  6 Jun 2020 13:27:45 -0400 (EDT)
Delivered-To: postfix-us...@cloud9.net
Received: from localhost (localhost [127.0.0.1])
by russian-caravan.cloud9.net (Postfix) with ESMTP id 1974F3426E0
for ; Sat,  6 Jun 2020 13:27:45 -0400 (EDT)
X-Virus-Scanned: amavisd-new at cloud9.net
Received: from russian-caravan.cloud9.net ([127.0.0.1])
by localhost (russian-caravan.cloud9.net [127.0.0.1]) (amavisd-new, 
port 10024)
with ESMTP id 1cuf_UbXmvqQ for ;
Sat,  6 Jun 2020 13:27:44 -0400 (EDT)
Received: from mx.xenaura.com (mx.xenaura.com [104.131.165.124])
by russian-caravan.cloud9.net (Postfix) with ESMTP id EA74A3426DA
for ; Sat,  6 Jun 2020 13:27:44 -0400 (EDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=sfina.com;
s=201605sfinacom; t=1591464461;
bh=sHbDD9R8M0N5Gew8gWUaOzwtex+Vk2SMhq9JdsEVnoU=;
h=Subject:From:To:Date:In-Reply-To:References:From;
b=pAJkiG3foTHhzylRoqaz5Bb7AelBoUxN/HR+samlGKsfamdhL51Cfa63E/l2H4c9o
/g1XmpfN1DSvpukkrH+HSRLNZSnMtaCwvy6xgJHN1TVukw8Kg4G1o3KHxjNsJK1PuR
9YvQIagrGXSV1HVOyXxSWBpunNKSNgCejHTa4DBEhxkfuqEMBtyGm1ozF624cX5pDU
PdQ1A+vi5ednV+mn4nchPHQMmihiClV7A23X2QXbAa/yHKs1WZmBRM9VNEQJNu1DS/
DoGJiu0KLohLmS0aW7J7Mv6hvR+GSe8dU3WFCrfW4yMCLLtFST/VRENy/ZupFk8zUP
Ds+mU6sfTat7g==
Message-ID: 
Subject: Re: The historical roots of our computer terms
From: yuv 
To: Jaroslaw Rafa , postfix-users@postfix.org
Date: Sat, 06 Jun 2020 13:27:43 -0400
In-Reply-To: <20200606171208.gb22...@rafa.eu.org>
References:

<20200606171208.gb22...@rafa.eu.org>
Content-Type: text/plain; charset="UTF-8"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Sender: owner-postfix-us...@postfix.org
Precedence: bulk
List-Id: Postfix users 
List-Post: 
List-Help: 
List-Unsubscribe: 
List-Subscribe: 
X-Spam-Flag: NO
X-Spam-Status: No, score=0.0 required=3.5 tests=[LOCAL_PRECEDENCE_BULK=0.01,
T_DKIM_INVALID=0.01] bayes=0. autolearn=ham autolearn_force=no
X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on rafa
Content-Length: 1521

It does have the "List-Id:" header, it doesn't have "Reply-To:" as this
list, by the administrators' choice, does not set "Reply-To:" to the list.
But this list is specific, most lists set "Reply-To:" to the list address
and in that case the "Reply-To:" header would be there.

And then the copy that went directly to me:

Return-Path: 
X-Original-To: r...@rafa.eu.org
Delivered-To: r...@rafa.eu.org
X-Greylist: delayed 453 seconds by postgrey-1.34 at rafa; Sat, 06 Jun 2020 
19:35:19 CEST
Received: from mx.xenaura.com (mx.xenaura.com [104.131.165.124])
by rafa.eu.org (Postfix) with ESMTP id 57A7740F06
for ; Sat,  6 Jun 2020 19:35:19 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=sfina.com;
s=201605sfinacom; t=1591464461;
bh=sHbDD9R8M0N5Gew8gWUaOzwtex+Vk2SMhq9JdsEVnoU=;
h=Subject:From:To:Date:In-Reply-To:References:From;
b=pAJkiG3foTHhzylRoqaz5Bb7AelBoUxN/HR+samlGKsfamdhL51Cfa63E/l2H4c9o
/g1XmpfN1DSvpukkrH+HSRLNZSnMtaCwvy6xgJHN1TVukw8Kg4G1o3KHxjNsJK1PuR
9YvQIagrGXSV1HVOyXxSWBpunNKSNgCejHTa4DBEhxkfuqEMBtyGm1ozF624cX5pDU
PdQ1A+vi5ednV+mn4nchPHQMmihiClV7A23X2QXbAa/yHKs1WZm

Re: empty message-ID

[@lbutlr]

On 24 Nov 2020, at 15:08, Jaroslaw Rafa  wrote:
> Dnia 24.11.2020 o godz. 14:37:13 @lbutlr pisze:
>>> Only the copy that went through the mailing list has those,
>> 
>> No.
>> 
>> Please re-read what I wrote.
> 
> So, I looked through my archives and found an actual message from this
> mailing list, that someone sent both to me and to list. Below are the actual
> headers in their entirety - sorry for posting long uninteresting stuff to
> the list, but seems there's no other way to demonstrate what I mean. :(

You could re-read what I wrote as I suggested instead of ignoring it completely 
and going off on some irrelevant tangent.

Here you go, with notes.

> When I get a message from bob that was also sent
> to postfix-user several things happen.

A message FROM BOB (as in not from the list)

> First, the message has a reply-to: header added for the postfix list. All

Note, the header is ADDED when I get a message FROM BOB.

> mailing lists get a reply-to header added with the list address.

Note, ALL messages directed to a mial list get a reply-to added when I receive 
them. Many mailing list do not set this header, or set the list up so that 
reply to user is the default.

> Second, the message has a "X-List-name: postfix" added to it regardless of
> who sent it.

Please not the word "added" and the phrase "regardless of who sent it"

> First the message that went through mailing list:

Note that the headers you include have NEITHER a reply-to header NOR an 
X-Listname header. Why? Because those are added BY MY MAILSERVER.

> To: Jaroslaw Rafa , postfix-users@postfix.org

[snip]

> Not any trace of mailing list-specific headers here.

Would you like to read the one header line I left intact and see it that might 
indicate the message was sent to a mailing list? Anything there? Anything at 
all?

> If I were an average mail user and replied to this message (by using

Not talking about average mail users, Never have been talking about average 
mail users. This started by mentioning that s lot of people using procmail 
setup their recipes to delete duplicate message. No average users use or ever 
used procmail.

> That's the difference I'm talking about all the time.

Here is the sieve block for this list, with comments so that maybe you will 
read what I wrote this time.

elsif anyof (header :contains "List-id" "postfix-users",
 address ["to", "cc"] "postfix-users@postfix.org") {

# Anyof means it the message has EITHER the first OR the second header (or 
both).

   deleteheader "Reply-To";
   addheader "Reply-To" "postfix-users@postfix.org";

# Adds the reply to header so that replies go to the list 
# and removes any previous reply-to header.

   addheader "X-Listname" "postfix";

# Adds the header for my sorting routines

   fileinto :create "lists";
stop;}


-- 
Eliot: We'll figure it out. We always do.
Margo: When it's a test to cheat on. Not when we're stuck in some epic fantasy
   that likes to behead heroes halfway through season one. If we even are
   heroes. We might be comic relief.

Re: empty message-ID

[Jaroslaw Rafa]

Dnia 24.11.2020 o godz. 16:25:54 @lbutlr pisze:
> 
> Note that the headers you include have NEITHER a reply-to header NOR an
> X-Listname header.  Why?  Because those are added BY MY MAILSERVER.
[...]
> Not talking about average mail users, Never have been talking about
> average mail users. This started by mentioning that s lot of people using
> procmail setup their recipes to delete duplicate message. No average
> users use or ever used procmail.

1) An average user does not use procmail, but their mail provider may
implement a de-duplication solution for them, without them even knowing -
Gmail has already been mentioned here in that context.

2) Even if someone uses procmail to de-duplicate messages, it does not imply
that they will also add additional headers like you do. Strictly speaking,
you do more than just de-duplication. You cannot assume that everybody who
uses procmail to de-duplicate mail does the same what you do.

That's the reason why I ignore the specifics of your setup and concentrate
on de-duplication in general. We are not talking about you specifically
here, we are talking on anybody that uses some solution that de-duplicates
messages based on Message-Id: header.

And without your specific setup, the problem does still exist, in both cases
mentioned above.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."

Re: empty message-ID

[@lbutlr]

On 24 Nov 2020, at 17:23, Jaroslaw Rafa  wrote:
> That's the reason why I ignore the specifics of your setup and concentrate
> on de-duplication in general.

Interesting rationalization. But sure, have it your way.

-- 
'A man like that could inspire a handful of broken men to conquer a
country.' 'Fine. Just so long as he does it on his day off.'

Re: empty message-ID

[Peter]

On 24/11/20 3:56 am, Wietse Venema wrote:

Jaroslaw Rafa:

Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:


I used to have a client who was not getting emails from one of his
friends.  Turned out that the friend's client/MUA was not adding the
message ID.


Doesn't Postfix automatically add Message-Id: header upon sending a message
if none is present?


For the last 17 years, Message-ID is added to "local" email only.

http://www.postfix.org/postconf.5.html#always_add_missing_headers


Might want to clarify that in cleanup(8):

   The cleanup(8) daemon always performs the following transformations:

   ·  Insert missing message headers: (Resent-) From:, To:, 
Message-Id:, and Date:.


...the explicit use of the word "always" there is misleading.


Peter

Re: empty message-ID

[Peter]

On 25/11/20 1:53 am, Matus UHLAR - fantomas wrote:

However, this thread is off-topic. We should close it with conclusion:

don't avoid duplicate mail based on Message-Id:


A better conclusion would be to not consider messages with a missing or 
empty Message-Id to be duplicates of each other.  If the Message-Id is 
present and the same then it's reasonable to assume that it's a duplicate.



peter

Re: empty message-ID

[Matus UHLAR - fantomas]

On 25/11/20 1:53 am, Matus UHLAR - fantomas wrote:

However, this thread is off-topic. We should close it with conclusion:

don't avoid duplicate mail based on Message-Id:


On 25.11.20 15:04, Peter wrote:
A better conclusion would be to not consider messages with a missing 
or empty Message-Id to be duplicates of each other.  If the Message-Id 
is present and the same then it's reasonable to assume that it's a 
duplicate.


different paths, different headers, different MUA response to them.
You avoid "duplicates" and miss important info.

Funny how half of this thread is discussing it.
You can do it yourself but don't complain then...


On 24 Nov 2020, at 05:50, Matus UHLAR - fantomas  wrote:

note that it's possible to Bcc: message to mailing list so it does not
contain list address in To:/Cc:


On 24.11.20 12:01, @lbutlr wrote:

Depends on the mailing list, many (most? Some?) mailing lists reject messages 
if the list address is not in the To or CC.


If this mailing list followed it, you wouldn't be able to reply my mail
- I only sent it to the list, while only your address was in the To:

--
Matus UHLAR - fantomas, uh...@fantomas.sk ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
Christian Science Programming: "Let God Debug It!".

Re: empty message-ID

[Wietse Venema]

Peter:
> On 24/11/20 3:56 am, Wietse Venema wrote:
> > Jaroslaw Rafa:
> >> Dnia 23.11.2020 o godz. 10:18:39 D'Arcy Cain pisze:
> >>>
> >>> I used to have a client who was not getting emails from one of his
> >>> friends.  Turned out that the friend's client/MUA was not adding the
> >>> message ID.
> >>
> >> Doesn't Postfix automatically add Message-Id: header upon sending a message
> >> if none is present?
> > 
> > For the last 17 years, Message-ID is added to "local" email only.
> > 
> > http://www.postfix.org/postconf.5.html#always_add_missing_headers
> 
> Might want to clarify that in cleanup(8):
> 
> The cleanup(8) daemon always performs the following transformations:
> 
> ?  Insert missing message headers: (Resent-) From:, To:, 
> Message-Id:, and Date:.
> 
> ...the explicit use of the word "always" there is misleading.

Thanks for the pointer. I am updating ths manpage.

Wietse

Re: empty message-ID

[Wietse Venema]

Wietse Venema:
> Peter:
> > Might want to clarify that in cleanup(8):
> > 
> > The cleanup(8) daemon always performs the following transformations:
> > 
> > ?  Insert missing message headers: (Resent-) From:, To:, 
> > Message-Id:, and Date:.
> > 
> > ...the explicit use of the word "always" there is misleading.
> 
> Thanks for the pointer. I am updating ths manpage.

Updated text: http://www.porcupine.org/postfix-mirror/cleanup.8.html
I'll fix some font inconsistencies later.

Wietse

Re: empty message-ID

[Peter]

On 26/11/20 5:03 am, Wietse Venema wrote:

Wietse Venema:

Peter:

Might want to clarify that in cleanup(8):

 The cleanup(8) daemon always performs the following transformations:

 ?  Insert missing message headers: (Resent-) From:, To:,
Message-Id:, and Date:.

...the explicit use of the word "always" there is misleading.


Thanks for the pointer. I am updating ths manpage.


Updated text: http://www.porcupine.org/postfix-mirror/cleanup.8.html
I'll fix some font inconsistencies later.


Much better, thanks.


Peter