[mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Frido Otten via mailop]

We're currently having issues when someone with a gmail address sends an 
email to a 3rd party email address which has a forward to another gmail 
address. These messages don't arrive in the final recipient mailbox, not 
even in the spam folder. The forwarded message from 3rd party domain to 
the gmail server is accepted. There's no bounce message or any evidence 
for the sender that delivery failed. I've tested this with or without 
SRS in place, but that makes no difference.
Is this behaviour intended by Google and is there a way around this 
(except by creating a maillist)?


Frido

___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Taavi Eomäe via mailop]

Do those forwarded letters without SRS have intact DKIM?

A second thing you can try is ARC (without the SRS), I've gotten the 
impression that Gmail "likes" original letters (and ARC) more than it 
likes any kind of mangling, including SRS.




smime.p7s
Description: S/MIME Cryptographic Signature
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Jaroslaw Rafa via mailop]

Dnia 17.05.2023 o godz. 10:53:58 Frido Otten via mailop pisze:
> We're currently having issues when someone with a gmail address
> sends an email to a 3rd party email address which has a forward to
> another gmail address. These messages don't arrive in the final
> recipient mailbox, not even in the spam folder. The forwarded
> message from 3rd party domain to the gmail server is accepted.
> There's no bounce message or any evidence for the sender that
> delivery failed. I've tested this with or without SRS in place, but
> that makes no difference.
> Is this behaviour intended by Google and is there a way around this
> (except by creating a maillist)?

I have an intuitive impression that this may be related to Gmail's "feature"
which causes your messages that you send to a mailing list not arrive back
in your mailbox (this behavior applies to any mailing list, either hosted on
Google Groups or anywhere else). Maybe a faulty implementation of this
"feature" (basically, a very wrong feature if you ask me, especially that
you can't turn it off) causes the disappearance of messages you are talking
about.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Al Iverson via mailop]

Try rewriting the message ID. I think Gmail is believing the message
to be a duplicate and in some cases it will silently eat duplicates.

I have a forwarding feature for my own domains that in addition to
using SRS and re-signing DKIM with my domain, it rewrites the message
ID.

Good point about ARC; I don't have proof that it would fix things but
it would potentially be something to try.

Cheers,
Al

On Wed, May 17, 2023 at 4:23 AM Taavi Eomäe via mailop
 wrote:
>
> Do those forwarded letters without SRS have intact DKIM?
>
> A second thing you can try is ARC (without the SRS), I've gotten the
> impression that Gmail "likes" original letters (and ARC) more than it
> likes any kind of mangling, including SRS.
>
> ___
> mailop mailing list
> mailop@mailop.org
> https://list.mailop.org/listinfo/mailop



-- 

Al Iverson / Deliverability blogging at www.spamresource.com
Subscribe to the weekly newsletter at wombatmail.com/sr.cgi
DNS Tools at xnnd.com / (312) 725-0130 / Chicago (Central Time)
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Bob Proulx via mailop]

> Frido Otten pisze:
> > We're currently having issues when someone with a gmail address
> > sends an email to a 3rd party email address which has a forward to
> > another gmail address. These messages don't arrive in the final
> > recipient mailbox, not even in the spam folder. The forwarded
> > message from 3rd party domain to the gmail server is accepted.
> > There's no bounce message or any evidence for the sender that
> > delivery failed. I've tested this with or without SRS in place, but
> > that makes no difference.
> > Is this behaviour intended by Google and is there a way around this
> > (except by creating a maillist)?

Al Iverson via mailop wrote:
> Try rewriting the message ID. I think Gmail is believing the message
> to be a duplicate and in some cases it will silently eat duplicates.

Jaroslaw Rafa wrote:
> I have an intuitive impression that this may be related to Gmail's "feature"
> which causes your messages that you send to a mailing list not arrive back
> in your mailbox (this behavior applies to any mailing list, either hosted on
> Google Groups or anywhere else). Maybe a faulty implementation of this
> "feature" (basically, a very wrong feature if you ask me, especially that
> you can't turn it off) causes the disappearance of messages you are talking
> about.

This feels like the Gmail feature as noted.  But I believe Google has
implemented it in a slightly different way from the impression I get
from the above.  Please pardon me if I give a somewhat different
description of what I think is happening.

Google's Gmail has one mail folder with all messages in it.  Including
sent messages.  It tags outgoing messages with "Sent".  Therefore the
message does already exist in the Gmail mail folder.  It existed there
immediately when the user sent it.  And it does not show up in the
Inbox view because the message is tagged Sent and not tagged Inbox.

Because the Message-Id exists in the user's mail folder already the
message is accepted and then discarded immediately as a duplicate of
the message that is already present.

The message sent is tagged "Sent" in the mail folder.  It does not get
tagged "Inbox" unless the user writes a special rule to tag outgoing
messages with an Inbox tag.  I have heard that some people do write a
rule to add an Inbox tag to outgoing messages.  (I am not a Gmail user
so doing this is left as an exercise for those interested.)  If one
looks at messages tagged as Sent then they will find their copy of the
message.  But one can only assume that it might have actually been
delivered to the recipients listed since the local copy didn't make
that trip.

Then for mailing list messages, which have a similar behavior, the
exact same thing happens.  But there are subtleties.  A Gmail user
sends a message to a mailing list and never gets a reply in their
mailbox tagged Inbox.  Because it is a duplicate of their Sent tagged
message.  The mailing list sent it to Google but Google discarded it
as a duplicate of the Sent tagged message.  But then someone else
makes a follow-up to that message to the mailing list.  That new
message follow-up arrives and is placed in the Gmail user's mail
folder and tagged "Inbox".

Now some subtlety.  Since the follow-up was In-Reply-To the previous
message it has the effect of pulling the *original outgoing* message
into that "Gmail conversation" too.  Conversations are grouped by
subject.  Since the new message is a reply to the Sent tagged message.
If there is a follow-up then suddenly the Gmail user sees their
outgoing message in their Inbox view making it appear as if their
messages only appear when someone follows up to it and are lost
otherwise.

This behavior is such that one either loves it or hates it.  Certainly
without understanding what's happening it makes using mailing lists
terribly confusing.  If you are a person who wants to de-duplicate
messages then you love it.  If you are a person like me who wants to
see the Received headers in each message to see the trip it took to
get there then you hate it since it loses information and makes
debugging problems quite difficult.

Pretty sure that is what is happening.  It fits the model.  I have
spent a bunch of time debugging this behavior for Gmail users trying
to use mailing lists as a mailing list admin and have deduced this
model from the observed behavior.  But of course who knows what evil
is actually happening?

Bob

Reference:


http://mail.google.com/support/bin/answer.py?hl=en&answer=6588&topic=1668979 
(now dead)


https://web.archive.org/web/2027050813/http://mail.google.com/support/bin/answer.py?hl=en&answer=6588&topic=1668979

Messages sent to mailing lists don't show in my inbox When you send a
message to any mailing list you subscribe to, Gmail automatically
skips your inbox and archives the message to save you time and prevent
clutter.  The message will appear in your inbox if someone responds to
it or if there is an error delivering 

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Jaroslaw Rafa via mailop]

Dnia 17.05.2023 o godz. 15:22:22 Bob Proulx via mailop pisze:
> This behavior is such that one either loves it or hates it.  Certainly
> without understanding what's happening it makes using mailing lists
> terribly confusing.  If you are a person who wants to de-duplicate
> messages then you love it.  If you are a person like me who wants to
> see the Received headers in each message to see the trip it took to
> get there then you hate it since it loses information and makes
> debugging problems quite difficult.

Problem would be solved if this behavior was a switchable option one can
turn on and off. Both types of people could be satisfied then. And by
looking into settings, it would be clear what happens.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Brandon Long via mailop]

On Wed, May 17, 2023 at 3:18 PM Jaroslaw Rafa via mailop 
wrote:

> Dnia 17.05.2023 o godz. 15:22:22 Bob Proulx via mailop pisze:
> > This behavior is such that one either loves it or hates it.  Certainly
> > without understanding what's happening it makes using mailing lists
> > terribly confusing.  If you are a person who wants to de-duplicate
> > messages then you love it.  If you are a person like me who wants to
> > see the Received headers in each message to see the trip it took to
> > get there then you hate it since it loses information and makes
> > debugging problems quite difficult.
>
> Problem would be solved if this behavior was a switchable option one can
> turn on and off. Both types of people could be satisfied then. And by
> looking into settings, it would be clear what happens.
>

The vast majority of users don't understand getting multiple copies of a
message, regardless
of the path they take.  They are not going to be checking the Received
headers.  They don't
understand that maybe the mailing list didn't forward the message (if they
even realize what a
mailing list is).  We also have noticed that some enterprise customers have
complicated mail routing
and expansion settings which can result in a ridiculous number of copies of
a single message,
one customer's "notify all" list would result in up to 9k duplicate
messages for individual mailboxes.

Every setting that exists is a burden on the maintainability of the system
and on the customers
trying to understand the tool.  A setting that a small fraction of people
would actually want/use will
actually get enabled by more people than that, who will find the results
bewildering and annoying.

I was one of the people who liked that old behavior when I ran my own
server, but our experience
here trying it out on folks was not great (the cl description to disable
the internal test was "stop the madness") .
Maybe we could have improved feature enough for everyone, but the it was
decided that the effort was
better spent elsewhere.  We had an open feature request for a decade for
this that never made the priority
list.  I did want a message log feature which would be similar to a
per-mailbox version of the Workspace
Email Log Search  but that
never quite made the cut either.  In particular, this duplicate message
behavior
did result in a large fraction of "missing email" escalations, along with
folks not understanding filters, third party
apps deleting messages, customers deleting messages and forgetting, and
multiple users using a single mailbox
and not realizing someone else deleted it.

It is also very common for these complaints to neglect to mention that the
sender and recipient mailbox are the
same.

I know some mailing list software does modify messages in order to defeat
the duplicate detection in
Gmail.  The duplicate detection requires that the date, "clean" subject and
messageid are identical.
The "clean" subject is one that tries to remove various mailing list
subject prefixes/suffixes and reply markers.
Modifying the messageid of course would break the DKIM signature, but most
mailing list software already does
that.

Brandon
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Jaroslaw Rafa via mailop]

Dnia 17.05.2023 o godz. 17:25:36 Brandon Long via mailop pisze:
> The vast majority of users don't understand getting multiple copies of a
> message, regardless
> of the path they take.  They are not going to be checking the Received
> headers.  They don't
> understand that maybe the mailing list didn't forward the message (if they
> even realize what a
> mailing list is).

I don't quite understand you. Are you suggesting that people who willingly
subscribe to a mailing list don't know what a mailing list is? It's hard for
me to believe that. Or do you mean something else?

> We also have noticed that some enterprise customers have
> complicated mail routing
> and expansion settings which can result in a ridiculous number of copies of
> a single message,
> one customer's "notify all" list would result in up to 9k duplicate
> messages for individual mailboxes.

I work in a company that is your enterprise customer. We host all our mail
on Gsuite. We have over a hundred internal mailing lists hosted on Google
Groups used within our company, and it repeats over and over that a new
employee who subscribes to a mailing list has doubts whether the list works,
because he/she doesn't get back their messages sent to the list. It needs to
be explained to these people over and over again that "Google just works so
and there's nothing you can do about it".

People who *do* use mailing lists (I'm not talking about those who don't)
routinely use their message that is coming back from the list as a check
that the list is "alive" and working properly. Lack of that message causes
confusion. I have seen it hundreds of times.

As for the single message resulting in 9k duplicates, I remember a few cases
when something like this happened. It was usually when by mistake someone
sent a message intended for someone else to a mailing list, and it resulted
in a flood of "Please remove me from the mailing list" type messages. But
such incidents are basically unavoidable...

> never quite made the cut either.  In particular, this duplicate message
> behavior
> did result in a large fraction of "missing email" escalations, along with
> folks not understanding filters, third party
> apps deleting messages, customers deleting messages and forgetting, and
> multiple users using a single mailbox
> and not realizing someone else deleted it.

Most "missing email" problems I have seen occur when people are using
ordinary mail clients to access their Gsuite email accounts (which is a
common corporate practice, as Gmail/Gsuite web interface is just unusable
for professional use where you are handling a lot of email), not realizing
that Google is different and it doesn't have folders, but "labels", and the
same message can have multiple "labels" (while with ordinary mail server, the
same message cannot be in multiple different folders). Someone wants to
delete a "duplicate" message that he/she has in another "folder" and ends up
deleting the message from all "folders", which is a totally unexpected
result.
-- 
Regards,
   Jaroslaw Rafa
   r...@rafa.eu.org
--
"In a million years, when kids go to school, they're gonna know: once there
was a Hushpuppy, and she lived with her daddy in the Bathtub."
___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Byung-Hee HWANG via mailop]

On Wed, May 17, 2023 at 05:25:36PM -0700, Brandon Long via mailop wrote:
> (...)
> 
> I know some mailing list software does modify messages in order to defeat
> the duplicate detection in
> Gmail.  The duplicate detection requires that the date, "clean" subject and
> messageid are identical.
> The "clean" subject is one that tries to remove various mailing list
> subject prefixes/suffixes and reply markers.
> Modifying the messageid of course would break the DKIM signature, but most
> mailing list software already does
> that.
> 

Dear Brandon,

Thanks for information. It is something i have been curious about for a
long time.

Sincerely, Byung-Hee

___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop

Re: [mailop] Forwarding mail originating from gmail via 3rd party to gmail

[Frido Otten via mailop]

I've looked into ARC which looks promising, but ARC is still 
experimental according to the wiki page so it's not really an option in 
my situation. The case is that addess a...@gmail.com sends a message to 
add...@somewhere.tld which has a forward to address b...@gmail.com. So it's 
not that the message is blocked because it would loop back to its 
sender. It looks like these messages mainly get lost (sometimes blocked) 
because the Google mailserver doesn't like a message originating from 
the gmail.com domain to come from a third party server, even if the DKIM 
signature of the original message is intact.



Op 17-05-2023 om 16:06 schreef Al Iverson via mailop:

Try rewriting the message ID. I think Gmail is believing the message
to be a duplicate and in some cases it will silently eat duplicates.

I have a forwarding feature for my own domains that in addition to
using SRS and re-signing DKIM with my domain, it rewrites the message
ID.

Good point about ARC; I don't have proof that it would fix things but
it would potentially be something to try.

Cheers,
Al

On Wed, May 17, 2023 at 4:23 AM Taavi Eomäe via mailop
 wrote:

Do those forwarded letters without SRS have intact DKIM?

A second thing you can try is ARC (without the SRS), I've gotten the
impression that Gmail "likes" original letters (and ARC) more than it
likes any kind of mangling, including SRS.

___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop




___
mailop mailing list
mailop@mailop.org
https://list.mailop.org/listinfo/mailop