Re: [PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
Applied with a release note.
Daniel Axtens writes:
> Andrew Donnellan writes:
>
>> To avoid triggering spam filters due to failed signature validation, many
>> mailing lists mangle the From header to change the From address to be the
>> address of the list, typically where the sender's domain has a strict DMARC
>> policy enabled.
>>
>> In this case, we should try to unmangle the From header.
>>
>> Add support for using the X-Original-From or Reply-To headers, as used by
>> Google Groups and Mailman respectively, to unmangle the From header when
>> necessary and associate the patch with the correct submitter based on the
>> unmangled email address.
>>
>> When downloading mboxes, rewrite the From header using the unmangled
>> address, and preserve the original header as X-Patchwork-Original-From in
>> case someone needs it for some reason. The original From header will still
>> be stored in the database and exposed via the API, as we want to keep
>> messages as close to the original received format as possible.
>>
>> Closes: #64 ("Incorrect submitter when using googlegroups")
>> Reported-by: Alexandre Belloni
>> Reported-by: Stephen Rothwell
>> Signed-off-by: Andrew Donnellan
>
> Tested-by: Daniel Axtens # mailman only
>
> I'll apply it shortly to master. You could convince me to get it
> backported to stable if you really wanted.
>
> Do we need something to go through the db and fix up old ones? Or is it
> best to let sleeping patches lie?
>
> Regards,
> Daniel
>>
>> ---
>>
>> v1->v2:
>> - use X-Original-From rather than X-Original-Sender
>> - unmangle From header when downloading mbox
>>
>> rewrite from header
>>
>> Signed-off-by: Andrew Donnellan
>>
>> use x original from
>>
>> Signed-off-by: Andrew Donnellan
>> ---
>> patchwork/parser.py | 75 +++
>> patchwork/tests/test_mboxviews.py | 21 +
>> patchwork/tests/test_parser.py| 68 ++--
>> patchwork/views/utils.py | 12 +
>> 4 files changed, 163 insertions(+), 13 deletions(-)
>>
>> diff --git a/patchwork/parser.py b/patchwork/parser.py
>> index 7dc66bc05a5b..be1e51652dd3 100644
>> --- a/patchwork/parser.py
>> +++ b/patchwork/parser.py
>> @@ -321,12 +321,7 @@ def find_series(project, mail, author):
>> return _find_series_by_markers(project, mail, author)
>>
>>
>> -def get_or_create_author(mail):
>> -from_header = clean_header(mail.get('From'))
>> -
>> -if not from_header:
>> -raise ValueError("Invalid 'From' header")
>> -
>> +def split_from_header(from_header):
>> name, email = (None, None)
>>
>> # tuple of (regex, fn)
>> @@ -355,6 +350,65 @@ def get_or_create_author(mail):
>> (name, email) = fn(match.groups())
>> break
>>
>> +return (name, email)
>> +
>> +
>> +# Unmangle From addresses that have been mangled for DMARC purposes.
>> +#
>> +# To avoid triggering spam filters due to failed signature validation, many
>> +# mailing lists mangle the From header to change the From address to be the
>> +# address of the list, typically where the sender's domain has a strict
>> +# DMARC policy enabled.
>> +#
>> +# Unfortunately, there's no standardised way of preserving the original
>> +# From address.
>> +#
>> +# Google Groups adds an X-Original-From header. If present, we use that.
>> +#
>> +# Mailman preserves the original address by adding a Reply-To, except in the
>> +# case where the list is set to either reply to list, or reply to a specific
>> +# address, in which case the original From is added to Cc instead. These
>> corner
>> +# cases are dumb, but we try and handle things as sensibly as possible by
>> +# looking for a name in Reply-To/Cc that matches From. It's inexact but
>> should
>> +# be good enough for our purposes.
>> +def get_original_sender(mail, name, email):
>> +if name and ' via ' in name:
>> +# Mailman uses the format " via "
>> +# Google Groups uses "'' via "
>> +stripped_name = name[:name.rfind(' via ')].strip().strip("'")
>> +
>> +original_from = clean_header(mail.get('X-Original-From', ''))
>> +if original_from:
>> +new_email = split_from_header(original_from)[1].strip()[:255]
>> +return (stripped_name, new_email)
>> +
>> +addrs = []
>> +reply_to_headers = mail.get_all('Reply-To') or []
>> +cc_headers = mail.get_all('Cc') or []
>> +for header in reply_to_headers + cc_headers:
>> +header = clean_header(header)
>> +addrs = header.split(",")
>> +for addr in addrs:
>> +new_name, new_email = split_from_header(addr)
>> +if new_name:
>> +new_name = new_name.strip()[:255]
>> +if new_email:
>> +new_email = new_email.strip()[:255]
>> +if new_name == stripped_name:
>> +return (stripped_name, new_email)
>> +
>> +# If we can't figure out the original sender, just keep it as is
>> +return
Re: [PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
On Fri, 2019-10-18 at 15:16 +1100, Andrew Donnellan wrote: > On 18/10/19 3:06 pm, Daniel Axtens wrote: > > I'll apply it shortly to master. You could convince me to get it > > backported to stable if you really wanted. > > > > Do we need something to go through the db and fix up old ones? Or is it > > best to let sleeping patches lie? > > Personally I was thinking we should leave previous patches as-is, it > seems like a fair bit of extra code for fairly small gain. Agreed. It would be a large data migration that I'd rather not have to do. If people really care, they can probably produce a script to run via 'manage.py (db)shell' to fix these up manually or we can fix ourselves later. Stephen ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
On 18/10/19 3:06 pm, Daniel Axtens wrote: I'll apply it shortly to master. You could convince me to get it backported to stable if you really wanted. Do we need something to go through the db and fix up old ones? Or is it best to let sleeping patches lie? Personally I was thinking we should leave previous patches as-is, it seems like a fair bit of extra code for fairly small gain. -- Andrew Donnellan OzLabs, ADL Canberra a...@linux.ibm.com IBM Australia Limited ___ Patchwork mailing list Patchwork@lists.ozlabs.org https://lists.ozlabs.org/listinfo/patchwork
Re: [PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
Andrew Donnellan writes:
> To avoid triggering spam filters due to failed signature validation, many
> mailing lists mangle the From header to change the From address to be the
> address of the list, typically where the sender's domain has a strict DMARC
> policy enabled.
>
> In this case, we should try to unmangle the From header.
>
> Add support for using the X-Original-From or Reply-To headers, as used by
> Google Groups and Mailman respectively, to unmangle the From header when
> necessary and associate the patch with the correct submitter based on the
> unmangled email address.
>
> When downloading mboxes, rewrite the From header using the unmangled
> address, and preserve the original header as X-Patchwork-Original-From in
> case someone needs it for some reason. The original From header will still
> be stored in the database and exposed via the API, as we want to keep
> messages as close to the original received format as possible.
>
> Closes: #64 ("Incorrect submitter when using googlegroups")
> Reported-by: Alexandre Belloni
> Reported-by: Stephen Rothwell
> Signed-off-by: Andrew Donnellan
Tested-by: Daniel Axtens # mailman only
I'll apply it shortly to master. You could convince me to get it
backported to stable if you really wanted.
Do we need something to go through the db and fix up old ones? Or is it
best to let sleeping patches lie?
Regards,
Daniel
>
> ---
>
> v1->v2:
> - use X-Original-From rather than X-Original-Sender
> - unmangle From header when downloading mbox
>
> rewrite from header
>
> Signed-off-by: Andrew Donnellan
>
> use x original from
>
> Signed-off-by: Andrew Donnellan
> ---
> patchwork/parser.py | 75 +++
> patchwork/tests/test_mboxviews.py | 21 +
> patchwork/tests/test_parser.py| 68 ++--
> patchwork/views/utils.py | 12 +
> 4 files changed, 163 insertions(+), 13 deletions(-)
>
> diff --git a/patchwork/parser.py b/patchwork/parser.py
> index 7dc66bc05a5b..be1e51652dd3 100644
> --- a/patchwork/parser.py
> +++ b/patchwork/parser.py
> @@ -321,12 +321,7 @@ def find_series(project, mail, author):
> return _find_series_by_markers(project, mail, author)
>
>
> -def get_or_create_author(mail):
> -from_header = clean_header(mail.get('From'))
> -
> -if not from_header:
> -raise ValueError("Invalid 'From' header")
> -
> +def split_from_header(from_header):
> name, email = (None, None)
>
> # tuple of (regex, fn)
> @@ -355,6 +350,65 @@ def get_or_create_author(mail):
> (name, email) = fn(match.groups())
> break
>
> +return (name, email)
> +
> +
> +# Unmangle From addresses that have been mangled for DMARC purposes.
> +#
> +# To avoid triggering spam filters due to failed signature validation, many
> +# mailing lists mangle the From header to change the From address to be the
> +# address of the list, typically where the sender's domain has a strict
> +# DMARC policy enabled.
> +#
> +# Unfortunately, there's no standardised way of preserving the original
> +# From address.
> +#
> +# Google Groups adds an X-Original-From header. If present, we use that.
> +#
> +# Mailman preserves the original address by adding a Reply-To, except in the
> +# case where the list is set to either reply to list, or reply to a specific
> +# address, in which case the original From is added to Cc instead. These
> corner
> +# cases are dumb, but we try and handle things as sensibly as possible by
> +# looking for a name in Reply-To/Cc that matches From. It's inexact but
> should
> +# be good enough for our purposes.
> +def get_original_sender(mail, name, email):
> +if name and ' via ' in name:
> +# Mailman uses the format " via "
> +# Google Groups uses "'' via "
> +stripped_name = name[:name.rfind(' via ')].strip().strip("'")
> +
> +original_from = clean_header(mail.get('X-Original-From', ''))
> +if original_from:
> +new_email = split_from_header(original_from)[1].strip()[:255]
> +return (stripped_name, new_email)
> +
> +addrs = []
> +reply_to_headers = mail.get_all('Reply-To') or []
> +cc_headers = mail.get_all('Cc') or []
> +for header in reply_to_headers + cc_headers:
> +header = clean_header(header)
> +addrs = header.split(",")
> +for addr in addrs:
> +new_name, new_email = split_from_header(addr)
> +if new_name:
> +new_name = new_name.strip()[:255]
> +if new_email:
> +new_email = new_email.strip()[:255]
> +if new_name == stripped_name:
> +return (stripped_name, new_email)
> +
> +# If we can't figure out the original sender, just keep it as is
> +return (name, email)
> +
> +
> +def get_or_create_author(mail, project=None):
> +from_header = clean_header(mail.get('From'))
> +
> +if not from_header:
> +raise ValueError("Invalid
[PATCH v2] parser: Unmangle From: headers that have been mangled for DMARC purposes
To avoid triggering spam filters due to failed signature validation, many
mailing lists mangle the From header to change the From address to be the
address of the list, typically where the sender's domain has a strict DMARC
policy enabled.
In this case, we should try to unmangle the From header.
Add support for using the X-Original-From or Reply-To headers, as used by
Google Groups and Mailman respectively, to unmangle the From header when
necessary and associate the patch with the correct submitter based on the
unmangled email address.
When downloading mboxes, rewrite the From header using the unmangled
address, and preserve the original header as X-Patchwork-Original-From in
case someone needs it for some reason. The original From header will still
be stored in the database and exposed via the API, as we want to keep
messages as close to the original received format as possible.
Closes: #64 ("Incorrect submitter when using googlegroups")
Reported-by: Alexandre Belloni
Reported-by: Stephen Rothwell
Signed-off-by: Andrew Donnellan
---
v1->v2:
- use X-Original-From rather than X-Original-Sender
- unmangle From header when downloading mbox
rewrite from header
Signed-off-by: Andrew Donnellan
use x original from
Signed-off-by: Andrew Donnellan
---
patchwork/parser.py | 75 +++
patchwork/tests/test_mboxviews.py | 21 +
patchwork/tests/test_parser.py| 68 ++--
patchwork/views/utils.py | 12 +
4 files changed, 163 insertions(+), 13 deletions(-)
diff --git a/patchwork/parser.py b/patchwork/parser.py
index 7dc66bc05a5b..be1e51652dd3 100644
--- a/patchwork/parser.py
+++ b/patchwork/parser.py
@@ -321,12 +321,7 @@ def find_series(project, mail, author):
return _find_series_by_markers(project, mail, author)
-def get_or_create_author(mail):
-from_header = clean_header(mail.get('From'))
-
-if not from_header:
-raise ValueError("Invalid 'From' header")
-
+def split_from_header(from_header):
name, email = (None, None)
# tuple of (regex, fn)
@@ -355,6 +350,65 @@ def get_or_create_author(mail):
(name, email) = fn(match.groups())
break
+return (name, email)
+
+
+# Unmangle From addresses that have been mangled for DMARC purposes.
+#
+# To avoid triggering spam filters due to failed signature validation, many
+# mailing lists mangle the From header to change the From address to be the
+# address of the list, typically where the sender's domain has a strict
+# DMARC policy enabled.
+#
+# Unfortunately, there's no standardised way of preserving the original
+# From address.
+#
+# Google Groups adds an X-Original-From header. If present, we use that.
+#
+# Mailman preserves the original address by adding a Reply-To, except in the
+# case where the list is set to either reply to list, or reply to a specific
+# address, in which case the original From is added to Cc instead. These corner
+# cases are dumb, but we try and handle things as sensibly as possible by
+# looking for a name in Reply-To/Cc that matches From. It's inexact but should
+# be good enough for our purposes.
+def get_original_sender(mail, name, email):
+if name and ' via ' in name:
+# Mailman uses the format " via "
+# Google Groups uses "'' via "
+stripped_name = name[:name.rfind(' via ')].strip().strip("'")
+
+original_from = clean_header(mail.get('X-Original-From', ''))
+if original_from:
+new_email = split_from_header(original_from)[1].strip()[:255]
+return (stripped_name, new_email)
+
+addrs = []
+reply_to_headers = mail.get_all('Reply-To') or []
+cc_headers = mail.get_all('Cc') or []
+for header in reply_to_headers + cc_headers:
+header = clean_header(header)
+addrs = header.split(",")
+for addr in addrs:
+new_name, new_email = split_from_header(addr)
+if new_name:
+new_name = new_name.strip()[:255]
+if new_email:
+new_email = new_email.strip()[:255]
+if new_name == stripped_name:
+return (stripped_name, new_email)
+
+# If we can't figure out the original sender, just keep it as is
+return (name, email)
+
+
+def get_or_create_author(mail, project=None):
+from_header = clean_header(mail.get('From'))
+
+if not from_header:
+raise ValueError("Invalid 'From' header")
+
+name, email = split_from_header(from_header)
+
if not email:
raise ValueError("Invalid 'From' header")
@@ -362,6 +416,9 @@ def get_or_create_author(mail):
if name is not None:
name = name.strip()[:255]
+if project and email.lower() == project.listemail.lower():
+name, email = get_original_sender(mail, name, email)
+
# this correctly handles the case where we lose the race to create
# the person and another process beats us to it. (If the record
# does
