[issue27321] Email parser creates a message object that can't be flattened

[miss-islington]

miss-islington  added the comment:


New changeset 72ce82abcf9051b18a05350936de7ecab7306662 by Miss Skeleton (bot) 
in branch '3.9':
bpo-27321 Fix email.generator.py to not replace a non-existent header. 
(GH-18074)
https://github.com/python/cpython/commit/72ce82abcf9051b18a05350936de7ecab7306662


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[miss-islington]

miss-islington  added the comment:


New changeset 371146a3f8a989964e2a9c0efc7d776815410fac by Miss Skeleton (bot) 
in branch '3.8':
bpo-27321 Fix email.generator.py to not replace a non-existent header. 
(GH-18074)
https://github.com/python/cpython/commit/371146a3f8a989964e2a9c0efc7d776815410fac


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Barry A. Warsaw]

Change by Barry A. Warsaw :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions: +Python 3.10, Python 3.9 -Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[miss-islington]

miss-islington  added the comment:


New changeset bf838227c35212709dc43b3c3c57f8e1655c1d24 by Mark Sapiro in branch 
'master':
bpo-27321 Fix email.generator.py to not replace a non-existent header. 
(GH-18074)
https://github.com/python/cpython/commit/bf838227c35212709dc43b3c3c57f8e1655c1d24


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[miss-islington]

Change by miss-islington :


--
pull_requests: +21753
pull_request: https://github.com/python/cpython/pull/22797

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[miss-islington]

Change by miss-islington :


--
pull_requests: +21752
pull_request: https://github.com/python/cpython/pull/22796

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Mark Sapiro  added the comment:

I work around it with
```
class Message(email.message.Message):

def as_string(self):
# Work around for https://bugs.python.org/issue27321 and
# https://bugs.python.org/issue32330.
try:
value = email.message.Message.as_string(self)
except (KeyError, LookupError, UnicodeEncodeError):
value = email.message.Message.as_bytes(self).decode(
'ascii', 'replace')
# Also ensure no unicode surrogates in the returned string.
return email.utils._sanitize(value)
```
This is easy for me because it's Mailman which already subclasses 
email.message.Message for other reasons. It is perhaps more difficult if you 
aren't already subclassing email.message.Message for other purposes.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Diekhans]

Mark Diekhans  added the comment:

any chance of getting this merged?  A work-around is not obvious

--
nosy: +diekhans

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Change by Mark Sapiro :


--
pull_requests: +17467
pull_request: https://github.com/python/cpython/pull/18074

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Cheryl Sabella]

Cheryl Sabella  added the comment:

@r.david.murray, it appears that all your requested changes have been addressed 
on the PR.  Please re-review this when you get a chance.  Thanks!

--
nosy: +cheryl.sabella
versions:  -Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Stéphane Wirtel]

Change by Stéphane Wirtel :


--
versions: +Python 3.8 -Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[David Cannings]

David Cannings  added the comment:

Ping on an ETA for this fix?

--
nosy: +edeca

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[R. David Murray]

R. David Murray  added the comment:

Requested a small additional change to the new tests, and then this will be 
ready to go in.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[R. David Murray]

R. David Murray  added the comment:

Note: I reviewed this a while ago but the review comments haven't been 
addressed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Johannes Löthberg]

Johannes Löthberg added the comment:

Ping?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

ok, I have tested on 3.6 and 3.5, just with the test. and in this case, we get 
the errors on both.
if we apply the patch of Johannes, the test passes and there is no issues.

+1 the backports for 3.5 and 3.6 is just a git cherry-picking.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[R. David Murray]

R. David Murray added the comment:

I'm going to try to review this this weekend.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Stéphane Wirtel]

Changes by Stéphane Wirtel :


--
stage: needs patch -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Stéphane Wirtel]

Stéphane Wirtel added the comment:

maybe we could merge the PR, and I could propose a backport for 3.5 and 3.6.

2.7 is affected ?

--
nosy: +matrixise

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Johannes Löthberg]

Changes by Johannes Löthberg :


--
pull_requests: +2043

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Johannes Löthberg]

Johannes Löthberg added the comment:

Ah, didn't even see your comment before I did it!  Fix to the comments are on 
the same branch, will be rebased before PR is up.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Mark Sapiro added the comment:

It looks like Johannes beat me to it. Thanks for that, but see my comments in 
the diff at 
https://github.com/kyrias/cpython/commit/a986a8274a522c73d87360da6930e632a3eb4ebb

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Johannes Löthberg]

Johannes Löthberg added the comment:

Fix: 
https://github.com/kyrias/cpython/commit/a986a8274a522c73d87360da6930e632a3eb4ebb
Testcase: 
https://github.com/kyrias/cpython/commit/9a510426522e1d714cd0ea238b14de0fc76862b2

Can start a PR once my CLA signature goes through I guess.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Mark Sapiro added the comment:

I considered look before you leap, but I decided since we're munging the 
headers anyway, preserving their order is not that critical, but the patch is 
easy enough. I'll work on that and a test.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[R. David Murray]

Changes by R. David Murray :


--
stage:  -> needs patch
type:  -> behavior
versions: +Python 3.6, Python 3.7 -Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[R. David Murray]

R. David Murray added the comment:

replace_header has a different semantic than del-and-set (replace_header leaves 
the header in the same location in the list, rather than appending it to the 
end...that's it's purpose).  If replace_header is throwing a key error, then I 
guess we need a look-before-you-leap if statement.

And a test :)

Note that a correct fix would preserve the broken input email, but since we're 
obviously already doing munging I'm fine with just making this work for now.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Johannes Löthberg]

Johannes Löthberg added the comment:

Any updates on this?  I'm having the same problem with some non-spam emails 
while trying to use some mail-handling tools written in Python.

--
nosy: +Johannes Löthberg

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Barry A. Warsaw]

Barry A. Warsaw added the comment:

While I was reviewing 
https://gitlab.com/mailman/mailman/merge_requests/197/diffs I noticed the 
KeyError and it made me thing "hmm, I wonder if this should be turned into one 
of the email package errors"?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Mark Sapiro added the comment:

One additional observation. The original message contained no 
Content-Transfer-Encoding header even though the message body was raw koi8-r 
characters. Adding

Content-Transfer-Encoding: 8bit

to the message headers avoids the issue, but that is not a practical solution 
as the message was Russian spam received by a Mailman list and the resultant 
KeyError caused problems in Mailman.

We can work on defending against this in Mailman, but I suggest that the 
munge_cte code in generator._write() avoid the documented possible KeyError 
raised by replace_header() by using __delitem__() and __setitem__() instead as 
in the attached generator.patch.

--
keywords: +patch
Added file: http://bugs.python.org/file43394/generator.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Maciej Szulik]

Changes by Maciej Szulik :


--
nosy: +maciej.szulik

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue27321] Email parser creates a message object that can't be flattened

[Mark Sapiro]

New submission from Mark Sapiro:

The attached file, bad_email, can be parsed via

msg = email.message_from_binary_file(open('bad_email', 'rb'))

but then msg.as_string() prodices the following:

Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.5/email/message.py", line 159, in as_string
g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
self._write(msg)
  File "/usr/lib/python3.5/email/generator.py", line 189, in _write
msg.replace_header('content-transfer-encoding', munge_cte[0])
  File "/usr/lib/python3.5/email/message.py", line 559, in replace_header
raise KeyError(_name)
KeyError: 'content-transfer-encoding'

--
components: email
files: bad_email
messages: 268580
nosy: barry, msapiro, r.david.murray
priority: normal
severity: normal
status: open
title: Email parser creates a message object that can't be flattened
versions: Python 3.4, Python 3.5
Added file: http://bugs.python.org/file43391/bad_email

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com