[issue32330] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Change by Mark Sapiro :


--
keywords: +patch
pull_requests: +17453
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/18059

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32330] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Change by Mark Sapiro :


--
versions: +Python 3.7, Python 3.8, Python 3.9

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32330] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Mark Sapiro  added the comment:

> I do wonder where you are using the string version of messages :)

Probably some places where we could use bytes, but one of the problem areas is 
where we save the content of a message held for moderation.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32330] Email parser creates a message object that can't be flattened

[R. David Murray]

R. David Murray  added the comment:

I do wonder where you are using the string version of messages :)

I actually thought I'd already done this (errors=replace), but obviously not.  
I don't have time now to work on a patch for this, and the patch in the other 
issue hasn't be updated to reflect the review I did :(

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32330] Email parser creates a message object that can't be flattened

[Mark Sapiro]

Mark Sapiro  added the comment:

Yes. I think errors=replace is a good solution. In Mailman, we have our own 
mailman.email.message.Message class which is a subclass of 
email.message.Message and what we do to work around this and issue27321 is 
override as_string() with:

def as_string(self):
# Work around for https://bugs.python.org/issue27321 and
# https://bugs.python.org/issue32330.
try:
value = email.message.Message.as_string(self)
except (KeyError, UnicodeEncodeError):
value = email.message.Message.as_bytes(self).decode(
'ascii', 'replace')
return value

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32330] Email parser creates a message object that can't be flattened

[R. David Murray]

R. David Murray  added the comment:

What would you like to see happen in that situation?  Should we use 
errors=replace like we do for headers?  (That seems reasonable to me.)

Note that it can be re-serialized as binary.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue32330] Email parser creates a message object that can't be flattened

[Mark Sapiro]

New submission from Mark Sapiro :

This is related to https://bugs.python.org/issue27321 but a different exception 
is thrown for a different reason. This is caused by a defective spam message. I 
don't actually have the offending message from the wild, but the attached 
bad_email_2.eml illustrates the problem.

The defect is the message declares the content charset as us-ascii, but the 
body contains non-ascii. When the message is parsed into an 
email.message.Message object and the objects as_string() method is called, 
UnicodeEncodeError is thrown as follows:

>>> import email
>>> with open('bad_email_2.eml', 'rb') as fp:
... msg = email.message_from_binary_file(fp)
... 
>>> msg.as_string()
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python3.5/email/message.py", line 159, in as_string
g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
self._write(msg)
  File "/usr/lib/python3.5/email/generator.py", line 181, in _write
self._dispatch(msg)
  File "/usr/lib/python3.5/email/generator.py", line 214, in _dispatch
meth(msg)
  File "/usr/lib/python3.5/email/generator.py", line 243, in _handle_text
msg.set_payload(payload, charset)
  File "/usr/lib/python3.5/email/message.py", line 316, in set_payload
payload = payload.encode(charset.output_charset)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 31-33: 
ordinal not in range(128)

--
components: email
files: bad_email_2.eml
messages: 308353
nosy: barry, msapiro, r.david.murray
priority: normal
severity: normal
status: open
title: Email parser creates a message object that can't be flattened
type: behavior
versions: Python 3.5, Python 3.6
Added file: https://bugs.python.org/file47333/bad_email_2.eml

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com