[issue36520] Email header folded incorrectly

[Irit Katriel]

Change by Irit Katriel :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Irit Katriel]

Irit Katriel  added the comment:

This seems complete, can it be closed?

--
nosy: +iritkatriel

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[miss-islington]

miss-islington  added the comment:


New changeset 36eea7af48ca0a1c96b78c82bf95bbd29d2332da by Miss Islington (bot) 
(Abhilash Raj) in branch '3.8':
[3.8] bpo-36520: Email header folded incorrectly (GH-13608) (GH-13909)
https://github.com/python/cpython/commit/36eea7af48ca0a1c96b78c82bf95bbd29d2332da


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[miss-islington]

miss-islington  added the comment:


New changeset 0745cc66db3acbe7951073071cf063db6337dd10 by Miss Islington (bot) 
(Abhilash Raj) in branch '3.7':
[3.7] bpo-36520: Email header folded incorrectly (GH-13608) (GH-13910)
https://github.com/python/cpython/commit/0745cc66db3acbe7951073071cf063db6337dd10


--
nosy: +miss-islington

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Abhilash Raj]

Change by Abhilash Raj :


--
pull_requests: +13782
pull_request: https://github.com/python/cpython/pull/13909

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Abhilash Raj]

Change by Abhilash Raj :


--
pull_requests: +13783
pull_request: https://github.com/python/cpython/pull/13910

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Barry A. Warsaw]

Barry A. Warsaw  added the comment:


New changeset f6713e84afc5addcfa8477dbdf2c027787f711c0 by Barry Warsaw 
(websurfer5) in branch 'master':
bpo-36520: Email header folded incorrectly (#13608)
https://github.com/python/cpython/commit/f6713e84afc5addcfa8477dbdf2c027787f711c0


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jeffrey Kintscher]

Jeffrey Kintscher  added the comment:

The pull request has been submitted with both the code fix and tests.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
pull_requests: +13515
pull_request: https://github.com/python/cpython/pull/13610

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
keywords: +patch
pull_requests: +13514
stage:  -> patch review
pull_request: https://github.com/python/cpython/pull/13608

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jeffrey Kintscher]

Jeffrey Kintscher  added the comment:

I uploaded a test script with some test cases:

The failure mode occurs when

1. line folding occurs
2. the first folded line has two or more words with UTF-8 characters
3. subsequent lines contain a word with UTF-8 characters located at a different 
offset than the last encoded substring in the first line

For example, the first folded and encoded line of 'Hello Wörld! Hello Wörld! 
Hello Wörld! Hello Wörld!Hello Wörld!' is

b'Subject: Hello =?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!_Hello_W=C3=B6rld!?='

and the second line should be

b' Hello =?utf-8?q?W=C3=B6rld!Hello_W=C3=B6rld!?='

but instead, it is

b' Hello =?utf-8?=?utf-8?q?q=3FW=3DC3=3DB6rld!Hello=3F=3D_W=C3=B6rld!?='

The function at fault is _refold_parse_tree() in 
Lib/email/_header_value_parser.py. In the first line, it encodes the first 
UTF-8 word and saves the starting offset in the output string (15). When it 
encounters the second UTF-8 word, it re-encodes the entire string starting at 
the saved offset. This is to help reduce the bloat added by multiple 
'=?utf-8?q?' start-of-encoding tokens. When it encodes the first UTF-8 word on 
the second line, it tries to store it at the saved offset into the second line 
output string, but that is past the end of the string so it just gets appended. 
When it encounter the second UTF-8 word in the second line, it re-encodes the 
entire second-line string starting at the saved offset (15), which is in the 
middle of the first encoded UTF-8 string.

The failure mode is not triggered if there is at most one UTF-8 word in each 
folded line. It also is not triggered when folding occurs in the middle of a 
word instead of at whitespace because the code follows a different path.

The solution is to set the saved starting offset to None when starting a new 
folded line when the fold-point is whitespace.

I will submit a pull request soon with a fix.

--
Added file: https://bugs.python.org/file48366/bpo-36520-test.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jeffrey Kintscher]

Jeffrey Kintscher  added the comment:

To aid with debugging the code, the Subject line can be simplified:

>>> from email.message import EmailMessage
>>> m = EmailMessage()
>>> m['Subject'] = 'Hello 
>>> =?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!_Hello_W=C3=B6rld!?= Hello 
>>> Wörld!Hello Wörld!'
>>> print(bytes(m))
b'Subject: Hello =?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!_Hello_W=C3=B6rld!?=\n 
Hello =?utf-8?=?utf-8?q?q=3FW=3DC3=3DB6rld!Hello=3F=3D_W=C3=B6rld!?=\n\n'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[R. David Murray]

R. David Murray  added the comment:

Nevermind, I was testing with the wrong version of python.  This bug was 
introduced somewhere after 3.4 :(

>>> from email.message import EmailMessage
>>> m = EmailMessage()
>>> m['Subject'] = 'Hello Wörld! Hello Wörld! Hello Wörld! Hello Wörld!Hello 
>>> Wörld!'
>>> bytes(m)
b'Subject: Hello =?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!_Hello_W=C3=B6rld!?=\n 
Hello =?utf-8?=?utf-8?q?q=3FW=3DC3=3DB6rld!Hello=3F=3D_W=C3=B6rld!?=\n\n'

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[R. David Murray]

R. David Murray  added the comment:

Can you demonstrate the problem with an actual email object?  
header_store_parse is not meant to be called directly.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jeffrey Kintscher]

Change by Jeffrey Kintscher :


--
nosy: +websurfer5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue36520] Email header folded incorrectly

[Jonathan Horn]

New submission from Jonathan Horn :

I encountered a problem with replacing the 'Subject' header of an email. After 
serializing it again, the utf8 encoding was wrong. It seems to be occurring 
when folding the internal header objects.

Example:
>> email.policy.default.fold_binary('Subject', 
>> email.policy.default.header_store_parse('Subject', 'Hello Wörld! Hello 
>> Wörld! Hello Wörld! Hello Wörld!Hello Wörld!')[1])
Expected output: b'Subject: Hello 
=?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!_Hello_W=C3=B6rld!?=\n Hello 
=?utf-8?q?W=C3=B6rld!Hello_W=C3=B6rld!?=\n' (or similar)
Actual output: b'Subject: Hello 
=?utf-8?q?W=C3=B6rld!_Hello_W=C3=B6rld!_Hello_W=C3=B6rld!?=\n Hello 
=?utf-8?=?utf-8?q?q=3FW=3DC3=3DB6rld!Hello=3F=3D_W=C3=B6rld!?=\n'

I'm running Python 3.7.3 on Arch Linux using Linux 5.0.

--
components: email
messages: 339419
nosy: Jonathan Horn, barry, r.david.murray
priority: normal
severity: normal
status: open
title: Email header folded incorrectly
type: behavior
versions: Python 3.7

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com