[issue24848] Warts in UTF-7 error handling

Changes by Serhiy Storchaka : -- assignee: -> serhiy.storchaka resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue24848] Warts in UTF-7 error handling

Roundup Robot added the comment: New changeset ff1366ff2761 by Serhiy Storchaka in branch '2.7': Issue #24848: Fixed yet one bug in UTF-7 decoder. Testing for BASE64 character https://hg.python.org/cpython/rev/ff1366ff2761 -- ___ Python tracker

[issue24848] Warts in UTF-7 error handling

Serhiy Storchaka added the comment: The difference between 2.7 and 3.x is that 2.7 uses isalnum() in IS_BASE64, and 3.x test concrete ranges. Therefore depending on platform and locale 2.7 can accept wrong bytes as BASE64 characters and return incorrect result. Following patch makes 2.7 code

[issue24848] Warts in UTF-7 error handling

STINNER Victor added the comment: The patch looks good to me. -- ___ Python tracker ___ ___ Python-bugs-list

[issue24848] Warts in UTF-7 error handling

Serhiy Storchaka added the comment: Test failure is random. With build 3435 tests are successful, with all other are failed. The same with other buildbot: http://buildbot.python.org/all/builders/x86%20Windows7%202.7/ . 3345 and 3347 are green, others are red. --

[issue24848] Warts in UTF-7 error handling

STINNER Victor added the comment: http://buildbot.python.org/all/builders/x86%20XP-4%202.7/builds/3431/steps/test/logs/stdio == FAIL: test_errors (test.test_codecs.UTF7Test)

[issue24848] Warts in UTF-7 error handling

Roundup Robot added the comment: New changeset 3c13567ea642 by Serhiy Storchaka in branch '3.4': Issue #24848: Fixed bugs in UTF-7 decoding of misformed data: https://hg.python.org/cpython/rev/3c13567ea642 New changeset a61fa2b08f87 by Serhiy Storchaka in branch '3.5': Issue #24848: Fixed bugs

[issue24848] Warts in UTF-7 error handling

Serhiy Storchaka added the comment: Have no ideas why tests are failed and only on this buildbot. -- ___ Python tracker ___

[issue24848] Warts in UTF-7 error handling

STINNER Victor added the comment: Oops, ignore my comment, I forgot to recompile Python. "make" and the bug is done :-) -- ___ Python tracker ___

[issue24848] Warts in UTF-7 error handling

STINNER Victor added the comment: > Have no ideas why tests are failed and only on this buildbot. test_codecs always crash on Python 3.6 with Python compiled in debug mode: test_errors (test.test_codecs.UTF7Test) ... python: Objects/unicodeobject.c:1263: _copy_characters: Assertion `ch <=

[issue24848] Warts in UTF-7 error handling

Serhiy Storchaka added the comment: Updated patch fixes also a bug in _PyUnicodeWriter. Other affected encoding is "unicode-escape": >>> br'\u;'.decode('unicode-escape', 'replace') 'ý;' -- Added file: http://bugs.python.org/file40604/utf7_error_handling-2.patch

[issue24848] Warts in UTF-7 error handling

Changes by Serhiy Storchaka : -- nosy: +pitrou ___ Python tracker ___ ___

[issue24848] Warts in UTF-7 error handling

Serhiy Storchaka added the comment: There is a reason for behavior in case 2. This is likely a truncated data and it is safer to raise an exception than silently produce lone surrogate. Current UTF-7 encoder always adds '-' after ending shift sequence. I suppose this is not a bug. However

[issue24848] Warts in UTF-7 error handling

New submission from Serhiy Storchaka: Trying to implement UTF-7 codec in Python I found some warts in error handling. 1. Non-ASCII bytes. No errors: 'a€b'.encode('utf-7') b'a+IKw-b' b'a+IKw-b'.decode('utf-7') 'a€b' Terminating '-' at the end of the string is optional.