[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Ned Deily added the comment: New changeset 86fdad093b863db7ef6a3a00c9cff724c09442e7 by Ned Deily (Xiang Zhang) in branch '3.7': bpo-32583: Fix possible crashing in builtin Unicode decoders (#5325)

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Xiang Zhang : -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker ___

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Xiang Zhang added the comment: New changeset ea94fce6960d90fffeeda131e31024617912d231 by Xiang Zhang in branch '3.6': [3.6] bpo-32583: Fix possible crashing in builtin Unicode decoders (GH-5325) (#5459)

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Xiang Zhang : -- pull_requests: +5285 ___ Python tracker ___ ___

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Xiang Zhang added the comment: New changeset 2c7fd46e11333ef5e5cce34212f7d087694f3658 by Xiang Zhang in branch 'master': bpo-32583: Fix possible crashing in builtin Unicode decoders (#5325) https://github.com/python/cpython/commit/2c7fd46e11333ef5e5cce34212f7d087694f3658

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Xiang Zhang : -- pull_requests: +5170 ___ Python tracker ___ ___

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Xiang Zhang added the comment: I write a draft patch, without tests yet. I'll add them later. Reviews are appreciated. I also check the Windows codepage equivalent and encoders, look to me they don't suffer the problem. -- keywords: +patch stage: needs patch ->

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Xiang Zhang added the comment: Another way to crash: >>> import codecs >>> def replace_with_longer(exc): ... exc.object = b'\xa0\x00' * 100 ... return ('\ufffd', exc.end) ... >>> codecs.register codecs.register( codecs.register_error( >>>

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Xiang Zhang : -- stage: patch review -> needs patch ___ Python tracker ___ ___

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Xiang Zhang added the comment: The problem is utf16 decoder almost always assumes that two bytes decodes to one unicode character, so when allocating memory, it assumes (bytes_number+1)/2 unicode slots is enough, there is even a comment in the code. And in

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Serhiy Storchaka : -- components: +Interpreter Core nosy: +serhiy.storchaka stage: test needed -> patch review versions: +Python 3.6 -Python 3.5 ___ Python tracker

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Terry J. Reedy added the comment: As written, decode_crash.py crashes on Windows also. Passing 'replace' instead of 'w3lib_replace' results in no crash and lots of boxes and blanks. -- nosy: +benjamin.peterson, ezio.melotti, lemburg, terry.reedy, vstinner stage: ->

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

New submission from Alexander Sibiryakov : The CPython interpreter gets SIGSEGV or SIGABRT during the run. The script attempts to decode binary file using UTF-16-LE encoding and custom error handler. The error handler is poorly built, and doesn't respect the unicode

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Alexander Sibiryakov : Added file: https://bugs.python.org/file47392/test_string.bin ___ Python tracker ___

[issue32583] Crash during decoding using UTF-16/32 and custom error handler

Change by Alexander Sibiryakov : -- files: decode_crash.py nosy: sibiryakov priority: normal severity: normal status: open title: Crash during decoding using UTF-16/32 and custom error handler type: crash versions: Python 3.5, Python 3.7 Added file: