[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Antoine Pitrou pit...@free.fr added the comment: Committed to py3k and release30-maint in r67760 and r67759. Needs backporting to 2.x. -- priority: release blocker - normal ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4574

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Antoine Pitrou pit...@free.fr added the comment: Backported to trunk and 2.6.2 in r67762 and r67764. -- resolution: - fixed status: open - closed ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4574

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Antoine Pitrou pit...@free.fr added the comment: A couple of suggestions: - if IncrementalNewlineDecoder gets an encoding argument, it can also instantiate the decoder itself; that way the API is a bit simpler - to encode '\r' without the BOM, you can e.g. use an incremental encoder and encode

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Changes by Antoine Pitrou pit...@free.fr: ___ Python tracker rep...@bugs.python.org http://bugs.python.org/issue4574 ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Antoine Pitrou pit...@free.fr added the comment: Here is a simpler patch with a different approach and a lot of tests. The advantage is that it doesn't break the API. Added file: http://bugs.python.org/file12344/utf16_newlines.patch ___ Python tracker

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Antoine Pitrou pit...@free.fr added the comment: This new variant also removes the dangerous hack in getstate / setstate. Added file: http://bugs.python.org/file12345/utf16_newlines2.patch ___ Python tracker rep...@bugs.python.org

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Gregory P. Smith g...@krypto.org added the comment: utf16_newlines2.patch looks good to me. This is a data corruption issue. If it is deferred for 3.0.1 it must be fixed in 3.0.2. +1 on putting this in 3.0.1. -- assignee: - pitrou nosy: +gregory.p.smith priority: - release blocker

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

STINNER Victor [EMAIL PROTECTED] added the comment: The bug is in IncrementalNewlineDecoder, not in the codec nor TextIOWrapper. -- nosy: +haypo ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4574

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

STINNER Victor [EMAIL PROTECTED] added the comment: Smaller example to demonstrate the problem. Added file: http://bugs.python.org/file12295/dec.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4574 ___

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Changes by STINNER Victor [EMAIL PROTECTED]: Removed file: http://bugs.python.org/file12295/dec.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4574 ___ ___

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Changes by STINNER Victor [EMAIL PROTECTED]: Added file: http://bugs.python.org/file12296/incremental_newline_decoder_bug.py ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4574 ___

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

STINNER Victor [EMAIL PROTECTED] added the comment: Here is a patch for test_io.py: check the problem by adding new encodings to TextIOWrapperTest.testNewlines(). -- keywords: +patch Added file: http://bugs.python.org/file12297/test_io.patch ___

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

STINNER Victor [EMAIL PROTECTED] added the comment: Ugly patch to fix this issue: - add more regression tests for charsets UTF-16*, UTF-32* - add mandatory argument encoding to io.IncrementalNewlineDecoder constructor = BREAK THE API - use the encoding the encode \r - most ulgy hack: strip

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

New submission from John Machin [EMAIL PROTECTED]: Problem in the newline handling in io.py, class IncrementalNewlineDecoder, method decode. It reads text files in 128- byte chunks. Converting CR LF to \n requires special case handling when '\r' is detected at the end of the decoded chunk in

[issue4574] reading UTF16-encoded text file crashes if \r on 64-char boundary

Changes by Antoine Pitrou [EMAIL PROTECTED]: -- nosy: +pitrou ___ Python tracker [EMAIL PROTECTED] http://bugs.python.org/issue4574 ___ ___ Python-bugs-list mailing list