[issue17828] More informative error handling when encoding and decoding

Christian Heimes added the comment: Coverity has found two issues in your patch. I have fixed them both. -- nosy: +christian.heimes ___ Python tracker ___ ___

[issue17828] More informative error handling when encoding and decoding

Roundup Robot added the comment: New changeset 26121ae22016 by Christian Heimes in branch 'default': Issue #17828: _PyObject_GetDictPtr() may return NULL instead of a PyObject** http://hg.python.org/cpython/rev/26121ae22016 -- ___ Python tracker

[issue17828] More informative error handling when encoding and decoding

Roundup Robot added the comment: New changeset 99ba1772c469 by Christian Heimes in branch 'default': Issue #17828: va_start() must be accompanied by va_end() http://hg.python.org/cpython/rev/99ba1772c469 -- ___ Python tracker

[issue17828] More informative error handling when encoding and decoding

Roundup Robot added the comment: New changeset 854a2cea31b9 by Nick Coghlan in branch 'default': Close #17828: better handling of codec errors http://hg.python.org/cpython/rev/854a2cea31b9 -- nosy: +python-dev resolution: -> fixed stage: commit review -> committed/rejected status: open

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Patch for the final version that I'm about to commit. - I realised the exception chaining would only trigger for the encode() and decode() methods, when it was equally applicable to the codecs.encode() and codecs.decode() functions, so I updated the test cases a

[issue17828] More informative error handling when encoding and decoding

Marc-Andre Lemburg added the comment: On 10.11.2013 15:39, Nick Coghlan wrote: > > On 10 November 2013 23:21, Marc-Andre Lemburg wrote: >> >> This doesn't look right: >> >> diff -r 1ee45eb6aab9 Include/pyerrors.h >> --- a/Include/pyerrors.hSat Nov 09 23:15:52 2013 +0200 >> +++ b/Include

[issue17828] More informative error handling when encoding and decoding

Changes by Nick Coghlan : Added file: http://bugs.python.org/file32562/issue17828_improved_codec_errors_v6.diff ___ Python tracker ___ ___ Py

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: On 10 November 2013 23:21, Marc-Andre Lemburg wrote: > > This doesn't look right: > > diff -r 1ee45eb6aab9 Include/pyerrors.h > --- a/Include/pyerrors.hSat Nov 09 23:15:52 2013 +0200 > +++ b/Include/pyerrors.hSun Nov 10 22:54:04 2013 +1000 > ... >

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: On 10 November 2013 23:21, Marc-Andre Lemburg wrote: > > Marc-Andre Lemburg added the comment: > > On 10.11.2013 14:03, Nick Coghlan wrote: >> >> Updated patch (v5) with a more robust chaining mechanism provided as a >> private "_PyErr_TrySetFromCause" API. This

[issue17828] More informative error handling when encoding and decoding

Marc-Andre Lemburg added the comment: On 10.11.2013 14:03, Nick Coghlan wrote: > > Updated patch (v5) with a more robust chaining mechanism provided as a > private "_PyErr_TrySetFromCause" API. This version eliminates the previous > whitelist in favour of checking directly for the ability to r

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Updated patch (v5) with a more robust chaining mechanism provided as a private "_PyErr_TrySetFromCause" API. This version eliminates the previous whitelist in favour of checking directly for the ability to replace the exception with another instance of the same

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Updated patch adds systematic tests for the new error handling to test_codecs.TransformTests I also moved the codecs changes up to a "Codec handling improvements" section. My rationale for doing that is that this is actually a pretty significant usability enhan

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Checking the other binary<->binary and str<->str codecs with input type and value restrictions: - they all throw TypeError and get wrapped appropriately when asked to encode str input (rot_13 throws the output type error) - rot_13 throws an appropriately wrappe

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: New and improved implementation attached that extracts the exception chaining to a helper functions and calls it only when it is the call in to the codecs machinery that failed (eliminating the need for the output flag, and covering decoding as well as encoding)

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: I think I figured out a better way to structure this that avoids the need for the output flag and is more easily expanded to whitelist additional exception types as safe to wrap. I'll try to come up with a new patch tonight. -- assignee: -> ncoghlan _

[issue17828] More informative error handling when encoding and decoding

Marc-Andre Lemburg added the comment: On 04.11.2013 14:30, STINNER Victor wrote: > > It would be simpler to just drop these custom codecs (rot13, bz2, hex, etc.) > instead of helping to use them :-) -1 for the same reasons I keep repeating over and over and over again :-) The codec system was

[issue17828] More informative error handling when encoding and decoding

STINNER Victor added the comment: It would be simpler to just drop these custom codecs (rot13, bz2, hex, etc.) instead of helping to use them :-) -- nosy: +haypo ___ Python tracker

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: The other thing is that this patch doesn't wrap AttributeError. I'm OK with that, since I believe the only codec in the standard library that currently throws that for a bad input type is rot_13. -- ___ Python tracker

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Ah, came up with a relatively simple solution based on an internal helper function with an optional output flag: >>> import codecs >>> codecs.encode(b"hello", "bz2_codec").decode("bz2_codec") Traceback (most recent call last): File "", line 1, in TypeError: 'b

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Updated patch. The results of this suggests to me that the input wrappers are likely infeasible at this point in time, but improving the errors for the wrong *output* type is entirely feasible. Since the main conversion we need to prompt is things like "binary_o

[issue17828] More informative error handling when encoding and decoding

Ezio Melotti added the comment: The attached proof of concept catches Type/ValueError in str.encode and raises another exception with a better message: >>> 'example'.encode('hex_codec') Traceback (most recent call last): File "", line 1, in TypeError: invalid input type for hex_codec codec ('

[issue17828] More informative error handling when encoding and decoding

Ezio Melotti added the comment: To summarize: * str.encode does only str->bytes; * bytes.decode does only bytes-> str; * codecs.encode/decode do obj->obj; The things that could go wrong are: 1) the input type is wrong (i.e. the codec doesn't accept the type of the input); 2) the input valu

[issue17828] More informative error handling when encoding and decoding

Ezio Melotti added the comment: The attached patch changes the error message of str.encode/bytes.decode when the codec returns the wrong type: >>> import codecs >>> 'example'.encode('rot_13') TypeError: encoder returned 'str' instead of 'bytes', use codecs.decode for str->str conversions >>> c

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Ezio pointed out on IRC that the extra type checks in str.encode, bytes.decode and bytearray.decode should reference the appopriate codecs module function in addition to the codec in use. So if str.encode produces something other than bytes, it should reference

[issue17828] More informative error handling when encoding and decoding

Changes by Barry A. Warsaw : -- nosy: +barry ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: Here's an example of the specific type errors raised by additional checks in the text-encoding specific methods. I believe the main improvement needed here is to mention the encoding name in the exception message: "example".encode("rot_13") Traceback (most recen

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: I tracked down the proximate cause of the weird exception in the bytes.decode case: the base64 module only accepts bytes and bytearray objects, instead of using memoryview to accept anything that supports the buffer API and provides a C-contiguous 8-bit view of

[issue17828] More informative error handling when encoding and decoding

Changes by Florent Xicluna : -- nosy: +flox ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://mail.python.

[issue17828] More informative error handling when encoding and decoding

Nick Coghlan added the comment: There may also be some specific improvement to be made to str.encode, bytes.decode and bytearray.decode in relation to the additional type checks they do to enforce the appropriate input and output types (see the bizarre "expected bytes, not memoryview" example)

[issue17828] More informative error handling when encoding and decoding

Changes by Ezio Melotti : -- nosy: +ezio.melotti stage: -> needs patch type: -> enhancement ___ Python tracker ___ ___ Python-bugs-l

[issue17828] More informative error handling when encoding and decoding

New submission from Nick Coghlan: Passing the wrong types to codecs can currently lead to rather confusing exceptions, like: >>> b"ZXhhbXBsZQ==\n".decode("base64_codec") Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.2/encodings/base64_