[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Nick Coghlan added the comment: People may use SystemError for other purposes, but the docs are pretty clear it is only for internal errors that indicate an interpreter bug: http://docs.python.org/dev/library/exceptions.html#SystemError Extension modules or an embedding application passing in

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Alexander Belopolsky wrote: > > Alexander Belopolsky added the comment: > > On Thu, Feb 24, 2011 at 3:54 PM, Antoine Pitrou > wrote: > .. >> I've committed the part of the patch which disallows a NULL data pointer >> with PyMemoryView_FromBuffer in r885

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Nick Coghlan added the comment: A SystemError indicates that an internal API was given bogus input or produces bogus output (i.e. we screwed up somewhere, or a third party is messing with interfaces they shouldn't be) If data validation fails for part of the public C API (whether it is visibl

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > It seems appropriate to consult python-dev on this. I thought > ValueError was for values that are valid Python objects but out of > acceptable range of the function. Errors that can only be triggered > in C code normally handled with either assert() or rais

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: It seems appropriate to consult python-dev on this. I thought ValueError was for values that are valid Python objects but out of acceptable range of the function. Errors that can only be triggered in C code normally handled with either assert() or raise

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > > I've committed the part of the patch which disallows a NULL data pointer > > with PyMemoryView_FromBuffer in r88550 and r88551. > > Is it possible to create such buffer in Python (other than by > exploiting a bug or writing a rogue extension module)? If no

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: On Thu, Feb 24, 2011 at 3:54 PM, Antoine Pitrou wrote: .. > I've committed the part of the patch which disallows a NULL data pointer > with PyMemoryView_FromBuffer in r88550 and r88551. Is it possible to create such buffer in Python (other than by exploi

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: I've committed the part of the patch which disallows a NULL data pointer with PyMemoryView_FromBuffer in r88550 and r88551. -- ___ Python tracker

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: Committed in r88546 (3.3) and r88548 (3.2). Note that a simple work-around before 3.2.1 is to spell encoding as 'latin-1' or 'iso-8859-1' in pickle.loads(). -- components: +Extension Modules -Library (Lib) resolution: -> fixed stage: commit rev

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Changes by Alexander Belopolsky : Removed file: http://bugs.python.org/file20856/issue11286.diff ___ Python tracker ___ ___ Python-bugs-list m

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: I am attaching a new version of issue11286.diff which fixes the issue by removing special handling of n == 0 case from _Unpickler_Read(). Note that _Unpickler_Read() (formerly known as unpickler_read()) only started to return null pointer instead of a

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Changes by Alexander Belopolsky : -- assignee: -> belopolsky ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: h

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: On Wed, Feb 23, 2011 at 10:22 AM, Antoine Pitrou wrote: .. > Well, a theoretical argument could be made that some codec could return > a non-empty string when asked to decode an empty bytestring, but I'm not > sure it has much practical worth :) I was th

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > It may be better to modify _Unpickler_Read() so that it returns > self->input_buffer (or even self->input_buffer + self->next_read_idx) > for zero n. I agree this would be better for readability and maintainability. > On the other hand, my patch also elimina

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: Antoine is right, my patch is only of "discussion" quality and if my approach gets support, I will produce a more polished patch. While I am fairly certain that this bug should be fixed where it was introduced, namely in the _pickle module, I am not sur

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: So, about Alexander's patch: - it lacks a test - it doesn't solve the issue with PyUnicode_Decode's confusing error message when a NULL is passed ("ValueError: operation forbidden on released memoryview object"); if we want to disallow NULL, we should have a c

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: Oops, I hadn't seen Alexander's patch. Sorry. -- ___ Python tracker ___ ___ Python-bugs-list mailin

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Jesús Cea Avión wrote: > > Jesús Cea Avión added the comment: > > What if we commit Antoine patch for 3.2.x, and the "correct" patch for py3k > trunk?. > > I am actually +1 to Marc-Andre. I feel in my guts that the provided patch is > hidding a deeper

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > Please go with Alexander's solution of fixing the higher level code > rather than silently trying to introduce a new feature in > PyUnicode_Decode() that hides programming errors. I'm sorry, I'm perfectly fine with my own patch, so someone else will have to p

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Please go with Alexander's solution of fixing the higher level code rather than silently trying to introduce a new feature in PyUnicode_Decode() that hides programming errors. Thanks. -- ___ Python tracker

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > > Nope. Your suggestion would be a new feature and those are not > > allowed in patch level releases. > > What new feature are you talking about? I think you misunderstood the > actual issue: NULL as an empty string *worked* in 3.1 And, with a very high like

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Jesús Cea Avión added the comment: What if we commit Antoine patch for 3.2.x, and the "correct" patch for py3k trunk?. I am actually +1 to Marc-Andre. I feel in my guts that the provided patch is hidding a deeper issue. But avoiding surprises for third parties in 3.2.1 is a good idea. -

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > > So, IMO, practicality beats purity here. Especially since it is bound to > > land in a bugfix release (3.2.1), which users don't expect to produce > > regressions in their own code. > > Nope. Your suggestion would be a new feature and those are not > allowe

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > >> PyUnicode_Decode() et al. are conversion functions and these >> require valid content to work on. Passing in a NULL pointer >> does not fit that specification and so allowing for this >> wou

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > PyUnicode_Decode() et al. are conversion functions and these > require valid content to work on. Passing in a NULL pointer > does not fit that specification and so allowing for this > would hide programming errors. "Valid content" doesn't mean a lot when the

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > >> Antoine Pitrou wrote: >>> >>> Antoine Pitrou added the comment: >>> > I am not sure PyUnicode_Decode() should treat NULL as an empty string. Definitely not. That would hide pr

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > Antoine Pitrou wrote: > > > > Antoine Pitrou added the comment: > > > >>> I am not sure PyUnicode_Decode() should treat NULL as an empty string. > >> > >> Definitely not. That would hide programming errors. > > > > Well, this could break some third-party c

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Antoine Pitrou wrote: > > Antoine Pitrou added the comment: > >>> I am not sure PyUnicode_Decode() should treat NULL as an empty string. >> >> Definitely not. That would hide programming errors. > > Well, this could break some third-party code. If that

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: > > I am not sure PyUnicode_Decode() should treat NULL as an empty string. > > Definitely not. That would hide programming errors. Well, this could break some third-party code. -- title: Some "trivial" python 2.x pickles fails to load in Python

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Marc-Andre Lemburg added the comment: Alexander Belopolsky wrote: > > I am not sure PyUnicode_Decode() should treat NULL as an empty string. Definitely not. That would hide programming errors. -- nosy: +lemburg title: Some "trivial" python 2.x pickles fails to load in Python 3.2 -> So

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Alexander Belopolsky added the comment: I am not sure PyUnicode_Decode() should treat NULL as an empty string. Decoding empty string is wasteful and if the caller knows that the string is empty, it should skip decoding because the result is empty. Providing *two* ways to invoke expensive Py

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Changes by Alexander Belopolsky : -- nosy: +belopolsky ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: http://m

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Nick Coghlan added the comment: Antoine's patch looks good to me. Given the assumptions in the memoryview code, disallowing NULL for the buf pointer sounds like the right thing to do, even for zero-length buffers. -- ___ Python tracker

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: Here is a patch with tests. -- keywords: +patch stage: needs patch -> patch review Added file: http://bugs.python.org/file20842/null_ptr_buffer.patch ___ Python tracker __

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: This is because of the buffer pointer passed to the codecs machinery being NULL when the empty string is being decoded. Quick patch follows (needs a test). Also, it is not clear whether it is allowed to store a NULL pointer in the "buf" member of a Py_buffer

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Jesús Cea Avión added the comment: Storing the pickle using protocol 0, it works OK. Using protocol 1 or 2, it fails. -- ___ Python tracker ___

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Antoine Pitrou added the comment: Ok, so it boils down to: >>> pickle.loads(b'\x80\x02}q\x00U\x00q\x01h\x01s.') {'': ''} >>> pickle.loads(b'\x80\x02}q\x00U\x00q\x01h\x01s.', encoding='latin1') Traceback (most recent call last): File "", line 1, in ValueError: operation forbidden on released

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

Jesús Cea Avión added the comment: I got a trivial (41 bytes long) reproductable case: Pickle the following structure in Python 2.7, and try to unpickle with Python 3.2, using a "latin1" encoding: """ {'ya_volcados': {'comment': ''}} """ Load with: """ #!/usr/local/bin/python3 import pickl

[issue11286] Some "trivial" python 2.x pickles fails to load in Python 3.2

New submission from Jesús Cea Avión : I have 10MB pickled structure generated in Python 2.7. I only use basic types (no clases) like sets, dictionaries, lists, strings, etc. The pickle stores a lot of strings. Some of them should be "bytes", while other should be "unicode". My idea is to import