[issue6395] Infinite Recursion during Unpickling a codecs Object

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is also a problem with deepcopying:

>>> class MemReader:
... def __init__(self, data):
... self.buf = memoryview(data).cast('B')
... def read(self, size=-1):
... if size < 0:
... size = len(self.buf)
... res = bytes(self.buf[:size])
... self.buf = self.buf[size:]
... return res
... def close():
... pass
... def __deepcopy__(self, memo):
... return MemReader(self.buf)
... def __reduce__(self):
... return MemReader, (bytes(self.buf),)
... 
>>> import codecs, copy
>>> s1 = codecs.getreader('utf-8')(MemReader(b'abcd'))
>>> s2 = copy.deepcopy(s1)
>>> s1.read()
'abcd'
>>> s2.read()
b'abcd'
>>> s2
<__main__.MemReader object at 0xb701988c>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 02.12.2015 21:29, Serhiy Storchaka wrote:
> 
> Serhiy Storchaka added the comment:
> 
>> I think all we need to do is add a .__reduce__()
>> method to StreamWriter and StreamReader, which then
>> raises a PickleError.
> 
> Rather TypeError. Yes, it is the least that we should to do in maintained 
> releases. If codecs_stream_delegating_2.patch is considered too drastic for 
> bugfix. But this can be only a part of problem. May be there are issues with 
> other optional special methods. And adding __reduce_ex__ breaks subclass 
> pickleability if it was implemented with __getstate__ and __getnewargs__.
> 
> Here is a patch for this way.

Thanks. I think using __reduce__ instead of __reduce_ex__ is
safer, since subclasses will more likely override __reduce__
than __reduce_ex__.

The other approach will have too many backwards incompatible side
effects, e.g. the repr() over StreamReader instances would change.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> I think all we need to do is add a .__reduce__()
> method to StreamWriter and StreamReader, which then
> raises a PickleError.

Rather TypeError. Yes, it is the least that we should to do in maintained 
releases. If codecs_stream_delegating_2.patch is considered too drastic for 
bugfix. But this can be only a part of problem. May be there are issues with 
other optional special methods. And adding __reduce_ex__ breaks subclass 
pickleability if it was implemented with __getstate__ and __getnewargs__.

Here is a patch for this way.

--
Added file: http://bugs.python.org/file41217/codecs_stream_forbid_pickling.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Added tests for pickling and deepcopying.

--
Added file: http://bugs.python.org/file41215/codecs_stream_delegating_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Marc-Andre Lemburg]

Marc-Andre Lemburg added the comment:

On 02.12.2015 20:16, Serhiy Storchaka wrote:
> 
> Serhiy Storchaka added the comment:
> 
>> If the StreamWriter/Reader cannot pickle the underlying stream (which is
>> probably always the case), why should the object itself be pickleable ?
> 
> io.BytesIO() and io.StringIO() are pickleable.

Ok, but I still don't see the use case :-)

I think all we need to do is add a .__reduce__()
method to StreamWriter and StreamReader, which then
raises a PickleError.

Example:

>>> import sys, codecs, pickle
>>> r = codecs.getreader('latin-1')
>>> class MyReader(r):
...def __reduce__(self, *args):
...   raise pickle.PickleError
...
>>> s = MyReader(sys.stdin)
>>> pickle.dumps(s)
Traceback (most recent call last):
  File "", line 1, in 
  File "", line 3, in __reduce__
_pickle.PickleError
> (3)__reduce__()

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

> If the StreamWriter/Reader cannot pickle the underlying stream (which is
probably always the case), why should the object itself be pickleable ?

io.BytesIO() and io.StringIO() are pickleable.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Implementing only .__get/setstate__() doesn't fix all problem. We have 
implement also __getnewargs_ex__(). But implemented __getnewargs_ex__() has 
priority over __getnewargs__() implemented in subclasses.

And may be there are problems with other optional special methods that are 
incorrectly delegated to the stream in codecs IO classes.

I think more reliable way is to disallow delegating all special (or may be even 
private) methods. Here is a patch.

--
assignee:  -> serhiy.storchaka
nosy: +serhiy.storchaka
stage:  -> patch review
type: enhancement -> behavior
versions: +Python 3.5, Python 3.6 -Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
keywords: +patch
Added file: http://bugs.python.org/file41184/codecs_stream_delegating.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[Georg Brandl]

Changes by Georg Brandl :


--
versions: +Python 3.3, Python 3.4 -Python 3.2

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue6395] Infinite Recursion during Unpickling a codecs Object

[ThomasH]

ThomasH  added the comment:

Sounds good to me. The main intention of this bug is not to make codecs
objects pickleable by all means (I don't know if this would be sensible
and/or possible), but at least to have them degrade gracefully during
pickling, particularly without infinite recursion. I've changed the bug
title to reflect this.

--
title: Add Pickle Support to the codecs Module -> Infinite Recursion during 
Unpickling a codecs Object

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com