[issue11299] Allow deepcopying and pickling paused generators

[Ram Rachum]

New submission from Ram Rachum :

Please allow to deepcopy and to pickle paused generators, including all their 
state.

This is implemented in Pypy:

Python 2.5.2 (335e875cb0fe, Dec 28 2010, 20:31:56)
[PyPy 1.4.1] on win32
Type "copyright", "credits" or "license()" for more information.
DreamPie 1.1.1
>>> import pickle, copy
>>> def next(thing): # For compatibility
... return thing.next()
>>> def g():
... for i in range(4):
... yield i
>>> list(g())
[0, 1, 2, 3]
>>> live_generator = g()
>>> next(live_generator)
0
>>> next(live_generator)
1

Now `live_generator` holds a generator which is in the middle of its operation. 
It went through 0 and 1, and it still has 2 and 3 to yield.

We deepcopy it: 

>>> live_generator_deepcopy = copy.deepcopy(live_generator)

The deepcopied generator assumes the same state of the original one. Let's 
exhaust it:  

>>> list(live_generator_deepcopy)
[2, 3]
>>> list(live_generator_deepcopy)
[]

Pypy also lets us pickle and unpickle the live generator:

>>> live_generator_pickled = pickle.dumps(live_generator)
>>> live_generator_unpickled = pickle.loads(live_generator_pickled)
>>> list(live_generator_unpickled)
[2, 3]
>>> list(live_generator_unpickled)
[]

And the original live generator was unchanged by all these operations:

>>> list(live_generator)
[2, 3]
>>> list(live_generator)
[]

All the above was demonstrated in Pypy. In Python 3.2, trying to pickle a live 
generator raises this exception:

>>> pickle.dumps(live_generator)
Traceback (most recent call last):
  File "", line 1, in 
_pickle.PicklingError: Can't pickle : attribute 
lookup builtins.generator failed

And trying to deepcopy one raises this exception:

>>> copy.deepcopy(live_generator)
Traceback (most recent call last):
  File "", line 1, in 
  File "c:\Python32\lib\copy.py", line 174, in deepcopy
y = _reconstruct(x, rv, 1, memo)
  File "c:\Python32\lib\copy.py", line 285, in _reconstruct
y = callable(*args)
  File "c:\Python32\lib\copyreg.py", line 88, in __newobj__
return cls.__new__(cls, *args)
TypeError: object.__new__(generator) is not safe, use 
generator.__new__()

It would be nice if Python 3.2 could pickle and deepcopy live generators.

--
components: Library (Lib)
messages: 129213
nosy: cool-RR
priority: normal
severity: normal
status: open
title: Allow deepcopying and pickling paused generators
type: feature request
versions: Python 3.3

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Ram Rachum]

Ram Rachum  added the comment:

P.S. I'm willing to write a test-case if it will help.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Test cases always help when appropriate.
A link to the Pypy code that does this might also help.
Or perhaps ask them to submit a patch to this issue.

--
nosy: +alexandre.vassalotti, pitrou, terry.reedy
stage:  -> test needed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Ram Rachum]

Ram Rachum  added the comment:

Tests attached.

--
Added file: http://bugs.python.org/file20898/test_live_generator.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Alexandre Vassalotti]

Alexandre Vassalotti  added the comment:

Although, I would really like to see support of pickling generators. It is not 
really possible in CPython. This is recurrent request. I even wrote a small 
article about it.

http://peadrop.com/blog/2009/12/29/why-you-cannot-pickle-generators/

Looking how PyPy did it, I see portability problems with the fact it dumps the 
byte-code of the generator to disk. Python's documentation clearly states that 
the byte-code is an implementation details and can (and does) change between 
releases. Hence, this method is not really suitable for pickle which needs to 
remain compatible across releases.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Ram Rachum]

Ram Rachum  added the comment:

Hi Alexandre,

I read your blog post, but I don't understand-- Why does bytecode need to be 
pickled in order to pickle live generators? I understand that the local 
variables need to be pickled, (and let's assume they're all pickleable,) and 
that a pointer to the current instruction needs to be pickled, but why the 
bytecode? When you pickle a normal function or a class, no bytecode gets 
pickled, so why does it have to be pickled here?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Alexandre Vassalotti]

Alexandre Vassalotti  added the comment:

The value of the instruction pointer depends on the byte-code. So it's not 
portable either.

But, the bigger issue is the fact generator objects do not have names we can 
refer to, unlike top-level functions and classes which pickle supports. 
Similarly, we can't pickle lambdas and nested-functions for this exact reason.

Personally, I haven't found a way around this. But, that doesn't mean there 
isn't one. If you find one, I will more than pleased to review it.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Ram Rachum]

Ram Rachum  added the comment:

*"generator objects do not have names we can refer to"*

How about referring to the generator function that created them and to all the 
arguments?

Regarding instruction pointer, I really don't know the internals of how this 
works. Can we make some arrangement so we can specify line number (and possibly 
column number too) as a cross-interpreter instruction pointer?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Terry J. Reedy]

Terry J. Reedy  added the comment:

Alexandre,
Do the considerations against pickling apply to deep copying? It would seem 
that copying bytecode and pointer within a run should be ok.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue11299] Allow deepcopying and pickling paused generators

[Bruno Dupuis]

Changes by Bruno Dupuis :


--
nosy: +bruno.dupuis

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
http://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com