[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
resolution:  -> fixed
stage: patch review -> resolved
status: open -> closed
versions:  -Python 2.7, Python 3.4

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Roundup Robot]

Roundup Robot added the comment:

New changeset f3c54cbac3de by Serhiy Storchaka in branch '3.5':
Issue #25718: Fixed pickling and copying the accumulate() iterator with total 
is None.
https://hg.python.org/cpython/rev/f3c54cbac3de

New changeset 49c035b30e18 by Serhiy Storchaka in branch 'default':
Issue #25718: Fixed pickling and copying the accumulate() iterator with total 
is None.
https://hg.python.org/cpython/rev/49c035b30e18

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Raymond Hettinger]

Raymond Hettinger added the comment:

+1

--
assignee: rhettinger -> serhiy.storchaka

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Raymond, could you please look at the patch?

--
assignee: kristjan.jonsson -> rhettinger
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping again.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ping.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

What would you say about the patch Kristján?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Kristján Valur Jónsson]

Kristján Valur Jónsson added the comment:

This could be fixed by saving the accumulate state in a tuple.  It would break 
protocol, though.  I don't recall the rules for backwards compatibility of 
pickles.
I've argued before that the state of runtime structures such as generators is 
so intimately tied with the currently executing python that we shouldn't worry 
to much about it.  This sort of stuff is used for caches, IPC, and so on.  But 
it's not my call.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Well, we can fix this without breaking the protocol. The accumulate() object 
with state is None can be reconstructed as islice(accumulate(chain([None], 
iterator)), 1, None).

Proposed patch implements this solution.

--
Added file: 
http://bugs.python.org/file41261/itertools_accumulate_state_is_none.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Roundup Robot]

Roundup Robot added the comment:

New changeset 095d21df9374 by Serhiy Storchaka in branch '3.4':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/095d21df9374

New changeset 3d39e1e2dcb3 by Serhiy Storchaka in branch '2.7':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/3d39e1e2dcb3

New changeset fecb07050aae by Serhiy Storchaka in branch '3.5':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/fecb07050aae

New changeset 962086677306 by Serhiy Storchaka in branch 'default':
Issue #25718: Fixed copying object with state with boolean value is false.
https://hg.python.org/cpython/rev/962086677306

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The bug in the copy module is fixed. As for accumulate's __reduce__, I don't 
think we can do something.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

In this case yes, you are unlucky. But this is an artificial example, and I 
don't think this happens in any real code.

However specific meaning of None should be documented (there are other wrong 
words about __getstate__).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

__reduce__() and __setstate__() methods of itertools.accumulate() look correct. 
The problem is not in itertools.accumulate(), but in the copy module. It uses 
the same __reduce__ protocol as pickle, but in different way. In the pickle 
module (in both Python and C implementations) state value is ignored only if it 
is None. In the copy module any state with boolean value is False is ignored.

Proposed patch fixes the copy module.

--
components: +Library (Lib) -Extension Modules
nosy: +alexandre.vassalotti, serhiy.storchaka
stage:  -> patch review
versions: +Python 2.7, Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
keywords: +patch
Added file: http://bugs.python.org/file41151/copy_state_is_false.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Armin Rigo]

New submission from Armin Rigo:

itertools.accumulate() has got methods __reduce__() and __setstate__() which 
fail at saving/restoring the state in all cases.  Example:

   >>> a = itertools.accumulate([0.0, 42])
   >>> next(a)
   0.0
   >>> b = copy.deepcopy(a)
   >>> next(a)
   42.0
   >>> next(b)
   42   # should have been the same as the previous result

More precisely, the problem occurs when the C-level "total" field has some 
value whose truth-value is false.  Then __reduce__/__setstate__ will not 
restore the value of this field, but keep it NULL.  That's why in the example 
above the intermediate total of 0.0 is forgotten, and the next(b) call will 
again just pick the next value without doing any addition.  (The problem can 
hurt more than in this example if we use a custom function instead of addition.)

A fix is easy but, as far as I can tell, it requires breaking the pickled 
format of accumulate() iterators that have already started.

--
components: Extension Modules
messages: 255252
nosy: arigo
priority: normal
severity: normal
status: open
title: itertools.accumulate __reduce__/__setstate__ bug
type: behavior
versions: Python 3.6

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25718] itertools.accumulate __reduce__/__setstate__ bug

[Raymond Hettinger]

Changes by Raymond Hettinger :


--
assignee:  -> kristjan.jonsson
nosy: +kristjan.jonsson

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com