[issue25776] More compact pickle of iterators etc

[Raymond Hettinger]

Raymond Hettinger added the comment:

I would like to pass on this one.  It isn't common to pickle iterators, the 
savings is very small, and some of the code looks less pleasant.

--
resolution:  -> rejected
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


Added file: http://bugs.python.org/file44430/iterators_pickle_4.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

The patch is synchronized with current sources. Added mandatory braces.

--
Added file: http://bugs.python.org/file42077/iterators_pickle_3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

There is the extra special case code for bytes and bytearray iterators and 
reversed list iterator. The patch just extends the optimizations to other 
builtin iterators.

In the example with itertools.islice() the overhead of pickling redundant data 
is 20%. If there are a lot of iterators, the overhead can be up to 90%.

>>> import pickle, pickletools, itertools
>>> len(pickletools.optimize(pickle.dumps([itertools.islice('abcdefgh', 4) for 
>>> i in range(1000)], 4)))
Unpatched: 23059
Patched:   12059

Of course this is degenerated case. But if the data contains a lot of initial 
iterators of the same sequence or of short sequences, or exhausted iterators, 
the benefit can be not small.

Yet one benefit from the patch is that it speeds up copying and deepcopying 
initial and exhausted iterators.

$ ./python -m timeit -s "from itertools import islice; from copy import copy; 
it = islice('abcdefgh', 4)" -- "copy(it)"
Unpatched: 7.37 usec per loop
Patched:   6.4 usec per loop

$ ./python -m timeit -s "from itertools import islice; from copy import 
deepcopy; it = islice('abcdefgh', 4)" -- "deepcopy(it)"
Unpatched: 41.7 usec per loop
Patched:   32.6 usec per loop

$ ./python -m timeit -s "from copy import copy; it = iter('abcdefgh')" -- 
"copy(it)"
Unpatched: 10.4 usec per loop
Patched:   9.67 usec per loop

$ ./python -m timeit -s "from copy import deepcopy; it = iter('abcdefgh')" -- 
"deepcopy(it)"
Unpatched: 21.1 usec per loop
Patched:   18.3 usec per loop

$ ./python -m timeit -s "from copy import copy; it = iter(list('abcdefgh'))" -- 
"copy(it)"
Unpatched: 10.3 usec per loop
Patched:   9.54 usec per loop

$ ./python -m timeit -s "from copy import deepcopy; it = 
iter(list('abcdefgh'))" -- "deepcopy(it)"
Unpatched: 39.7 usec per loop
Patched:   36.8 usec per loop

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Raymond Hettinger]

Raymond Hettinger added the comment:

My first impression is that this isn't worth the extra special case code.  The 
savings is very small (only a few bytes) and it isn't common to pickle these 
iterators.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
assignee:  -> rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Ah, I already proposed tests in separate issue.

Raymond, could you please look at patches? The touch the code maintained by 
you: itertools and deque.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


Removed file: http://bugs.python.org/file41853/iterators_pickle_tests.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Updated patch conforms to new tests and to following rule: exhausted iterators 
of mutable sequences can be replaced with iter(()), non-exhausted iterators 
can't.

--
Added file: http://bugs.python.org/file41854/iterators_pickle_2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Serhiy Storchaka added the comment:

Proposed patch extends tests for pickling iterators of mutable sequences. Now 
tested iterators in different states: initial (no iterated yet), running (in 
the middle of iteration), empty (just the last item was emitted) and exhausted 
(tried to iterate past the end).

--
Added file: http://bugs.python.org/file41853/iterators_pickle_tests.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


Removed file: http://bugs.python.org/file41207/iterators_pickle.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

Changes by Serhiy Storchaka :


--
dependencies: +Add new tests for pickling iterators of mutable sequences

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue25776] More compact pickle of iterators etc

[Serhiy Storchaka]

New submission from Serhiy Storchaka:

Proposed patch makes a number of classes produce more compact pickle data in 
common case. This includes iterators of list, tuple, str, bytes, bytearray, 
enumerate, array, deque, iterator object for classes with __getitem__, some 
itertools iterators, and non-iterator objects: slice, bytearray, deque. This is 
achieved by omitting default constructor arguments or state.

Exhausted iterators are pickled as iter(()). This is not new, exhausted bytes, 
and bytearray iterators, and reversed list iterator are already pickled as 
iter('') or iter([]) correspondingly. iter(()) is just the simplest way to 
create an empty iterator and it has the most compact pickle representation.

An example.

Unpatched:
>>> import pickle, pickletools, itertools
>>> len(pickle.dumps(itertools.islice('abcdefgh', 4), 3))
80
>>> len(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh', 4), 3)))
66
>>> pickletools.dis(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh',
>>>  4), 3)))
0: \x80 PROTO  3
2: cGLOBAL 'itertools islice'
   20: (MARK
   21: cGLOBAL 'builtins iter'
   36: XBINUNICODE 'abcdefgh'
   49: \x85 TUPLE1
   50: RREDUCE
   51: KBININT10
   53: bBUILD
   54: KBININT10
   56: KBININT14
   58: KBININT11
   60: tTUPLE  (MARK at 20)
   61: RREDUCE
   62: KBININT10
   64: bBUILD
   65: .STOP
highest protocol among opcodes = 2

Patched:
>>> len(pickle.dumps(itertools.islice('abcdefgh', 4), 3))
69
>>> len(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh', 4), 3)))
55
>>> pickletools.dis(pickletools.optimize(pickle.dumps(itertools.islice('abcdefgh',
>>>  4), 3)))
0: \x80 PROTO  3
2: cGLOBAL 'itertools islice'
   20: cGLOBAL 'builtins iter'
   35: XBINUNICODE 'abcdefgh'
   48: \x85 TUPLE1
   49: RREDUCE
   50: KBININT14
   52: \x86 TUPLE2
   53: RREDUCE
   54: .STOP
highest protocol among opcodes = 2

--
components: Extension Modules, Interpreter Core
files: iterators_pickle.diff
keywords: patch
messages: 255699
nosy: alexandre.vassalotti, pitrou, rhettinger, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: More compact pickle of iterators etc
type: enhancement
versions: Python 3.6
Added file: http://bugs.python.org/file41207/iterators_pickle.diff

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com