[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2021-04-26 Thread Gregory P. Smith 


Gregory P. Smith  added the comment:

The PyList_Check -> PyList_CheckExact change led to bugs where subclasses of 
list that override extend can no longer be unpickled.

https://bugs.python.org/issue43946

--
nosy: +gregory.p.smith

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-10-23 Thread Serhiy Storchaka 

Change by Serhiy Storchaka :


--
pull_requests:  -836

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-03-31 Thread Donald Stufft 

Changes by Donald Stufft :


--
pull_requests: +836

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Thanks Victor.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread STINNER Victor 

STINNER Victor added the comment:

> Even if I don't see any refleak, you might just run "./python -m test -R 3:3 
> test_pickle" just to be sure :-)

Change 94d630a02a81 introduced a crash in test_pickle:

Fatal Python error: ..\Modules\_pickle.c:5847 object at 02B7F7BED2F8 has 
negative ref count -1

Seen on buildbots, but can always be reproduce on Linux as well.

It seems like you was biten by the surprising _Pickle_FastCall() API which 
decreases the reference counter of its second parameter. Don't ask me why it 
does that :-) (I don't know.)

I fixed the bug in the change 328147c0edc3 to repair buildbots.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Roundup Robot 

Roundup Robot added the comment:


New changeset f89fdc29937139b55dd68587759cadb8468d0190 by Serhiy Storchaka in 
branch 'master':
Issue #29368: The extend() method is now called instead of the append()
https://github.com/python/cpython/commit/f89fdc29937139b55dd68587759cadb8468d0190

New changeset 4d7e63d9773a766358294593fc00b1f8c8f41b5d by Victor Stinner in 
branch 'master':
Issue #29368: Fix _Pickle_FastCall() usage in do_append()
https://github.com/python/cpython/commit/4d7e63d9773a766358294593fc00b1f8c8f41b5d


--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Roundup Robot 

Changes by Roundup Robot :


--
resolution:  -> fixed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Roundup Robot 

Changes by Roundup Robot :


--
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Roundup Robot 

Changes by Roundup Robot :


--
stage: patch review -> resolved

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Roundup Robot 

Roundup Robot added the comment:

New changeset 328147c0edc3 by Victor Stinner in branch 'default':
Issue #29368: Fix _Pickle_FastCall() usage in do_append()
https://hg.python.org/cpython/rev/328147c0edc3

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-02 Thread Roundup Robot 

Roundup Robot added the comment:

New changeset 94d630a02a81 by Serhiy Storchaka in branch 'default':
Issue #29368: The extend() method is now called instead of the append()
https://hg.python.org/cpython/rev/94d630a02a81

--
nosy: +python-dev

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-01 Thread Raymond Hettinger 

Raymond Hettinger added the comment:

> Could anyone please make a review of my explanation comment? 
> I have doubts about a wording.

The wording is correct and clear.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-01 Thread STINNER Victor 

STINNER Victor added the comment:

pickle-appends-extend-3.patch LGTM.

Even if I don't see any refleak, you might just run "./python -m test -R 3:3 
test_pickle" just to be sure :-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-01 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Thank you Victor. Your wording looks simpler to me.

--
Added file: http://bugs.python.org/file46484/pickle-appends-extend-3.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-01 Thread STINNER Victor 

STINNER Victor added the comment:

> Could anyone please make a review of my explanation comment? I have doubts 
> about a wording.

I'm not fluent in english, so I'm not the best for this task. But I reviewed 
your patch ;-)

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-02-01 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Could anyone please make a review of my explanation comment? I have doubts 
about a wording.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-01-26 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

Updated patch addresses Antoine's comment. It adds the comment explaining a 
fallback.

--
Added file: http://bugs.python.org/file46421/pickle-appends-extend-2.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-01-26 Thread Raymond Hettinger 

Raymond Hettinger added the comment:

This code looks correct and reasonable.  The tests all pass for me.  I think 
you can go forward and apply the patch.

--
assignee:  -> serhiy.storchaka
nosy: +rhettinger

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-01-25 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

See issue17720 for a feedback from Alexandre.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects

2017-01-25 Thread STINNER Victor 

STINNER Victor added the comment:

> Every object that return listitems from the __reduce__() method must support 
> the extend() method according to the specification.

Hum ok. I don't know well the pickle module, but according to your quote, yeah, 
the patch is valid.

pickle-appends-extend.patch LGTM, except minor comments.

It would be nice to get a feedback from Alexandre, but I'm not sure that he is 
still around :-/

What about Antoine Pitrou, are you around? :-)

--
nosy: +pitrou

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects: use extend() instead of append()

2017-01-25 Thread STINNER Victor 

Changes by STINNER Victor :


--
title: Optimize unpickling list-like objects -> Optimize unpickling list-like 
objects: use extend() instead of append()

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects

2017-01-25 Thread Serhiy Storchaka 

Serhiy Storchaka added the comment:

collections.Sequence has no relation to the pickle protocol.

Every object that return listitems from the __reduce__() method must support 
the extend() method according to the specification.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects

2017-01-25 Thread STINNER Victor 

STINNER Victor added the comment:

What is the cost of adding an extra isinstance(d, collections.Sequence) check? 
It would be closer the current design ("white list" of types), safer and avoid 
bad surprises.

But Python has a long tradition of duck typing, so maybe isinstance() can be 
seen as overkill :-)

--
nosy: +haypo

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue29368] Optimize unpickling list-like objects

2017-01-25 Thread Serhiy Storchaka 

New submission from Serhiy Storchaka:

According to PEP 307 the extend method can be used for appending list items to 
the object.

listitemsOptional, and new in this PEP.
 If this is not None, it should be an iterator (not a
 sequence!) yielding successive list items.  These list
 items will be pickled, and appended to the object using
 either obj.append(item) or obj.extend(list_of_items).
 This is primarily used for list subclasses, but may
 be used by other classes as long as they have append()
 and extend() methods with the appropriate signature.
 (Whether append() or extend() is used depends on which
 pickle protocol version is used as well as the number
 of items to append, so both must be supported.)

Proposed patch makes the extend method be used in the APPENDS opcode. To avoid 
breaking existing code the use of the extend method is optional.

Microbenchmark:

$ ./python -m timeit -s "import pickle, collections; p = 
pickle.dumps(collections.deque([None]*1), 4)" -- "pickle.loads(p)"
Unpatched:  100 loops, best of 5: 2.02 msec per loop
Patched:500 loops, best of 5: 833 usec per loop

--
components: Library (Lib)
files: pickle-appends-extend.patch
keywords: patch
messages: 286237
nosy: alexandre.vassalotti, serhiy.storchaka
priority: normal
severity: normal
stage: patch review
status: open
title: Optimize unpickling list-like objects
type: performance
versions: Python 3.7
Added file: http://bugs.python.org/file46412/pickle-appends-extend.patch

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com