[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: New changeset 60bd1f88f21073965a444c8b39c4202d015da5d6 by Victor Stinner (Zackery Spytz) in branch 'master': bpo-36030: Fix a possible segfault in PyTuple_New() (GH-15670) https://github.com/python/cpython/commit/60bd1f88f21073965a444c8b39c4202d015da5d6

[issue36030] add internal API function to create tuple without items array initialization

Change by Zackery Spytz : -- pull_requests: +15333 pull_request: https://github.com/python/cpython/pull/15670 ___ Python tracker ___

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: New changeset 4fa10dde40356d7c71e5524233bafc221d9e2deb by Victor Stinner (Sergey Fedoseev) in branch 'master': bpo-36030: Improve performance of some tuple operations (GH-12052)

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: Well done Sergey Fedoseev: I merged your last PR for this issue ;-) I'm not sure if it's worth it to document these tuple micro-optimizations in What's New in Python 3.8 / 3.9. In the meanwhile, I close the issue. -- resolution: -> fixed stage:

[issue36030] add internal API function to create tuple without items array initialization

Change by Sergey Fedoseev : Added file: https://bugs.python.org/file48173/build_tuple_bench.py ___ Python tracker ___ ___ Python-bugs-list

[issue36030] add internal API function to create tuple without items array initialization

Change by Sergey Fedoseev : Added file: https://bugs.python.org/file48174/list_as_tuple_bench.py ___ Python tracker ___ ___

[issue36030] add internal API function to create tuple without items array initialization

Sergey Fedoseev added the comment: >> This optimization also can be used for BUILD_TUPLE opcode and in pickle >> module, if it's OK to add _PyTuple_StealFromArray() function :-) > I would like to see a micro-benchmark showing that it's faster.

[issue36030] add internal API function to create tuple without items array initialization

Sergey Fedoseev added the comment: > Can you please convert msg336142 into a perf script? > And then run again these benchmarks on PR 12052. ++-+--+ | Benchmark | ref | untracked|

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: > if LTO is enabled, the same stack bloat issues are possible Please test, I'm not interested to spend too much time on that topic. To be clear, _Py_NO_INLINE was a hack and a micro-optimization. It doesn't solve a real bug. Python has very weak promises on

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: > Victor/vstinner: Isn't PR 12032 reintroducing the issue fixed in #29234? No according to manual tests: https://github.com/python/cpython/pull/12032#issuecomment-467110233 Usage of the stack memory doesn't change with PR 12032. FYI I also tested manually

[issue36030] add internal API function to create tuple without items array initialization

Josh Rosenberg added the comment: Victor/vstinner: Isn't PR 12032 reintroducing the issue fixed in #29234? _PyStack_AsTuple was explicitly marked as _Py_NO_INLINE because inlining was creating excessive stack consumption in the callers (which were the bytecode interpreter loop), but the new

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: Can you please convert msg336142 into a perf script? See the doc: https://perf.readthedocs.io/en/latest/developer_guide.html And then run again these benchmarks on PR 12052. If you have a script, you can do: ./python-ref script.py -o ref.json

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: > This optimization also can be used for BUILD_TUPLE opcode and in pickle > module, if it's OK to add _PyTuple_StealFromArray() function :-) I would like to see a micro-benchmark showing that it's faster. -- ___

[issue36030] add internal API function to create tuple without items array initialization

Sergey Fedoseev added the comment: This optimization also can be used for BUILD_TUPLE opcode and in pickle module, if it's OK to add _PyTuple_StealFromArray() function :-) -- ___ Python tracker

[issue36030] add internal API function to create tuple without items array initialization

Sergey Fedoseev added the comment: I added WIP PR with discussed micro-optimization, here are benchmark results: $ python -m perf compare_to --min-speed 1 -G master.json tuple-untracked.json Slower (1): - sqlite_synth: 5.16 us +- 0.10 us -> 5.22 us +- 0.08 us: 1.01x slower (+1%) Faster (19):

[issue36030] add internal API function to create tuple without items array initialization

Change by Sergey Fedoseev : -- pull_requests: +12078 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: New changeset f1b9abe35f75393351b3d954a392122a3f8f6951 by Victor Stinner (Sergey Fedoseev) in branch 'master': bpo-36030: Remove _PyStack_AsTuple() and _PyStack_AsTupleSlice() (GH-12032)

[issue36030] add internal API function to create tuple without items array initialization

Change by Sergey Fedoseev : -- pull_requests: +12062 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: New changeset 234531b4462b20d668762bd78406fd2ebab129c9 by Victor Stinner (Sergey Fedoseev) in branch 'master': bpo-36030: Add _PyTuple_FromArray() function (GH-11954) https://github.com/python/cpython/commit/234531b4462b20d668762bd78406fd2ebab129c9

[issue36030] add internal API function to create tuple without items array initialization

Serhiy Storchaka added the comment: This change is virtually renaming _PyStack_AsTuple to _PyTuple_FromArray and using it in appropriate places for code simplification. I am +1 for such change. -- ___ Python tracker

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: I reviewed PR 11954: I asked to rework the PR to only add _PyTuple_FromArray() and the "unrelated" micro-optimization. So it would be easier to see code simplification and the micro-optimization. If the micro-optimization doesn't make the code more complex

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: > Once I had wrote a similar patch that adds PyTuple_FromArray, but never > published it because I did not found enough use cases for this function. > Although I considered using it only for removing some code duplication, but > Sergey shown that it can be

[issue36030] add internal API function to create tuple without items array initialization

Serhiy Storchaka added the comment: Once I had wrote a similar patch that adds PyTuple_FromArray, but never published it because I did not found enough use cases for this function. Although I considered using it only for removing some code duplication, but Sergey shown that it can be used

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: Please make PyTuple_FromArray private: rename to _PyTuple_FromArray. I would prefer to only add it to Include/internals/pycore_tupleobject.c. -- ___ Python tracker

[issue36030] add internal API function to create tuple without items array initialization

Sergey Fedoseev added the comment: Here's benchmarks for PyTuple_FromArray() PR: $ python -m perf timeit -s "l = [None]*0; tuple_ = tuple" "tuple_(l)" --duplicate=100 --compare-to=../cpython-master/venv/bin/python /home/sergey/tmp/cpython-master/venv/bin/python: . 50.5

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: I prefer PyTuple_FromArray() API idea. It's safer by design than you "list to tuple" function stealing references to list items if refcnt(list)==1. -- ___ Python tracker

[issue36030] add internal API function to create tuple without items array initialization

Change by Sergey Fedoseev : -- pull_requests: +11979 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: Raymond: > Also, the timing loop is unrealistic. We mostly care about small tuples. Sergey: can you please run benchmarks on small tuples? Example, 1, 5, 10, 20 and 100 items. Maybe not only tuple(list), but try some other operations? --

[issue36030] add internal API function to create tuple without items array initialization

Raymond Hettinger added the comment: Zeroing memory is usually not expensive relative to the cost of filling it in. Also, the timing loop is unrealistic. We mostly care about small tuples. For long term maintenance of the project, I recommend filling the code with these unsafe variants

[issue36030] add internal API function to create tuple without items array initialization

STINNER Victor added the comment: Sergey Fedoseev wrote similar change for list: bpo-34151. -- ___ Python tracker ___ ___

[issue36030] add internal API function to create tuple without items array initialization

Change by Stéphane Wirtel : -- nosy: +vstinner ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue36030] add internal API function to create tuple without items array initialization

Change by Sergey Fedoseev : -- keywords: +patch pull_requests: +11952 stage: -> patch review ___ Python tracker ___ ___

[issue36030] add internal API function to create tuple without items array initialization

New submission from Sergey Fedoseev : PyTuple_New() fills items array with NULLs to make usage of Py_DECREF() safe even when array is not fully filled with real items. There are multiple cases when this initialization step can be avoided to improve performance. For example it gives such