[issue29336] merge tuples in module

Change by INADA Naoki : -- resolution: postponed -> duplicate stage: -> resolved status: pending -> closed superseder: -> Same constants in tuples are not merged while compile() ___ Python tracker

[issue29336] merge tuples in module

INADA Naoki added the comment: As I said on ML, now I think embedding some co_* tuples into code object is better way to reduce number of tuples and memory usage. So I close this issue for now. Thanks for review and comments. -- resolution: -> postponed status: open -> pending

[issue29336] merge tuples in module

STINNER Victor added the comment: I vote +0 on merge-constants.patch: LGTM, may be useful, but not impressed :-) Hopefully, .py => .pyc compilation "should" occur only once. Except of the main script, but more and more applications use a tiny entry point of 3 lines. I don't expect any

[issue29336] merge tuples in module

Serhiy Storchaka added the comment: > merge-constants.patch looks simple enought, but I'm not really impressed by > such result. Is 2% worth it? I'm not impressed too. And merging constants can take time. > Since code objects loaded by import are likely for stay for the whole > lifetime of a

[issue29336] merge tuples in module

INADA Naoki added the comment: > More generally, would it be possible to share co_consts (None,) tuples > between code objects of two different modules? Or is it already the case with > merge-constants.patch? No. I didn't do it because I don't know it's worth enough. > merge-constants.patch

[issue29336] merge tuples in module

STINNER Victor added the comment: I don't know what is the most efficient technic to reduce the memory usage of module, but at least I can say imports are the top #1 memory consumer. Look at tracemalloc examples: https://docs.python.org/dev/library/tracemalloc.html#display-the-top-10 "We can

[issue29336] merge tuples in module

STINNER Victor added the comment: > Thanks. Your patch reduced memory consumption by 2%, merge-constants.patch looks simple enought, but I'm not really impressed by such result. Is 2% worth it? Since code objects loaded by import are likely for stay for the whole lifetime of a process, I

[issue29336] merge tuples in module

STINNER Victor added the comment: > But this can be done better with the AST optimizer (issue1346238, issue11549). There are also the PEP 511 (Code transformers) which was created to implement *external* AST optimizers, and the FAT Python which implements an AST optimizer. Hum, after "-o

[issue29336] merge tuples in module

Serhiy Storchaka added the comment: merge-constants.patch is rather a proof of concept. I think it may be more efficient after removing unneeded folded constants (issue28813). But this can be done better with the AST optimizer (issue1346238, issue11549). It would be worth also to merge nested

[issue29336] merge tuples in module

INADA Naoki added the comment: merge-constants.patch LGTM -- ___ Python tracker ___ ___ Python-bugs-list

[issue29336] merge tuples in module

INADA Naoki added the comment: Thanks. Your patch reduced memory consumption by 2%, and number of tuples by 15%. $ cat invtuple.py import app import sys import traceback import tracemalloc print(tracemalloc.get_traced_memory()) allobj = sys.getobjects(0, tuple) print(len(allobj)) $

[issue29336] merge tuples in module

Changes by Serhiy Storchaka : Added file: http://bugs.python.org/file46382/merge-constants.patch ___ Python tracker ___

[issue29336] merge tuples in module

Changes by Serhiy Storchaka : Removed file: http://bugs.python.org/file46371/merge-constants.patch ___ Python tracker ___

[issue29336] merge tuples in module

Serhiy Storchaka added the comment: Following patch uses _PyCode_ConstantKey() as a key. This allows supporting integers, floats, bytes, etc. But I think we can go further and merge constants recursively. See also issue28813. -- nosy: +rhettinger Added file:

[issue29336] merge tuples in module

Changes by INADA Naoki : Added file: http://bugs.python.org/file46368/mergetuples2.patch ___ Python tracker ___

[issue29336] merge tuples in module

INADA Naoki added the comment: Oh, thanks. I hadn't checked the warning. Since bytes are not important in this time, I'll remove bytes support. -- ___ Python tracker

[issue29336] merge tuples in module

Serhiy Storchaka added the comment: $ ./python -Wa -b -c "lambda: 'a'; lambda: b'a'" sys:1: BytesWarning: Comparison between bytes and string sys:1: BytesWarning: Comparison between bytes and string sys:1: BytesWarning: Comparison between bytes and string sys:1: BytesWarning: Comparison between

[issue29336] merge tuples in module

INADA Naoki added the comment: I tried this patch with attached script. ``` $ venv/bin/pip install django flask sqlalchemy $ PYTHONTRACEMALLOC=5 venv/bin/python3 tuplemem.py > tuples.txt $ sort tuples.txt | uniq -c | sort -nr > tuplecount ``` ## default memory: (32254693, 32292635) tuples:

[issue29336] merge tuples in module

New submission from INADA Naoki: Tuples consists of None, True, False, str and bytes can be merged safely. To avoid memory leak, this patch shares such tuples in compile unit (module, in most case.) -- files: merge-tuples.patch keywords: patch messages: 285933 nosy: haypo, inada.naoki