[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Change by Shantanu : -- nosy: +hauntsaninja nosy_count: 6.0 -> 7.0 pull_requests: +19428 pull_request: https://github.com/python/cpython/pull/19806 ___ Python tracker ___

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

STINNER Victor added the comment: Thanks Lumír and Miro! I close the issue. -- resolution: -> fixed stage: patch review -> resolved status: open -> closed ___ Python tracker

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

STINNER Victor added the comment: New changeset e77d428856fbd339faee44ff47214eda5fb51d57 by Lumír 'Frenzy' Balhar in branch 'master': bpo-40495: compileall option to hardlink duplicate pyc files (GH-19901) https://github.com/python/cpython/commit/e77d428856fbd339faee44ff47214eda5fb51d57

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

STINNER Victor added the comment: > Currently, it's possible to implement this optimization using the Unix > command "hardlink". PR 19901 avoids the dependency on external "hardlink" command. In practice, PR 19901 only impacts newly written PYC files, whereas using manually the "hardlink"

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

STINNER Victor added the comment: Currently, it's possible to implement this optimization using the Unix command "hardlink". Example: hardlink -c -v /usr/lib64/python3.8/__pycache__/*.pyc On my Fedora 32, this command says: Directories: 1 Objects: 520 Regular

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Miro Hrončok added the comment: > Is it possible that the PYC file of optimization level 0 content is modified > if the PY file content changed, with would make PYC files or optimization > level 1 and 2 inconsistent? ... Note that there is a test exactly for this, in case the implementation

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

STINNER Victor added the comment: While reviewing PR 19901, I was confused by py_compile and compileall documentation which is outdated: it doesn't mention that optimize argument can be a list of integers. https://docs.python.org/dev/library/py_compile.html#py_compile.compile "optimize

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

STINNER Victor added the comment: Is it possible that the PYC file of optimization level 0 content is modified if the PY file content changed, with would make PYC files or optimization level 1 and 2 inconsistent? Christian Heimes: > Python's import system is fully compatible with this

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Change by Lumír Balhar : -- keywords: +patch pull_requests: +19214 stage: -> patch review pull_request: https://github.com/python/cpython/pull/19901 ___ Python tracker ___

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Lumír Balhar added the comment: I forgot to mention that I am working on PR which should be ready soon because the implementation is already done and tested in compileall2. -- ___ Python tracker

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Christian Heimes added the comment: Brett, FYI -- nosy: +brett.cannon ___ Python tracker ___ ___ Python-bugs-list mailing list

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Christian Heimes added the comment: Python's import system is fully compatible with this approach. importlib never directly writes to a .pyc file. Instead it always creates a new temporary file next to the .pyc file and then overrides the .pyc file with an atomic file system operation. See

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Change by Filipe Laíns : -- nosy: +FFY00 ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

Change by Miro Hrončok : -- nosy: +hroncok ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue40495] compileall: option to hardlink duplicate optimization levels bytecode cache files

New submission from Lumír Balhar : We would like to include a possibility of hardlink deduplication of identical pyc files to compileall module in Python 3.9. We've discussed the change [0] and tested it in Fedora RPM build system via implementation in the compileall2 module [1]. The