PYTHON_EXT_SUFFIX value for shared libraries, pyc files and others
Hi, I have a problem with PYTHON_EXT_SUFFIX when I try to build ports (including python) with the option WITH_DEBUG. I want to build subset of ports with WITH_DEBUG by poudriere, but several python package builds have failed on phase "package" due to missing artifacts. The root cause of failure is that ports uses same macro PYTHON_EXT_SUFFIX for shared libraries and pyc files, but actual file names have different suffixes: - pyc files have the suffix ".cpython-39" as expected. - so files have the suffix ".cpython-39d". The "d" is ${PYTHON_ABIVER} and actual ABI flags of Python build. According to Python specifications, all pyc files must have a suffix without ABI flags. Shared libraries have suffixes with ABI flags, but I didn't find any spec about it. For instance, port gobject-introspection contains pyc files like: /usr/local/lib/gobject-introspection/giscanner/__pycache__/testcodegen.cpython-39.pyc and shared library like: /usr/local/lib/gobject-introspection/giscanner/_giscanner.cpython-39d.so The first idea came to me is to add an extra suffix PYTHON_EXTSO_SUFFIX with value ".cpython-${PYTHON_SUFFIX}${PYTHON_ABIVER}". It's easy to add it and replace all occurrences in pkg-plist and Makefile, but it may be hard to maintain it in future. Any thoughts? Thanks, Michael.
[Bug 255722] [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722 --- Comment #34 from commit-h...@freebsd.org --- A commit in branch main references this bug: URL: https://cgit.FreeBSD.org/ports/commit/?id=cc8a1878e0242055ab6a8c20d33654451f78720e commit cc8a1878e0242055ab6a8c20d33654451f78720e Author: Charlie Li AuthorDate: 2022-11-16 23:31:40 + Commit: Charlie Li CommitDate: 2023-01-11 05:22:16 + python.mk: introduce USE_PYTHON=pep517 for PEP-517 support USE_PYTHON=pep517 takes no arguments. Operation is similar to USE_PYTHON=distutils, although the build backend specified in pyproject.toml is to be specified in BUILD_DEPENDS explicitly. A usage guide and implementation primer is available at: https://wiki.freebsd.org/Python/PEP-517 With hat: python Approved by: fluffy (mentor) Co-authored by: yuri PR: 255722 Differential Revision: https://reviews.freebsd.org/D36290 CHANGES | 13 ++ Mk/Uses/python.mk | 77 --- 2 files changed, 87 insertions(+), 3 deletions(-) -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 255722] [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722 Charlie Li changed: What|Removed |Added Status|Open|Closed Resolution|--- |FIXED --- Comment #35 from Charlie Li --- This is now committed, with proper working USE_PYTHON=autoplist support, so that migrations from USE_PYTHON=distutils are as seamless as possible. Do mind that you will have to add the build backend specified in pyproject.toml as an explicit BUILD_DEPENDS. Feel free to report any problems as they arise here or elsewhere. -- You are receiving this mail because: You are on the CC list for the bug. You are the assignee for the bug.
[Bug 268018] textproc/py-sphinx: Updating to 5.3.0, ensuring themes are installed, using flit and pyproject-build
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268018 Bug 268018 depends on bug 255722, which changed state. Bug 255722 Summary: [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722 What|Removed |Added Status|Open|Closed Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 268283] databases/py-sqlite3: switch away from distutils
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268283 Bug 268283 depends on bug 255722, which changed state. Bug 255722 Summary: [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722 What|Removed |Added Status|Open|Closed Resolution|--- |FIXED -- You are receiving this mail because: You are the assignee for the bug.
[Bug 255722] [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722 Charlie Li changed: What|Removed |Added Keywords|needs-patch, needs-qa | -- You are receiving this mail because: You are the assignee for the bug. You are on the CC list for the bug.
Re: PYTHON_EXT_SUFFIX value for shared libraries, pyc files and others
Michael Zhilin wrote: Hi, I have a problem with PYTHON_EXT_SUFFIX when I try to build ports (including python) with the option WITH_DEBUG. I want to build subset of ports with WITH_DEBUG by poudriere, but several python package builds have failed on phase "package" due to missing artifacts. The root cause of failure is that ports uses same macro PYTHON_EXT_SUFFIX for shared libraries and pyc files, but actual file names have different suffixes: - pyc files have the suffix ".cpython-39" as expected. - so files have the suffix ".cpython-39d". The "d" is ${PYTHON_ABIVER} and actual ABI flags of Python build. According to Python specifications, all pyc files must have a suffix without ABI flags. Shared libraries have suffixes with ABI flags, but I didn't find any spec about it. PEP-3147 for byte compilation files (.pyc) and PEP-3149 for extension module files (.so). The first idea came to me is to add an extra suffix PYTHON_EXTSO_SUFFIX with value ".cpython-${PYTHON_SUFFIX}${PYTHON_ABIVER}". It's easy to add it and replace all occurrences in pkg-plist and Makefile, but it may be hard to maintain it in future. Looks like some unfinished business from PR 252057. In theory, should be able to append ${PYTHON_ABIVER} directly to ${PYTHON_EXT_SUFFIX}, since ${PYTHON_CMD}-config --abiflags returns the correct flags even if empty. Because the ABI changes when you build Python itself under --with-pydebug (WITH_DEBUG or DEBUG option), you cannot build only a subset of Python ports under WITH_DEBUG; it's an all-or-nothing situation. Byte compilation files are not compiled or installed when the port is built under PEP-517. Compiling and installing byte compilation files with distutils/setuptools at build/stage-time will be deprecated. A new install-time hook is under development to replace this functionality, which has the added benefit of less pkg-plist churn and framework maintenance. -- Charlie Li …nope, still don't have an exit line. OpenPGP_signature Description: OpenPGP digital signature
Re: PYTHON_EXT_SUFFIX value for shared libraries, pyc files and others
On Wed, Jan 11, 2023 at 9:45 AM Charlie Li wrote: > Michael Zhilin wrote: > > Hi, > > > > I have a problem with PYTHON_EXT_SUFFIX when I try to build ports > > (including python) with the option WITH_DEBUG. > > > > I want to build subset of ports with WITH_DEBUG by poudriere, but > > several python package builds have failed on phase "package" due to > > missing artifacts. The root cause of failure is that ports uses same > > macro PYTHON_EXT_SUFFIX for shared libraries and pyc files, but actual > > file names have different suffixes: > > - pyc files have the suffix ".cpython-39" as expected. > > - so files have the suffix ".cpython-39d". > > > > The "d" is ${PYTHON_ABIVER} and actual ABI flags of Python build. > > According to Python specifications, all pyc files must have a suffix > > without ABI flags. Shared libraries have suffixes with ABI flags, but I > > didn't find any spec about it. > > > PEP-3147 for byte compilation files (.pyc) and PEP-3149 for extension > module files (.so). > > The first idea came to me is to add an extra suffix PYTHON_EXTSO_SUFFIX > > with value ".cpython-${PYTHON_SUFFIX}${PYTHON_ABIVER}". It's easy to add > > it and replace all occurrences in pkg-plist and Makefile, but it may be > > hard to maintain it in future. > > > Looks like some unfinished business from PR 252057. In theory, should be > able to append ${PYTHON_ABIVER} directly to ${PYTHON_EXT_SUFFIX}, since > ${PYTHON_CMD}-config --abiflags returns the correct flags even if empty. > Because the ABI changes when you build Python itself under > --with-pydebug (WITH_DEBUG or DEBUG option), you cannot build only a > subset of Python ports under WITH_DEBUG; it's an all-or-nothing situation. > > Byte compilation files are not compiled or installed when the port is > built under PEP-517. Compiling and installing byte compilation files > with distutils/setuptools at build/stage-time will be deprecated. A new > install-time hook is under development to replace this functionality, > which has the added benefit of less pkg-plist churn and framework > maintenance. > > -- > Charlie Li > …nope, still don't have an exit line. > > Thank you! It looks very good to me. I forgot to mention CMake files that contains suffix with ABI flags as well, but approach mentioned by you works for them as well. Looking for deprecation of byte compilation. Please let me know if I can help to move it forward in any way. BR, Michael.