PYTHON_EXT_SUFFIX value for shared libraries, pyc files and others

2023-01-10 Thread Michael Zhilin
Hi,

I have a problem with PYTHON_EXT_SUFFIX when I try to build ports
(including python) with the option WITH_DEBUG.

I want to build subset of ports with WITH_DEBUG by poudriere, but several
python package builds have failed on phase "package" due to missing
artifacts. The root cause of failure is that ports uses same macro
PYTHON_EXT_SUFFIX for shared libraries and pyc files, but actual file names
have different suffixes:
 - pyc files have the suffix ".cpython-39" as expected.
 - so files have the suffix ".cpython-39d".

The "d" is ${PYTHON_ABIVER} and actual ABI flags of Python build. According
to Python specifications, all pyc files must have a suffix without ABI
flags. Shared libraries have suffixes with ABI flags, but I didn't find any
spec about it.

For instance, port gobject-introspection contains pyc files like:

/usr/local/lib/gobject-introspection/giscanner/__pycache__/testcodegen.cpython-39.pyc
and shared library like:
   /usr/local/lib/gobject-introspection/giscanner/_giscanner.cpython-39d.so

The first idea came to me is to add an extra suffix PYTHON_EXTSO_SUFFIX
with value ".cpython-${PYTHON_SUFFIX}${PYTHON_ABIVER}". It's easy to add it
and replace all occurrences in pkg-plist and Makefile, but it may be hard
to maintain it in future.

Any thoughts?

Thanks,
Michael.


[Bug 255722] [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support

2023-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722

--- Comment #34 from commit-h...@freebsd.org ---
A commit in branch main references this bug:

URL:
https://cgit.FreeBSD.org/ports/commit/?id=cc8a1878e0242055ab6a8c20d33654451f78720e

commit cc8a1878e0242055ab6a8c20d33654451f78720e
Author: Charlie Li 
AuthorDate: 2022-11-16 23:31:40 +
Commit: Charlie Li 
CommitDate: 2023-01-11 05:22:16 +

python.mk: introduce USE_PYTHON=pep517 for PEP-517 support

USE_PYTHON=pep517 takes no arguments. Operation is similar to
USE_PYTHON=distutils, although the build backend specified in
pyproject.toml is to be specified in BUILD_DEPENDS explicitly. A
usage guide and implementation primer is available at:
https://wiki.freebsd.org/Python/PEP-517

With hat: python
Approved by: fluffy (mentor)
Co-authored by: yuri
PR: 255722
Differential Revision: https://reviews.freebsd.org/D36290

 CHANGES   | 13 ++
 Mk/Uses/python.mk | 77 ---
 2 files changed, 87 insertions(+), 3 deletions(-)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


[Bug 255722] [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support

2023-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722

Charlie Li  changed:

   What|Removed |Added

 Status|Open|Closed
 Resolution|--- |FIXED

--- Comment #35 from Charlie Li  ---
This is now committed, with proper working USE_PYTHON=autoplist support, so
that migrations from USE_PYTHON=distutils are as seamless as possible. Do mind
that you will have to add the build backend specified in pyproject.toml as an
explicit BUILD_DEPENDS.

Feel free to report any problems as they arise here or elsewhere.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.


[Bug 268018] textproc/py-sphinx: Updating to 5.3.0, ensuring themes are installed, using flit and pyproject-build

2023-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268018
Bug 268018 depends on bug 255722, which changed state.

Bug 255722 Summary: [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722

   What|Removed |Added

 Status|Open|Closed
 Resolution|--- |FIXED

-- 
You are receiving this mail because:
You are the assignee for the bug.


[Bug 268283] databases/py-sqlite3: switch away from distutils

2023-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=268283
Bug 268283 depends on bug 255722, which changed state.

Bug 255722 Summary: [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722

   What|Removed |Added

 Status|Open|Closed
 Resolution|--- |FIXED

-- 
You are receiving this mail because:
You are the assignee for the bug.


[Bug 255722] [meta] Mk/Uses/python.mk: PEP-517 (pyproject.toml) Support

2023-01-10 Thread bugzilla-noreply
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=255722

Charlie Li  changed:

   What|Removed |Added

   Keywords|needs-patch, needs-qa   |

-- 
You are receiving this mail because:
You are the assignee for the bug.
You are on the CC list for the bug.


Re: PYTHON_EXT_SUFFIX value for shared libraries, pyc files and others

2023-01-10 Thread Charlie Li

Michael Zhilin wrote:

Hi,

I have a problem with PYTHON_EXT_SUFFIX when I try to build ports 
(including python) with the option WITH_DEBUG.


I want to build subset of ports with WITH_DEBUG by poudriere, but 
several python package builds have failed on phase "package" due to 
missing artifacts. The root cause of failure is that ports uses same 
macro PYTHON_EXT_SUFFIX for shared libraries and pyc files, but actual 
file names have different suffixes:

  - pyc files have the suffix ".cpython-39" as expected.
  - so files have the suffix ".cpython-39d".

The "d" is ${PYTHON_ABIVER} and actual ABI flags of Python build. 
According to Python specifications, all pyc files must have a suffix 
without ABI flags. Shared libraries have suffixes with ABI flags, but I 
didn't find any spec about it.


PEP-3147 for byte compilation files (.pyc) and PEP-3149 for extension 
module files (.so).
The first idea came to me is to add an extra suffix PYTHON_EXTSO_SUFFIX 
with value ".cpython-${PYTHON_SUFFIX}${PYTHON_ABIVER}". It's easy to add 
it and replace all occurrences in pkg-plist and Makefile, but it may be 
hard to maintain it in future.


Looks like some unfinished business from PR 252057. In theory, should be 
able to append ${PYTHON_ABIVER} directly to ${PYTHON_EXT_SUFFIX}, since 
${PYTHON_CMD}-config --abiflags returns the correct flags even if empty. 
Because the ABI changes when you build Python itself under 
--with-pydebug (WITH_DEBUG or DEBUG option), you cannot build only a 
subset of Python ports under WITH_DEBUG; it's an all-or-nothing situation.


Byte compilation files are not compiled or installed when the port is 
built under PEP-517. Compiling and installing byte compilation files 
with distutils/setuptools at build/stage-time will be deprecated. A new 
install-time hook is under development to replace this functionality, 
which has the added benefit of less pkg-plist churn and framework 
maintenance.


--
Charlie Li
…nope, still don't have an exit line.



OpenPGP_signature
Description: OpenPGP digital signature


Re: PYTHON_EXT_SUFFIX value for shared libraries, pyc files and others

2023-01-10 Thread Michael Zhilin
On Wed, Jan 11, 2023 at 9:45 AM Charlie Li  wrote:

> Michael Zhilin wrote:
> > Hi,
> >
> > I have a problem with PYTHON_EXT_SUFFIX when I try to build ports
> > (including python) with the option WITH_DEBUG.
> >
> > I want to build subset of ports with WITH_DEBUG by poudriere, but
> > several python package builds have failed on phase "package" due to
> > missing artifacts. The root cause of failure is that ports uses same
> > macro PYTHON_EXT_SUFFIX for shared libraries and pyc files, but actual
> > file names have different suffixes:
> >   - pyc files have the suffix ".cpython-39" as expected.
> >   - so files have the suffix ".cpython-39d".
> >
> > The "d" is ${PYTHON_ABIVER} and actual ABI flags of Python build.
> > According to Python specifications, all pyc files must have a suffix
> > without ABI flags. Shared libraries have suffixes with ABI flags, but I
> > didn't find any spec about it.
> >
> PEP-3147 for byte compilation files (.pyc) and PEP-3149 for extension
> module files (.so).
> > The first idea came to me is to add an extra suffix PYTHON_EXTSO_SUFFIX
> > with value ".cpython-${PYTHON_SUFFIX}${PYTHON_ABIVER}". It's easy to add
> > it and replace all occurrences in pkg-plist and Makefile, but it may be
> > hard to maintain it in future.
> >
> Looks like some unfinished business from PR 252057. In theory, should be
> able to append ${PYTHON_ABIVER} directly to ${PYTHON_EXT_SUFFIX}, since
> ${PYTHON_CMD}-config --abiflags returns the correct flags even if empty.
> Because the ABI changes when you build Python itself under
> --with-pydebug (WITH_DEBUG or DEBUG option), you cannot build only a
> subset of Python ports under WITH_DEBUG; it's an all-or-nothing situation.
>
> Byte compilation files are not compiled or installed when the port is
> built under PEP-517. Compiling and installing byte compilation files
> with distutils/setuptools at build/stage-time will be deprecated. A new
> install-time hook is under development to replace this functionality,
> which has the added benefit of less pkg-plist churn and framework
> maintenance.
>
> --
> Charlie Li
> …nope, still don't have an exit line.
>
>
Thank you!

It looks very good to me.
I forgot to mention CMake files that contains suffix with ABI flags as
well, but approach mentioned by you works for them as well.
Looking for deprecation of byte compilation. Please let me know if I can
help to move it forward in any way.

BR, Michael.