Re: [Distutils] cffi & Py_LIMITED_API
On 21 August 2016 at 10:32, Daniel Holth wrote: > Coded up in https://bitbucket.org/pypa/wheel/pull-requests/69 and > https://github.com/pypa/pip/pull/3922 and supported in setuptools 26. > > It's a multi-step process, unfortunately, but if you do it correctly then > you should need a lot fewer wheels: > > 0. Use only the functions from the limited api, for example, by using cffi > to generate your C extensions. #define Py_LIMITED_API to the desired value. > Python 3.3 is a good minimum, adding some useful functions forgotten in > Python 3.2's limited api. > 1. In setup.py, specify Extension(..., py_limited_api=True) to use .abi3.so > filenames on your C extensions > 2. In setup.cfg, specify [bdist_wheel] py_limited_api=cp32 # or greater, > tagging the wheel for compatibility with that version of CPython, or later. > This argument is ignored outside of CPython 3. > 3. A new enough version of pip recognizes the wheel tag > > Your extension uses the limited API, setuptools' Extension() changes the > filename so that more than just the exact version of Python can find it, > wheel adds a cp33-abi3-manylinux1_x86 tag, pip finds it, and you no longer > have to compile new wheels just because a new version of CPython came out. Very nice! Would you have the time to draft some initial additions to https://packaging.python.org/extensions/ to help us point folks towards how to do this? (And why they'd want to - i.e. not needing to rebuild their extension modules for each new release of CPython) Cheers, Nick. P.S. No worries if not - if nobody else beats me to it, I should be able to take a look at updating that page sometime after 3.6b1 next month. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
Coded up in https://bitbucket.org/pypa/wheel/pull-requests/69 and
https://github.com/pypa/pip/pull/3922 and supported in setuptools 26.
It's a multi-step process, unfortunately, but if you do it correctly then
you should need a lot fewer wheels:
0. Use only the functions from the limited api, for example, by using cffi
to generate your C extensions. #define Py_LIMITED_API to the desired value.
Python 3.3 is a good minimum, adding some useful functions forgotten in
Python 3.2's limited api.
1. In setup.py, specify Extension(..., py_limited_api=True) to use .abi3.so
filenames on your C extensions
2. In setup.cfg, specify [bdist_wheel] py_limited_api=cp32 # or greater,
tagging the wheel for compatibility with that version of CPython, or later.
This argument is ignored outside of CPython 3.
3. A new enough version of pip recognizes the wheel tag
Your extension uses the limited API, setuptools' Extension() changes the
filename so that more than just the exact version of Python can find it,
wheel adds a cp33-abi3-manylinux1_x86 tag, pip finds it, and you no longer
have to compile new wheels just because a new version of CPython came out.
On Tue, Aug 2, 2016 at 4:02 AM Nick Coghlan wrote:
> On 2 August 2016 at 02:24, Daniel Holth wrote:
> > On Mon, Aug 1, 2016 at 12:01 PM Steve Dower
> wrote:
> >> The DLL tag on Windows will have to just be ".pyd" if you want to
> >> support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
> >> ".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
> >> ".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
> >> make it.
> >>
> >> The wheel tag is more important than the DLL tag. (Possibly you're not
> >> even suggesting changing the DLL name at all and just passing the flag
> >> through for the build option? Hard to tell from the discussion.)
> >
> > I'm sure we are interpolating from Linux, where you'd be looking at
> > extensions .cpython-35m-x86_64-linux-gnu.so versus .abi3.so. But any DLL
> > extension that the target CPythons will agree to load, plus an
> appropriate
> > wheel tag, should be sufficient for our use case.
>
> Right, on *nix the SO names already include the SOABI details by
> default, so we really do want to change those to something less
> explicit, but I'm not sure it's even possible to using a completely
> unqualified SO name even if we wanted to.
>
> On Windows, if we have to rely on dropping any ABI indicated from the
> DLL entirely, we can leave with - pip should keep things from
> colliding too badly.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
>
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
On 2 August 2016 at 02:24, Daniel Holth wrote:
> On Mon, Aug 1, 2016 at 12:01 PM Steve Dower wrote:
>> The DLL tag on Windows will have to just be ".pyd" if you want to
>> support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
>> ".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
>> ".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
>> make it.
>>
>> The wheel tag is more important than the DLL tag. (Possibly you're not
>> even suggesting changing the DLL name at all and just passing the flag
>> through for the build option? Hard to tell from the discussion.)
>
> I'm sure we are interpolating from Linux, where you'd be looking at
> extensions .cpython-35m-x86_64-linux-gnu.so versus .abi3.so. But any DLL
> extension that the target CPythons will agree to load, plus an appropriate
> wheel tag, should be sufficient for our use case.
Right, on *nix the SO names already include the SOABI details by
default, so we really do want to change those to something less
explicit, but I'm not sure it's even possible to using a completely
unqualified SO name even if we wanted to.
On Windows, if we have to rely on dropping any ABI indicated from the
DLL entirely, we can leave with - pip should keep things from
colliding too badly.
Cheers,
Nick.
--
Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
On Mon, Aug 1, 2016 at 12:01 PM Steve Dower wrote:
> On 01Aug2016 0702, Nick Coghlan wrote:
> > On 1 August 2016 at 23:36, Daniel Holth wrote:
> >> build_ext command determines
> >> the DLL extension. It could be patched or modified to read an "I'm ABI3"
> >> flag on the Extension() object.
> >>
> >> We could pass an ABI3 flag to bdist_wheel in the same way we ask for
> >> universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
> >> extensions, and ignored on py2.
> >
> > The general idea sounds good to me, but as a slight bikeshed on the
> > flag name, perhaps "cpabi3"?
> >
> > That's a mash-up of the 'cp' interpreter code for CPython, with the
> > 'abi3' stable ABI tag.
> >
> > Longer term, we may want to allow people to version that (as new APIs
> > may sometimes be added to the stable ABI, which you gain access to at
> > the C level by setting Py_LIMITED_API to the corresponding CPython hex
> > version rather than just defining it [1]), but as a starting point
> > enabling access to the initial 3.2 stable ABI used by cffi should be
> > sufficient.
>
> The DLL tag on Windows will have to just be ".pyd" if you want to
> support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
> ".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
> ".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
> make it.
>
> The wheel tag is more important than the DLL tag. (Possibly you're not
> even suggesting changing the DLL name at all and just passing the flag
> through for the build option? Hard to tell from the discussion.)
>
I'm sure we are interpolating from Linux, where you'd be looking at
extensions .cpython-35m-x86_64-linux-gnu.so versus .abi3.so. But any DLL
extension that the target CPythons will agree to load, plus an appropriate
wheel tag, should be sufficient for our use case.
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
On 01Aug2016 0702, Nick Coghlan wrote:
On 1 August 2016 at 23:36, Daniel Holth wrote:
build_ext command determines
the DLL extension. It could be patched or modified to read an "I'm ABI3"
flag on the Extension() object.
We could pass an ABI3 flag to bdist_wheel in the same way we ask for
universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3
extensions, and ignored on py2.
The general idea sounds good to me, but as a slight bikeshed on the
flag name, perhaps "cpabi3"?
That's a mash-up of the 'cp' interpreter code for CPython, with the
'abi3' stable ABI tag.
Longer term, we may want to allow people to version that (as new APIs
may sometimes be added to the stable ABI, which you gain access to at
the C level by setting Py_LIMITED_API to the corresponding CPython hex
version rather than just defining it [1]), but as a starting point
enabling access to the initial 3.2 stable ABI used by cffi should be
sufficient.
The DLL tag on Windows will have to just be ".pyd" if you want to
support back prior to 3.5. In 3.5 you can use ".cp35-win32.pyd" or
".cp35-win_amd64.pyd" and the importer will prefer that DLL over a plain
".pyd", but my proposal to also support ".cp3-${PLAT}.pyd" here didn't
make it.
The wheel tag is more important than the DLL tag. (Possibly you're not
even suggesting changing the DLL name at all and just passing the flag
through for the build option? Hard to tell from the discussion.)
Cheers,
Steve
___
Distutils-SIG maillist - Distutils-SIG@python.org
https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
On 1 August 2016 at 23:36, Daniel Holth wrote: > On Mon, Aug 1, 2016 at 1:42 AM Nick Coghlan wrote: >> Status quo, on publication side: >> >> - require minimum cffi version 1.8 >> - build with setuptools >> - postprocessing step to rename shared library/DLL >> - postprocessing step to regenerate renamed whl file with renamed SO/DLL > > There wouldn't necessarily need to be renaming. Aye, that description was for the process if there weren't any changes to the toolchain. > build_ext command determines > the DLL extension. It could be patched or modified to read an "I'm ABI3" > flag on the Extension() object. > > We could pass an ABI3 flag to bdist_wheel in the same way we ask for > universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3 > extensions, and ignored on py2. The general idea sounds good to me, but as a slight bikeshed on the flag name, perhaps "cpabi3"? That's a mash-up of the 'cp' interpreter code for CPython, with the 'abi3' stable ABI tag. Longer term, we may want to allow people to version that (as new APIs may sometimes be added to the stable ABI, which you gain access to at the C level by setting Py_LIMITED_API to the corresponding CPython hex version rather than just defining it [1]), but as a starting point enabling access to the initial 3.2 stable ABI used by cffi should be sufficient. >> Of those changes, only the "somehow be Py_LIMITED_API aware" sounds >> potentially tricky to me, as I'd be surprised if there was any way >> around requiring a new explicit setting in either setup.py or >> setup.cfg. > > > Any alternative to an explicit flag that I can think of would be too magical > to consider, as poor as trying to inspect library symbols for manylinux1 > compat in bdist_wheel itself. Aye, that was my conclusion as well. Once my brain started going "Well, we could scan the source for..." I just went "Ewww, no, we can just ask people to set a new flag" :) However, would it make sense for the new flag to also implicitly define Py_LIMITED_API in the compile flags when building the extension, even if it isn't otherwise specified in the extension's source code? That way folks would still only need to define their intent to use the flag in one place, it's just that that place would be their build instructions rather than the extension module source code. Cheers, Nick. [1] https://docs.python.org/3/c-api/stable.html -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
On Mon, Aug 1, 2016 at 1:42 AM Nick Coghlan wrote: > On 1 August 2016 at 05:27, Daniel Holth wrote: > > The next version of cffi will contain small changes to generate code > > compliant with Python's Py_LIMITED_API: > > https://bitbucket.org/cffi/cffi/commits/8f867f5a869f > > > > (although cffi itself is not, the extensions it generates will be). > > Very cool! > > > If we also add an appropriate supported tag to pip ~= cp3.abi3.manylinux1 > > and provide a way to name the generated DLL's appropriately, it may > become > > possible to reduce the burden of distributing cffi extensions, especially > > for Windows. One compiled artifact should work on Python 3.2 and above. > > Aye, that would be very promising. I guess the first step would be to > document the current steps involved in doing this manually? And then > figure out what tweaks would be needed to setuptools and pip to allow > it to be automated? > > As an initial stab at that: > > Status quo, on publication side: > > - require minimum cffi version 1.8 > - build with setuptools > - postprocessing step to rename shared library/DLL > - postprocessing step to regenerate renamed whl file with renamed SO/DLL > There wouldn't necessarily need to be renaming. build_ext command determines the DLL extension. It could be patched or modified to read an "I'm ABI3" flag on the Extension() object. We could pass an ABI3 flag to bdist_wheel in the same way we ask for universal 'py2.py3-none-any'. To be set if the wheel contained only ABI3 extensions, and ignored on py2. > On consumption side: > > - requires "cp3.abi3" to be included at appropriate points in > compatibility tag list > > Which would make the necessary changes be: > > - updating setuptools to somehow be Py_LIMITED_API aware when naming > built extensions > - updating wheel to somehow be Py_LIMITED_API aware when naming whl files > - if necessary, updating pip's compatibility tag list > > Of those changes, only the "somehow be Py_LIMITED_API aware" sounds > potentially tricky to me, as I'd be surprised if there was any way > around requiring a new explicit setting in either setup.py or > setup.cfg. > Any alternative to an explicit flag that I can think of would be too magical to consider, as poor as trying to inspect library symbols for manylinux1 compat in bdist_wheel itself. > Regards, > Nick. > > -- > Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia > ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
Re: [Distutils] cffi & Py_LIMITED_API
On 1 August 2016 at 05:27, Daniel Holth wrote: > The next version of cffi will contain small changes to generate code > compliant with Python's Py_LIMITED_API: > https://bitbucket.org/cffi/cffi/commits/8f867f5a869f > > (although cffi itself is not, the extensions it generates will be). Very cool! > If we also add an appropriate supported tag to pip ~= cp3.abi3.manylinux1 > and provide a way to name the generated DLL's appropriately, it may become > possible to reduce the burden of distributing cffi extensions, especially > for Windows. One compiled artifact should work on Python 3.2 and above. Aye, that would be very promising. I guess the first step would be to document the current steps involved in doing this manually? And then figure out what tweaks would be needed to setuptools and pip to allow it to be automated? As an initial stab at that: Status quo, on publication side: - require minimum cffi version 1.8 - build with setuptools - postprocessing step to rename shared library/DLL - postprocessing step to regenerate renamed whl file with renamed SO/DLL On consumption side: - requires "cp3.abi3" to be included at appropriate points in compatibility tag list Which would make the necessary changes be: - updating setuptools to somehow be Py_LIMITED_API aware when naming built extensions - updating wheel to somehow be Py_LIMITED_API aware when naming whl files - if necessary, updating pip's compatibility tag list Of those changes, only the "somehow be Py_LIMITED_API aware" sounds potentially tricky to me, as I'd be surprised if there was any way around requiring a new explicit setting in either setup.py or setup.cfg. Regards, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
[Distutils] cffi & Py_LIMITED_API
The next version of cffi will contain small changes to generate code compliant with Python's Py_LIMITED_API: https://bitbucket.org/cffi/cffi/commits/8f867f5a869f (although cffi itself is not, the extensions it generates will be). If we also add an appropriate supported tag to pip ~= cp3.abi3.manylinux1 and provide a way to name the generated DLL's appropriately, it may become possible to reduce the burden of distributing cffi extensions, especially for Windows. One compiled artifact should work on Python 3.2 and above. ___ Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
