Re: [Python-Dev] What version is an extension module binary compatible with

[Petr Viktorin]

On 03/30/2017 06:31 AM, Nick Coghlan wrote:

On 29 March 2017 at 02:18, Paul Moore  wrote:

On 28 March 2017 at 12:24, Miro Hrončok  wrote:

I'd like some clarification on what ABI compatibility we can expect.
 * Should the ABI be stable across patch releases (so calling
PySlice_AdjustIndices from an existing macro would be a bug)?
 * Should the ABI be forward-compatible within a minor release (so modules
built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
 * Or should we expect the ABI to change even across patch releases?


Given that binary wheels are built against a specific minor version
(3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
release. That would fit with my expectations of the compatibility
guarantees on patch releases.

So I from what you describe, I'd consider this as a bug. Certainly, if
someone built a C extension as a wheel using Python 3.6.1, it would be
tagged as compatible with cp36, and pip would happily use it when
installing to a Python 3.6.0 system, where it would fail.


Right, this is the main problem - while "build against the X.Y.0
headers" is useful advice, it's not something we've ever explicitly
stated, and it's not something we can reasonably expect all providers
of pre-built binary modules to do.


Also, while building against 3.6.0 headers will ensure compatibility, it 
will also restore the original bug that PySlice_AdjustIndices fixes.
Expecting extension authors to build against x.y.0 would lock them out 
of such bug fixes in later releases.



Instead, it makes sense to explicitly strengthen the ABI guarantees
within CPython maintenance releases, and add some automated testing to
avoid accidental changes and oversights (similar to the pending test
to ensure magic number stability for cached bytecode files)

Cheers,
Nick.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Martin Panter]

On 30 March 2017 at 15:31, Nick Coghlan  wrote:
> On 29 March 2017 at 02:18, Paul Moore  wrote:
>> On 28 March 2017 at 12:24, Miro Hrončok  wrote:
>>> I'd like some clarification on what ABI compatibility we can expect.
>>>  * Should the ABI be stable across patch releases (so calling
>>> PySlice_AdjustIndices from an existing macro would be a bug)?
>>>  * Should the ABI be forward-compatible within a minor release (so modules
>>> built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
>>>  * Or should we expect the ABI to change even across patch releases?
>>
>> Given that binary wheels are built against a specific minor version
>> (3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
>> release. That would fit with my expectations of the compatibility
>> guarantees on patch releases.
>>
>> So I from what you describe, I'd consider this as a bug. Certainly, if
>> someone built a C extension as a wheel using Python 3.6.1, it would be
>> tagged as compatible with cp36, and pip would happily use it when
>> installing to a Python 3.6.0 system, where it would fail.
>
> Right, this is the main problem - while "build against the X.Y.0
> headers" is useful advice, it's not something we've ever explicitly
> stated, and it's not something we can reasonably expect all providers
> of pre-built binary modules to do.
>
> Instead, it makes sense to explicitly strengthen the ABI guarantees
> within CPython maintenance releases, and add some automated testing to
> avoid accidental changes and oversights (similar to the pending test
> to ensure magic number stability for cached bytecode files)

There's a website that has nice reports of ABI compatibility of
various packages and might useful for testing. It shows up the added
PySlice_AdjustIndices function in 3.6.1, along with PySlice_Unpack
(and some changes to internal names, so probably not such a problem).
https://abi-laboratory.pro/tracker/compat_report/python/3.6.0/3.6.1/496a4/abi_compat_report.html
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Nick Coghlan]

On 29 March 2017 at 02:18, Paul Moore  wrote:
> On 28 March 2017 at 12:24, Miro Hrončok  wrote:
>> I'd like some clarification on what ABI compatibility we can expect.
>>  * Should the ABI be stable across patch releases (so calling
>> PySlice_AdjustIndices from an existing macro would be a bug)?
>>  * Should the ABI be forward-compatible within a minor release (so modules
>> built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
>>  * Or should we expect the ABI to change even across patch releases?
>
> Given that binary wheels are built against a specific minor version
> (3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
> release. That would fit with my expectations of the compatibility
> guarantees on patch releases.
>
> So I from what you describe, I'd consider this as a bug. Certainly, if
> someone built a C extension as a wheel using Python 3.6.1, it would be
> tagged as compatible with cp36, and pip would happily use it when
> installing to a Python 3.6.0 system, where it would fail.

Right, this is the main problem - while "build against the X.Y.0
headers" is useful advice, it's not something we've ever explicitly
stated, and it's not something we can reasonably expect all providers
of pre-built binary modules to do.

Instead, it makes sense to explicitly strengthen the ABI guarantees
within CPython maintenance releases, and add some automated testing to
avoid accidental changes and oversights (similar to the pending test
to ensure magic number stability for cached bytecode files)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Nathaniel Smith]

On Wed, Mar 29, 2017 at 8:22 AM, Paul Moore  wrote:
> On 28 March 2017 at 17:31, Nathaniel Smith  wrote:
>> IMO this is a bug, and depending on how many packages are affected it might
>> even call for an emergency 3.6.2. The worst case is that we start getting
>> large numbers of packages uploaded to pypi that claim to be 3.6.0 compatible
>> but that crash like crash with an obscure error when people download them.
>
> Has anyone logged this on bugs.python.org? There's nothing in the
> Fedora bug referenced by the OP that indicates they've done so.

I didn't see one, so: https://bugs.python.org/issue29943

-n

-- 
Nathaniel J. Smith -- https://vorpus.org
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Paul Moore]

On 28 March 2017 at 17:31, Nathaniel Smith  wrote:
> IMO this is a bug, and depending on how many packages are affected it might
> even call for an emergency 3.6.2. The worst case is that we start getting
> large numbers of packages uploaded to pypi that claim to be 3.6.0 compatible
> but that crash like crash with an obscure error when people download them.

Has anyone logged this on bugs.python.org? There's nothing in the
Fedora bug referenced by the OP that indicates they've done so.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Nathaniel Smith]

On Mar 28, 2017 10:54 AM, "Steve Dower"  wrote:

On 28Mar2017 1035, Paul Moore wrote:

> On 28 March 2017 at 18:05, Glenn Linderman  wrote:
>
>> Somewhere I got the idea that extension authors were supposed to build
>> against the n.m.0 releases, expressly so that the extensions would then be
>> compatible with the whole n.m.x series of releases. Did I dream that?
>>
>
> I've certainly never heard it stated - and I'd think it's a pretty
> annoying requirement to make of extension builders.
> Paul
>

Agreed, we should avoid both additive and subtractive breaking changes to
the binary API within micro versions completely. Additive ones like this
are difficult to catch, unfortunately, and so building against the .0
release is not bad advice. Building with the .0 headers is probably
sufficient.

I wonder if there's a way we can preprocess the headers to define a
baseline? Then we could automatically compare against that and force
explicit decisions leading to public API changes (probably the process of
finishing off the limited API validation could include this task fairly
easily as well).


It wouldn't be as fancy as analyzing the headers, but a much easier and
still useful way to get started would be a test to check no new exported
symbols appear in the shared library during a stable release cycle.

If you want to get quite fancy, libabigail provides a toolkit that can read
debug information and check that all your structs remain the same size etc.
ELF only, but still would catch a lot:

https://sourceware.org/libabigail/manual/libabigail-overview.html

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Steve Dower]

On 28Mar2017 1035, Paul Moore wrote:

On 28 March 2017 at 18:05, Glenn Linderman  wrote:

Somewhere I got the idea that extension authors were supposed to build
against the n.m.0 releases, expressly so that the extensions would then be
compatible with the whole n.m.x series of releases. Did I dream that?


I've certainly never heard it stated - and I'd think it's a pretty
annoying requirement to make of extension builders.
Paul


Agreed, we should avoid both additive and subtractive breaking changes 
to the binary API within micro versions completely. Additive ones like 
this are difficult to catch, unfortunately, and so building against the 
.0 release is not bad advice. Building with the .0 headers is probably 
sufficient.


I wonder if there's a way we can preprocess the headers to define a 
baseline? Then we could automatically compare against that and force 
explicit decisions leading to public API changes (probably the process 
of finishing off the limited API validation could include this task 
fairly easily as well).


Cheers,
Steve
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Paul Moore]

On 28 March 2017 at 18:05, Glenn Linderman  wrote:
> Somewhere I got the idea that extension authors were supposed to build
> against the n.m.0 releases, expressly so that the extensions would then be
> compatible with the whole n.m.x series of releases. Did I dream that?

I've certainly never heard it stated - and I'd think it's a pretty
annoying requirement to make of extension builders.
Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Glenn Linderman]

On 3/28/2017 9:18 AM, Paul Moore wrote:

On 28 March 2017 at 12:24, Miro Hrončok  wrote:

I'd like some clarification on what ABI compatibility we can expect.
  * Should the ABI be stable across patch releases (so calling
PySlice_AdjustIndices from an existing macro would be a bug)?
  * Should the ABI be forward-compatible within a minor release (so modules
built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
  * Or should we expect the ABI to change even across patch releases?

Given that binary wheels are built against a specific minor version
(3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
release. That would fit with my expectations of the compatibility
guarantees on patch releases.

So I from what you describe, I'd consider this as a bug. Certainly, if
someone built a C extension as a wheel using Python 3.6.1, it would be
tagged as compatible with cp36, and pip would happily use it when
installing to a Python 3.6.0 system, where it would fail.

Somewhere I got the idea that extension authors were supposed to build 
against the n.m.0 releases, expressly so that the extensions would then 
be compatible with the whole n.m.x series of releases. Did I dream that?
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Nathaniel Smith]

On Mar 28, 2017 8:29 AM, "Serhiy Storchaka"  wrote:

On 28.03.17 14:24, Miro Hrončok wrote:

> However, recently we found an issue with this approach [1]: an extension
> module built against Python 3.6.1 cannot be run on Python 3.6.0, because
> it uses a macro that, in 3.6.1, uses the new PySlice_AdjustIndices
> function.
>

The macro expanding to PySlice_AdjustIndices is used only when
Py_LIMITED_API is not defined or is defined to the version equal or greater
the version in which PySlice_AdjustIndices was added.


That's nice, but not sufficient. Py_LIMITED_ABI is cool, but the vast
majority of distributed packages don't use it, and instead rely on the
"unlimited" ABI being forward and backwards compatible within each minor
release. For example, this assumption is hard coded in the wheel format,
which has no way to even describe a wheel that needs 3.6.x with x >= 1.
People uploading packages to pypi use whatever version of 3.6 they have
lying around and assume it will work for everyone downloading.

IMO this is a bug, and depending on how many packages are affected it might
even call for an emergency 3.6.2. The worst case is that we start getting
large numbers of packages uploaded to pypi that claim to be 3.6.0
compatible but that crash like crash with an obscure error when people
download them.

-n
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Paul Moore]

On 28 March 2017 at 12:24, Miro Hrončok  wrote:
> I'd like some clarification on what ABI compatibility we can expect.
>  * Should the ABI be stable across patch releases (so calling
> PySlice_AdjustIndices from an existing macro would be a bug)?
>  * Should the ABI be forward-compatible within a minor release (so modules
> built for 3.6.0 should be usable with 3.6.1, but not vice versa)?
>  * Or should we expect the ABI to change even across patch releases?

Given that binary wheels are built against a specific minor version
(3.6, 3.5, ...) I would expect the ABI to be consistent over a minor
release. That would fit with my expectations of the compatibility
guarantees on patch releases.

So I from what you describe, I'd consider this as a bug. Certainly, if
someone built a C extension as a wheel using Python 3.6.1, it would be
tagged as compatible with cp36, and pip would happily use it when
installing to a Python 3.6.0 system, where it would fail.

Paul
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

Re: [Python-Dev] What version is an extension module binary compatible with

[Serhiy Storchaka]

On 28.03.17 14:24, Miro Hrončok wrote:

However, recently we found an issue with this approach [1]: an extension
module built against Python 3.6.1 cannot be run on Python 3.6.0, because
it uses a macro that, in 3.6.1, uses the new PySlice_AdjustIndices
function.


The macro expanding to PySlice_AdjustIndices is used only when 
Py_LIMITED_API is not defined or is defined to the version equal or 
greater the version in which PySlice_AdjustIndices was added.



___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com

[Python-Dev] What version is an extension module binary compatible with

[Miro Hrončok]

Hi,

as per [0], ABI of the C API is generally not stable and the binary 
compatibility may break between versions. It is hard from the text to 
know whether it talks about minor versions (such as 3.6 vs 3.5) or patch 
versions (such as 3.6.1 vs 3.6.0).


In Fedora we currently only keep track about the minor version 
dependency. I.e. an RPM package with a Python module depends on Python 
3.6, not specifically on Python 3.6.1.


However, recently we found an issue with this approach [1]: an extension 
module built against Python 3.6.1 cannot be run on Python 3.6.0, because 
it uses a macro that, in 3.6.1, uses the new PySlice_AdjustIndices function.


I'd like some clarification on what ABI compatibility we can expect.
 * Should the ABI be stable across patch releases (so calling 
PySlice_AdjustIndices from an existing macro would be a bug)?
 * Should the ABI be forward-compatible within a minor release (so 
modules built for 3.6.0 should be usable with 3.6.1, but not vice versa)?

 * Or should we expect the ABI to change even across patch releases?

It would be nice to say this explicitly in the docs ([0] or another 
suitable place).



[0] https://docs.python.org/3/c-api/stable.html
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1435135

Thanks for clarification,
On behalf of the Fedora Python SIG,
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
Python-Dev mailing list
Python-Dev@python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: 
https://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com