[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[David Cuthbert via Python-Dev]

This is too-little, too-late (I was offline this past week), but for those who 
enjoy digging through historical archives, the Tcl folks had an interesting 
stubs mechanism that was *supposed* to solve the versioning issue (although I 
suspect it hasn’t actually done much in that regard) in addition to the 
“cross-platform dynamic linking is a royal pain” (circa 1999) issue: 
https://wiki.tcl-lang.org/page/Stubs

Basically, you just use a vtable of function pointers exported by version, e.g. 
Tcl_8_0_Stubs, Tcl_8_1_Stubs, etc. The clients call each function via a layer 
of indirection, Tcl_8_0_Stubs->Tcl_Tell(), Tcl_8_1_Stubs->Tcl_Tell(), etc.; 
this is hidden by C macros. (From what I recall of COM, this is a rudimentary 
implementation of a COM interface.)


I’m not sure I would recommend this today -- we were solving different problems 
back then (CAD workstations from various Unix vendors patched 
Frankenstein-esque library mismatches), and compiler optimizations weren’t as 
sophisticated. It had the nice advantage of crashing immediately due to a 
version mismatch rather than silently corrupting data structures that had 
undergone changes, etc.


Back then, GNU Binutils symbol versioning was just adding to our headaches 
(again, telling our customers to fix their workstations wasn’t a plausible 
solution, and it was only on Linux, which was then a very small part of the 
installed base).





> On Oct 1, 2019, at 1:45 AM, Victor Stinner  wrote:
> 
> Sorry for the bad timing. I dislike working under pressure. The
> discussion on python-dev, the bug tracker and pull requests was really
> interesting.
> 
> The issue has been fixed: the whole idea of stable ABI for PyConfig
> has been abandoned. If you want to switch to a different version of
> Python when you embed Python, you must recompile your code. It was
> always like that and the PEP 587 doesn't change anything.
> 
> PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig()
> initialize PyConfig to sane default configuration. So if we add new
> fields in Python 3.9, you don't have to manually initialize these
> fields, except if you explicitly want to use these new fields to
> benefit of new configuration options.
> 
> Victor
> 
> Le mar. 1 oct. 2019 à 00:38, Nick Coghlan  a écrit :
>> 
>> 
>> 
>> On Tue., 1 Oct. 2019, 6:47 am Victor Stinner,  wrote:
>>> 
>>> Hi back,
>>> 
>>> It seems like "config.struct_size = sizeof(PyConfig);" syntax is "not
>>> really liked".
>>> 
>>> I don't see any traction to provide a stable ABI for embedded Python.
>>> The consensus is more towards: "it doesn't work and we don't want to
>>> bother with false promises" (nor add code for that).
>>> 
>>> Since Lukasz is supposed to tag 3.8.0rc1 today, I propose to remove
>>> anything related to "stable API" or "stable ABI" from the PEP 587
>>> right now. I wrote two PRs to remove struct_size:
>>> 
>>> * CPython: https://github.com/python/cpython/pull/16500
>>> * PEP 587: https://github.com/python/peps/pull/1185
>>> 
>>> Later, if we decide to start proving a stable ABI for embedded Python,
>>> we can still add a "version" or "struct_size" field to PyConfig later
>>> (for example in Python 3.9).
>> 
>> 
>> Thanks Victor, I think this is the right way for us to go, given the 
>> relative timing in the release cycle.
>> 
>> The idea of some day expanding the stable ABI to cover embedding 
>> applications, not just extension modules, is an intriguing one, but I'm 
>> genuinely unsure anyone would ever actually use it.
>> 
>> Cheers,
>> Nick.
>> 
>> 
>> 
>>> 
>>> Victor
>>> ___
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-le...@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at 
>>> https://mail.python.org/archives/list/python-dev@python.org/message/2JAJQA5OANFPXAJ3327RRPHPQLKVP2EW/
> 
> 
> 
> -- 
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/36RTEPNE6HSPMXGDGSHWDJN56GZRX4YU/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U63GT4FYZN7PBG22ZJD522EPYKM3OPQP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Ronald Oussoren via Python-Dev]

> On 1 Oct 2019, at 10:55, Thomas Wouters  wrote:
> 
> 
> 
> On Mon, Sep 30, 2019 at 11:00 PM Brett Cannon  > wrote:
> Victor Stinner wrote:
> > Hi Nick,
> > Le dim. 29 sept. 2019 à 08:47, Nick Coghlan ncogh...@gmail.com 
> >  a écrit :
> > > I don't quite understand the purpose of this change,
> > > as there's no
> > > stable ABI for applications embedding CPython.
> > > Well, I would like to prepare Python to provide a stable ABI for
> > embedded Python. While it's not a design goal yet
> > (Include/cpython/initconfig.h is currently excluded from
> > Py_LIMITED_API), this change is a step towards that.
> 
> So then isn't this a very last-minute, premature optimization if it isn't a 
> design goal yet? If that's the case then I would vote not to make the change 
> and wait until there's been feedback on 3.8 and then look at stabilizing the 
> embedding API such that it can be considered stable.
> 
> I just want to chime in here and confirm, as a Professional CPython 
> Embedder(tm) that embedders cannot currently rely on a stable ABI, or the 
> limited API, and often not even the public API. It's not just about exposed 
> symbols and struct sizes, it's often also about the semantics of internals 
> (e.g. how importlib handles custom module finders and loaders) that subtly 
> changes. For anything but the simplest PyRun_SimpleString-based embedding 
> this is more the case when embedding than extending, and even than a regular 
> (complex) Python program. (I already showed Victor and a few others some of 
> the hoops we have to jump through at Google to embed CPython correctly, and 
> only half of those things are necessary just because of Google's environment.)

On the other hand, py2app uses a single executable that loads a python shared 
library, initialises the interpreter and runs a script with minimal changes 
over the years. I’ve basically only needed to recompile to support new macOS 
architectures and to add support for Python 3. 

That said, py2app’s stub executable is basically just “simple 
PyRun_SimpleString-based embedding” with ugly code to make it possible to use 
dlopen to load its dependencies. 

Ronald

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/N2GK4MDYGQYUXOJDKHXKW4ZIM7OQYGJY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Thomas Wouters]

On Mon, Sep 30, 2019 at 11:00 PM Brett Cannon  wrote:

> Victor Stinner wrote:
> > Hi Nick,
> > Le dim. 29 sept. 2019 à 08:47, Nick Coghlan ncogh...@gmail.com a écrit :
> > > I don't quite understand the purpose of this change,
> > > as there's no
> > > stable ABI for applications embedding CPython.
> > > Well, I would like to prepare Python to provide a stable ABI for
> > embedded Python. While it's not a design goal yet
> > (Include/cpython/initconfig.h is currently excluded from
> > Py_LIMITED_API), this change is a step towards that.
>
> So then isn't this a very last-minute, premature optimization if it isn't
> a design goal yet? If that's the case then I would vote not to make the
> change and wait until there's been feedback on 3.8 and then look at
> stabilizing the embedding API such that it can be considered stable.
>

I just want to chime in here and confirm, as a Professional CPython
Embedder(tm) that embedders cannot currently rely on a stable ABI, or the
limited API, and often not even the public API. It's not just about exposed
symbols and struct sizes, it's often also about the semantics of internals
(e.g. how importlib handles custom module finders and loaders) that subtly
changes. For anything but the simplest PyRun_SimpleString-based embedding
this is more the case when embedding than extending, and even than a
regular (complex) Python program. (I already showed Victor and a few others
some of the hoops we have to jump through at Google to embed CPython
correctly, and only half of those things are necessary just because of
Google's environment.)


>
> -Brett
>
> > > As a result, updating
> > > to a new X.Y.0 release always requires rebuilding the entire
> > > application, not just building and relinking CPython.
> > > In Python 3.8, C extensions are no longer linked to libpython which
> > allows to switch between a release build and a debug build of
> > libpython.
> > Can we imagine the same idea for embedded Python? I checked vim on
> > Linux: it's linked to libpython3.7m.so.1.0: a specific Python version,
> > library built in release mode.
> > > I could understand a change to require passing in an
> > > expected Python
> > > version so we can fail more gracefully on a bad link where an
> > > application that intended to embed Python 3.8 is incorrectly linked
> > > against Python 3.9 (for example), but performing that kind of check
> > > would require passing in PY_VERSION_HEX, not the size of the config
> > > struct.
> > > It seems simpler to me to pass the structure size rather than the
> > Python version. It avoids the risk of updating the structure without
> > update the Python version. I also avoids to have to change the Python
> > version immediately when PyConfig is modified. The main risk of
> > sizeof(PyConfig) comes if we remove a field and add a new field of
> > the same size: the structure size doesn't change... But in my
> > experience, we only add new ways to configure Pyhon, we never remove
> > old ones :-D
> > The question is if it's convenient to compute sizeof(PyConfig) in
> > programming languages other than C. Providing a "structure version" or
> > the structure size from a function call would not work. The value must
> > be known a compilation time, not at runtime. The purpose is to compare
> > the version/size between build and runtime (they must match).
> > In the first implementation of my PEP, I used an internal "config
> > version" provides by a macro. But it was said that macros are not
> > convenient.
> > PY_VERSION_HEX is provided as a macro, but we are now trying to avoid
> > macros in our C API, no? At least, it's what I understood from the PEP
> > 587 discussion.
> > > We don't support that - all our APIs that accept
> > > PyObject/PyTypeObject/etc require that the caller pass in structs of
> > > the correct size for the version of Python being used.
> > > For PyConfig, it's allocated (on the stack on or on heap) by the
> > application. So the application requires to access to the full
> > structure.
> > Objects (instances) are allocated by Python (on the heap).
> > Applications usually don't need to know/access the structure.
> > Python is far from being perfect, static types are still supported and
> > they are an issue for the stable ABI.
> > > The PyConfig
> > > and PyPreConfig structs are no different from PyObject in that regard:
> > > if there's a size mismatch, then the developers of the embedding
> > > application have somehow messed up their build process.
> > > In short, PyConfig initialization works like that:
> >
> > The application allocates a PyConfig object on the stack
> > Python calls memset(config, 0, sizeof(PyConfig))
> >
> > If there is a size mismatch, Python triggers a buffer overflow which
> > is likely to cause issues.
> > I prefer to have a clean API which makes buffer overflow impossible.
> > Embedding Python and handling different Python versions is not
> > trivial, especially if we start to add new fields to each PyConfig
> > (

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Victor Stinner]

Sorry for the bad timing. I dislike working under pressure. The
discussion on python-dev, the bug tracker and pull requests was really
interesting.

The issue has been fixed: the whole idea of stable ABI for PyConfig
has been abandoned. If you want to switch to a different version of
Python when you embed Python, you must recompile your code. It was
always like that and the PEP 587 doesn't change anything.

PyConfig_InitPythonConfig() and PyConfig_InitIsolatedConfig()
initialize PyConfig to sane default configuration. So if we add new
fields in Python 3.9, you don't have to manually initialize these
fields, except if you explicitly want to use these new fields to
benefit of new configuration options.

Victor

Le mar. 1 oct. 2019 à 00:38, Nick Coghlan  a écrit :
>
>
>
> On Tue., 1 Oct. 2019, 6:47 am Victor Stinner,  wrote:
>>
>> Hi back,
>>
>> It seems like "config.struct_size = sizeof(PyConfig);" syntax is "not
>> really liked".
>>
>> I don't see any traction to provide a stable ABI for embedded Python.
>> The consensus is more towards: "it doesn't work and we don't want to
>> bother with false promises" (nor add code for that).
>>
>> Since Lukasz is supposed to tag 3.8.0rc1 today, I propose to remove
>> anything related to "stable API" or "stable ABI" from the PEP 587
>> right now. I wrote two PRs to remove struct_size:
>>
>> * CPython: https://github.com/python/cpython/pull/16500
>> * PEP 587: https://github.com/python/peps/pull/1185
>>
>> Later, if we decide to start proving a stable ABI for embedded Python,
>> we can still add a "version" or "struct_size" field to PyConfig later
>> (for example in Python 3.9).
>
>
> Thanks Victor, I think this is the right way for us to go, given the relative 
> timing in the release cycle.
>
> The idea of some day expanding the stable ABI to cover embedding 
> applications, not just extension modules, is an intriguing one, but I'm 
> genuinely unsure anyone would ever actually use it.
>
> Cheers,
> Nick.
>
>
>
>>
>> Victor
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/2JAJQA5OANFPXAJ3327RRPHPQLKVP2EW/



-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/36RTEPNE6HSPMXGDGSHWDJN56GZRX4YU/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Steve Dower]

On 30Sep2019 1625, Nick Coghlan wrote:
After merging your PR and closing mine, I had an idea for Python 3.9: 
what if we offered a separate public "int 
Py_CheckVersionCompatibility(uint64_t header_version)" call? (64 bit 
input rather than 32 to allow for possible future changes to the version 
number formatting scheme)


The basic variant of that API would be what I had in my PR: release 
candidates and final releases allow an inexact match, other releases 
require the hex version to match exactly.


As I posted on the issue tracker, I don't think this kind of version 
verification is necessary unless we're genuinely going to get into the 
business of adding built-in shims for API changes.


Every supported platform is going to make you link against a dynamic 
library with the version number in its name. (Every approach that avoids 
this is not supported or does not support embedding.) So you get 
run-time checking for free, and you can add a compile-time check in 
three or four different ways depending on what makes the most sense for 
your build system.


Let's not complicate the embedding API further with unnecessary new 
APIs. I'm keen to design something broadly useful and simpler than what 
we have now, and this is not going to help with it.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/35WNLKQL4DWZZ6MS7OEZEDV7AM2IZFKV/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Tue., 1 Oct. 2019, 8:38 am Nick Coghlan,  wrote:

> Later, if we decide to start proving a stable ABI for embedded Python,
>> we can still add a "version" or "struct_size" field to PyConfig later
>> (for example in Python 3.9).
>>
>
> Thanks Victor, I think this is the right way for us to go, given the
> relative timing in the release cycle.
>
> The idea of some day expanding the stable ABI to cover embedding
> applications, not just extension modules, is an intriguing one, but I'm
> genuinely unsure anyone would ever actually use it.
>

After merging your PR and closing mine, I had an idea for Python 3.9: what
if we offered a separate public "int Py_CheckVersionCompatibility(uint64_t
header_version)" call? (64 bit input rather than 32 to allow for possible
future changes to the version number formatting scheme)

The basic variant of that API would be what I had in my PR: release
candidates and final releases allow an inexact match, other releases
require the hex version to match exactly.

If an embedding application called that before calling any PreConfig or
Config APIs, they'd get the same easier debugging for version mismatches,
without needing to change the config structs themselves.

Instead, we'd only need a new PreConfig field if we wanted to start having
other runtime APIs change their behaviour based on the active nominal
Python version.

As an added bonus, extension modules could also optionally call that
compatibility checking API early in their init function.

If we wanted to get more exotic with the internal design, we could maintain
a table of beta versions containing known ABI *breaks* (e.g. public structs
changing size), and also permit inexact matches for beta releases, as long
as the given header version was newer than the last ABI break, but older
than the runtime version.

That table could be reset to empty when the ABI was frozen for a release
series (thus causing a merge conflict if a backport was requested for a
patch that changed that table).

Cheers,
Nick.



>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7A7HYAAQWBZEEFRO4PCA5ARCX6U2DTVV/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Tue., 1 Oct. 2019, 6:47 am Victor Stinner,  wrote:

> Hi back,
>
> It seems like "config.struct_size = sizeof(PyConfig);" syntax is "not
> really liked".
>
> I don't see any traction to provide a stable ABI for embedded Python.
> The consensus is more towards: "it doesn't work and we don't want to
> bother with false promises" (nor add code for that).
>
> Since Lukasz is supposed to tag 3.8.0rc1 today, I propose to remove
> anything related to "stable API" or "stable ABI" from the PEP 587
> right now. I wrote two PRs to remove struct_size:
>
> * CPython: https://github.com/python/cpython/pull/16500
> * PEP 587: https://github.com/python/peps/pull/1185
>
> Later, if we decide to start proving a stable ABI for embedded Python,
> we can still add a "version" or "struct_size" field to PyConfig later
> (for example in Python 3.9).
>

Thanks Victor, I think this is the right way for us to go, given the
relative timing in the release cycle.

The idea of some day expanding the stable ABI to cover embedding
applications, not just extension modules, is an intriguing one, but I'm
genuinely unsure anyone would ever actually use it.

Cheers,
Nick.




> Victor
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/2JAJQA5OANFPXAJ3327RRPHPQLKVP2EW/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OMPEQEEJEKUH2KVSUV5UDGMI7IP3PP4Y/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Victor Stinner]

Le lun. 30 sept. 2019 à 23:26, Antoine Pitrou  a écrit :
> I think if you wanted to make the PyConfig apt at providing
> ABI-stability, you should have designed it differently.
> `PyType_FromSpec` provides a useful model (pass an arbitrary-sized
> static array of field initializers).

PyConfig is even more complex than PyTypeObject :-/ The following
PyConfig methods allocates memory on the heap using PyMem_RawMalloc():

PyConfig_SetArgv()
PyConfig_SetBytesArgv()
PyConfig_SetBytesString()
PyConfig_SetString()
PyConfig_SetWideStringList()

Setting a field can fail with a memory allocation failure,
preinitialization error, or other errors.

But right, I get your point, there are other solutions.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/KW367IBINBG7OWVOVU4DBWJLZDIII27J/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Antoine Pitrou]

On Mon, 30 Sep 2019 22:43:40 +0200
Victor Stinner  wrote:
> Hi back,
> 
> It seems like "config.struct_size = sizeof(PyConfig);" syntax is "not
> really liked".
> 
> I don't see any traction to provide a stable ABI for embedded Python.

Not at the last minute in a rc1, I'd say :-)

I think if you wanted to make the PyConfig apt at providing
ABI-stability, you should have designed it differently.
`PyType_FromSpec` provides a useful model (pass an arbitrary-sized
static array of field initializers).

Regards

Antoine.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/FJ2CT5ISRMH4PGI22VTU3JSUJ7AFSYLE/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Brett Cannon]

Victor Stinner wrote:
> Hi Nick,
> Le dim. 29 sept. 2019 à 08:47, Nick Coghlan ncogh...@gmail.com a écrit :
> > I don't quite understand the purpose of this change,
> > as there's no
> > stable ABI for applications embedding CPython.
> > Well, I would like to prepare Python to provide a stable ABI for
> embedded Python. While it's not a design goal yet
> (Include/cpython/initconfig.h is currently excluded from
> Py_LIMITED_API), this change is a step towards that.

So then isn't this a very last-minute, premature optimization if it isn't a 
design goal yet? If that's the case then I would vote not to make the change 
and wait until there's been feedback on 3.8 and then look at stabilizing the 
embedding API such that it can be considered stable.

-Brett

> > As a result, updating
> > to a new X.Y.0 release always requires rebuilding the entire
> > application, not just building and relinking CPython.
> > In Python 3.8, C extensions are no longer linked to libpython which
> allows to switch between a release build and a debug build of
> libpython.
> Can we imagine the same idea for embedded Python? I checked vim on
> Linux: it's linked to libpython3.7m.so.1.0: a specific Python version,
> library built in release mode.
> > I could understand a change to require passing in an
> > expected Python
> > version so we can fail more gracefully on a bad link where an
> > application that intended to embed Python 3.8 is incorrectly linked
> > against Python 3.9 (for example), but performing that kind of check
> > would require passing in PY_VERSION_HEX, not the size of the config
> > struct.
> > It seems simpler to me to pass the structure size rather than the
> Python version. It avoids the risk of updating the structure without
> update the Python version. I also avoids to have to change the Python
> version immediately when PyConfig is modified. The main risk of
> sizeof(PyConfig) comes if we remove a field and add a new field of
> the same size: the structure size doesn't change... But in my
> experience, we only add new ways to configure Pyhon, we never remove
> old ones :-D
> The question is if it's convenient to compute sizeof(PyConfig) in
> programming languages other than C. Providing a "structure version" or
> the structure size from a function call would not work. The value must
> be known a compilation time, not at runtime. The purpose is to compare
> the version/size between build and runtime (they must match).
> In the first implementation of my PEP, I used an internal "config
> version" provides by a macro. But it was said that macros are not
> convenient.
> PY_VERSION_HEX is provided as a macro, but we are now trying to avoid
> macros in our C API, no? At least, it's what I understood from the PEP
> 587 discussion.
> > We don't support that - all our APIs that accept
> > PyObject/PyTypeObject/etc require that the caller pass in structs of
> > the correct size for the version of Python being used.
> > For PyConfig, it's allocated (on the stack on or on heap) by the
> application. So the application requires to access to the full
> structure.
> Objects (instances) are allocated by Python (on the heap).
> Applications usually don't need to know/access the structure.
> Python is far from being perfect, static types are still supported and
> they are an issue for the stable ABI.
> > The PyConfig
> > and PyPreConfig structs are no different from PyObject in that regard:
> > if there's a size mismatch, then the developers of the embedding
> > application have somehow messed up their build process.
> > In short, PyConfig initialization works like that:
> 
> The application allocates a PyConfig object on the stack
> Python calls memset(config, 0, sizeof(PyConfig))
> 
> If there is a size mismatch, Python triggers a buffer overflow which
> is likely to cause issues.
> I prefer to have a clean API which makes buffer overflow impossible.
> Embedding Python and handling different Python versions is not
> trivial, especially if we start to add new fields to each PyConfig
> (which is very likely). If prefer to be extra careful.
> I also expect bad surprises even in CPython with Programs/_testembed:
> many tests use PyConfig. Depending if _testembed is properly rebuilt
> or not, bad thing will happen.
> --
> To implement my PEP 445 "Add new APIs to customize Python memory
> allocators", I added a PyMemAllocator structure which is part of the
> API.
> Quickly, I had to add a new field to PyMemAllocator.  But it wasn't
> possible to detect if a C extension used the old or the new
> structure... So I decided to rename the structure to PyMemAllocatorEx
> to ensure that the compilation of all C extensions using the API will
> fail... :-(
> I really dislike this solution. What will happen when we will add
> another field to the structure, like a new PyMem_Aligned() (similar to
> posix_memalign()) function? PyMem_Aligned() can be implementation on
> top of an existing memory allocator which doesn't support it natively.
> B

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Victor Stinner]

Hi back,

It seems like "config.struct_size = sizeof(PyConfig);" syntax is "not
really liked".

I don't see any traction to provide a stable ABI for embedded Python.
The consensus is more towards: "it doesn't work and we don't want to
bother with false promises" (nor add code for that).

Since Lukasz is supposed to tag 3.8.0rc1 today, I propose to remove
anything related to "stable API" or "stable ABI" from the PEP 587
right now. I wrote two PRs to remove struct_size:

* CPython: https://github.com/python/cpython/pull/16500
* PEP 587: https://github.com/python/peps/pull/1185

Later, if we decide to start proving a stable ABI for embedded Python,
we can still add a "version" or "struct_size" field to PyConfig later
(for example in Python 3.9).

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/2JAJQA5OANFPXAJ3327RRPHPQLKVP2EW/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Mon., 30 Sep. 2019, 7:43 pm Petr Viktorin,  wrote:

> On 2019-09-30 00:33, Nick Coghlan wrote:
> [...]
> > We'll never change the size of the config structs (or any other public
> > struct) in a maintenance branch
>
> But we *will* change it in alphas/betas, and then we will ask people to
> try their software out with these.
> Of course, there are no actual API/ABI guarantees with alphas/betas:
> officially, you need to rebuild everything for each new pre-release. But
> with the goal being to find bugs rather than provide correct software,
> it's OK to take shortcuts (if you can then distinguish the bugs the
> shortcuts cause from actual bugs in the code).
>
> Issuing proper version mismatch errors (instead of random cryptic
> segfaults) could make for a much nicer experience when testing several
> successive alphas/betas.
>

That's actually the variant I implemented in my PR (release candidates and
final releases allow nominally compatible inexact matches, everything else
requires an exact version match).

Cheers,
Nick.

___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/LEXOES2PE634F5JJQJGXFTFATX3CY3PA/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/WAHKL7GVBGFNGTO25BVJLDTRLUDLKDXX/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Tue., 1 Oct. 2019, 3:05 am Nick Coghlan,  wrote:

>
> The only outcome I'd consider undesirable is shipping a public API
> that's more awkward to use than it needs to be, doesn't conform to the
> accepted version of the PEP, and doesn't protect against most of the
> potential sources of segfaults arising from an ABI mismatch (and could
> even cause them if the supplied "struct_size" value were to be trusted
> directly, rather than being treated as a lookup value into a set of
> known-valid struct sizes. The currently checked in code at least
> doesn't do that, but that's only because it ignores the new
> struct_size field entirely when making the memset() calls to
> initialise the config structs).
>

My apologies, the "only" here is incorrect, as the existing code also has
guards to make sure that the passed in struct size is exactly the same as
the interpreter's compile time size (it's effectively a lookup set with one
member).

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IHRN6EFQ3OVPCJVDUIQRU4GVKUA3SYQC/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Mon, 30 Sep 2019 at 23:50, Victor Stinner  wrote:
>
> Le lun. 30 sept. 2019 à 13:45, Nick Coghlan  a écrit :
> > > I understand that your main motivation to use the Python version
> > > number rather than sizeof(PyConfig) is the error message.
> >
> > No, my main motivation is to create an API that can emit a useful
> > error message on *ALL* version conflicts between an embedding
> > application and the embedded runtime, not just version conflicts
> > involving versions that change the size of the config structs.
> >
> > The latter option is a good one if all we want to version is the
> > struct itself, but I don't think that's what we really want here: I
> > think we want to version check the entire API/ABI.
>
> IMHO this discussion is going too far away from the PEP 587 goals. The
> PEP 587 is unrelated to the stable API or the stable ABI.

I would be entirely happy with reverting to PEP 587 as originally
accepted (no public struct_size field, no hidden struct versioning
field), rather than making last minute changes to it right before the
first release candidate.

PyConfig and PyPreConfig aren't covered by the stable ABI, so there's
no current need for them to be able to evolve gracefully across ABI
version bumps. If we *do* decide to evolve them in a direction that
allows for more explicit ABI version checks, then doing that as a
separate PyVersionedConfig and PyVersionedPreConfig API would have the
virtue of keeping the simple case simple, while offering a slightly
more awkward to use, but also more forward compatible API for the
benefits of folks that wanted to use it.

However, *if* we were to add a versioning scheme for 3.8.0rc1, then
https://github.com/python/cpython/pull/16496/files covers what I think
it should look like (a header_version field that gets set to
PY_VERSION_HEX by the embedding application).

The only outcome I'd consider undesirable is shipping a public API
that's more awkward to use than it needs to be, doesn't conform to the
accepted version of the PEP, and doesn't protect against most of the
potential sources of segfaults arising from an ABI mismatch (and could
even cause them if the supplied "struct_size" value were to be trusted
directly, rather than being treated as a lookup value into a set of
known-valid struct sizes. The currently checked in code at least
doesn't do that, but that's only because it ignores the new
struct_size field entirely when making the memset() calls to
initialise the config structs).

> I suggest to develop a new use case and a new solution to handle this
> case. The solution can be *very different* than "compare two versions
> at startup".
>
> For example, we may need a CI to ensure that we don't break the stable ABI:
> https://pythoncapi.readthedocs.io/stable_abi.html#check-for-abi-changes
>
> Another solution is to write a completely new C API from scratch. For
> example, HPy ("PyHandle") project proposes to write a new API
> implemented on top of the existing API:
> https://github.com/pyhandle/hpy
>
> This project is even developed outside CPython. So the solution can be
> found *outside* CPython as well.

These are far outside the bounds of "Don't segfault when an embedding
application tries to do something that's obviously broken", which I
consider the only reason to even consider changing the public API away
from what was accepted in PEP 587 in the 3.8rc1 time frame.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UXRKYP3YCAY2K2GD3QLCGZGQ45WMKORZ/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Antoine Pitrou]

I agree that passing a struct size as struct member values sounds a bit
unidiomatic.

Also, it doesn't achieve ABI stability, just allows erroring out in
case the user links with the wrong Python version.

Regards

Antoine.


On Sun, 29 Sep 2019 16:47:41 +1000
Nick Coghlan  wrote:
> On Sat, 28 Sep 2019 at 12:56, Victor Stinner  wrote:
> >
> > Hi,
> >
> > I dislike having to do that, but I had to make a last minute change in
> > my PEP 587 "Python Initialization Configuration" to allow to modify
> > the structure in the future without breaking the backward
> > compatibility. I added a "struct_size" field to PyPreConfig and
> > PyConfig:
> >
> > * https://bugs.python.org/issue38304
> > * 
> > https://github.com/python/peps/commit/afa38c0bef7738b8fcc3173daee8488e0420833d
> >
> > The example:
> >
> >PyConfig config;
> >PyStatus status = PyConfig_InitPythonConfig(&config);
> >...
> >
> > must now be written:
> >
> >PyConfig config;
> >config.struct_size = sizeof(PyConfig);
> >PyStatus status = PyConfig_InitPythonConfig(&config);
> >...
> >
> > At the beginning, I used a private "_config_version" field which was
> > initialized statically by a macro. But it was decided to replace
> > macros with functions. I only noticed today that the conversion to
> > function broke the API/ABI future compatibility.
> >
> > PyConfig_InitPythonConfig() got uninitialized memory and didn't know
> > the size of the config variable.  
> 
> I don't quite understand the purpose of this change, as there's no
> stable ABI for applications embedding CPython. As a result, updating
> to a new X.Y.0 release always requires rebuilding the entire
> application, not just building and relinking CPython.
> 
> I could understand a change to require passing in an expected Python
> version so we can fail more gracefully on a bad link where an
> application that intended to embed Python 3.8 is incorrectly linked
> against Python 3.9 (for example), but performing that kind of check
> would require passing in PY_VERSION_HEX, not the size of the config
> struct.
> 
> > With my change, the function now requires the "struct_size" field to
> > be set, and so it can support internally different versions of the
> > PyConfig structure.
> >
> > Storing the structure size directly in the structure is a common
> > practice in the Windows API which is a good example of long term ABI
> > compatibility.  
> 
> This analogy doesn't hold, as Microsoft explicitly support running old
> binaries on new versions of Windows, and hence need a way to determine
> which version of the structure is being passed in.
> 
> We don't support that - all our APIs that accept
> PyObject/PyTypeObject/etc require that the caller pass in structs of
> the correct size for the version of Python being used. The PyConfig
> and PyPreConfig structs are no different from PyObject in that regard:
> if there's a size mismatch, then the developers of the embedding
> application have somehow messed up their build process.
> 
> The stable ABI is a different story, but that's why we try very hard
> to avoid exposing any structs in the stable ABI.
> 
> Cheers,
> Nick.
> 


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZDZ7DFZZIUYGQJOGZ3ONTTCRYGZ6KVCS/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Victor Stinner]

Le lun. 30 sept. 2019 à 13:45, Nick Coghlan  a écrit :
> > I understand that your main motivation to use the Python version
> > number rather than sizeof(PyConfig) is the error message.
>
> No, my main motivation is to create an API that can emit a useful
> error message on *ALL* version conflicts between an embedding
> application and the embedded runtime, not just version conflicts
> involving versions that change the size of the config structs.
>
> The latter option is a good one if all we want to version is the
> struct itself, but I don't think that's what we really want here: I
> think we want to version check the entire API/ABI.

IMHO this discussion is going too far away from the PEP 587 goals. The
PEP 587 is unrelated to the stable API or the stable ABI.

I suggest to develop a new use case and a new solution to handle this
case. The solution can be *very different* than "compare two versions
at startup".

For example, we may need a CI to ensure that we don't break the stable ABI:
https://pythoncapi.readthedocs.io/stable_abi.html#check-for-abi-changes

Another solution is to write a completely new C API from scratch. For
example, HPy ("PyHandle") project proposes to write a new API
implemented on top of the existing API:
https://github.com/pyhandle/hpy

This project is even developed outside CPython. So the solution can be
found *outside* CPython as well.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UTAGQ3GIP3OZBLE4LWCI2GVDCDAVRPTF/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Mon, 30 Sep 2019 at 09:37, Victor Stinner  wrote:
>
> Le lun. 30 sept. 2019 à 00:33, Nick Coghlan  a écrit :
> > As noted above, despite what I wrote on BPO, you no longer need to persuade 
> > me that the version check is desirable, only that a narrow check on 
> > specific struct sizes is preferable to a broad check on the expected API 
> > version.
>
> I understand that your main motivation to use the Python version
> number rather than sizeof(PyConfig) is the error message.

No, my main motivation is to create an API that can emit a useful
error message on *ALL* version conflicts between an embedding
application and the embedded runtime, not just version conflicts
involving versions that change the size of the config structs.

The latter option is a good one if all we want to version is the
struct itself, but I don't think that's what we really want here: I
think we want to version check the entire API/ABI.

The improved error message is just a nice bonus, as is the fact that I
think the following code is more self-explanatory than the size
checks:

   PyConfig config;
   config.header_version = PY_VERSION_HEX;
   PyStatus status = PyConfig_InitPythonConfig(&config);

The documentation would say something like "header_version should
specify the version of the CPython headers that was used to build and
link the embedding application. The simplest way to set it correctly
is to set it to ``PY_VERSION_HEX``".

> If we implement support for older PyConfig ("stable ABI"), you will
> simply never see this error: it will just work transparently.

But that's true regardless of whether the consistency check is based
on the struct size or the header version.

> IMHO the current error message is good enough:
>
> if (config->struct_size != sizeof(PyConfig)) {
> return _PyStatus_ERR("unsupported PyConfig structure size "
>  "(Python version mismatch?)");
> }

That error message won't trigger if the public API struct that changed
size is PyTypeObject or PyObject (or any of the other structs in the
full API/ABI), whereas a PY_VERSION_HEX based check can be made to
fail based on any policy we choose to enforce.

For example, we could make the runtime enforce the following pair of rules:

* if both the header version and runtime version are final releases or
release candidates, then only the major and minor version fields need
to match (the X.Y in X.Y.Z)
* if either the header version or the runtime version is an alpha or
beta release (or an unknown release type), then the entire version
number must match exactly

Or we could be more permissive, and only ever enforce the first rule,
even for alpha and beta releases.

> I wrote a proof-of-concept to check if it would be doable to support
> multiple versions (sizes) of PyConfig: it's doable and it's quite easy
> to implement, a few lines of code. For example, support Python 3.8
> PyConfig in Python 3.9.

That doesn't change all that much with a Py_VERSION_HEX based check -
it just needs an extra table for range based lookups to find the
relevant struct size for the given version, together with keeping
definitions of the old config structs around.

typedef struct {
uint32_t firstVersion;
Py_ssize_t structSize;
} _Py_versionToSize;

static _Py_versionToSize _PyConfigStructSizes {
{0x0309, sizeof(_PyConfig_3_9)},
{0, sizeof(_PyConfig_3_8)}
}

If the given version matched, you wouldn't need to do the size lookup.

You also wouldn't be constrained to having the lookup table be from
versions to struct sizes - it could instead be from versions to
function pointers, or some other such thing, based on whatever made
the most sense given how the rest of the config system worked in any
given release.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RL2TEQ6BEGOHSWPPUVML5SLNIBVGFJ5G/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Petr Viktorin]

On 2019-09-30 00:33, Nick Coghlan wrote:
[...]
We'll never change the size of the config structs (or any other public 
struct) in a maintenance branch


But we *will* change it in alphas/betas, and then we will ask people to 
try their software out with these.
Of course, there are no actual API/ABI guarantees with alphas/betas: 
officially, you need to rebuild everything for each new pre-release. But 
with the goal being to find bugs rather than provide correct software, 
it's OK to take shortcuts (if you can then distinguish the bugs the 
shortcuts cause from actual bugs in the code).


Issuing proper version mismatch errors (instead of random cryptic 
segfaults) could make for a much nicer experience when testing several 
successive alphas/betas.

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LEXOES2PE634F5JJQJGXFTFATX3CY3PA/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Eric V. Smith]

On 9/29/2019 5:13 PM, Victor Stinner wrote:

It seems simpler to me to pass the structure size rather than the
Python version. It avoids the risk of updating the structure without
update the Python version. I also avoids to have to change the Python
version immediately when PyConfig is modified. 


In Win32, Microsoft does this a lot. For example, 
https://docs.microsoft.com/en-us/windows/win32/api/winuser/ns-winuser-wndclassexa



The main risk of
sizeof(PyConfig) comes if we *remove* a field and add a new field of
the same size: the structure size doesn't change... But in my
experience, we only add new ways to configure Pyhon, we never remove
old ones :-D


I agree this is unlikely.

Eric
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J2NYQCXYNNTSG7HBXWXZDZH7K7BMLMKT/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Victor Stinner]

Le lun. 30 sept. 2019 à 00:33, Nick Coghlan  a écrit :
> As noted above, despite what I wrote on BPO, you no longer need to persuade 
> me that the version check is desirable, only that a narrow check on specific 
> struct sizes is preferable to a broad check on the expected API version.

I understand that your main motivation to use the Python version
number rather than sizeof(PyConfig) is the error message.

If we implement support for older PyConfig ("stable ABI"), you will
simply never see this error: it will just work transparently.

IMHO the current error message is good enough:

if (config->struct_size != sizeof(PyConfig)) {
return _PyStatus_ERR("unsupported PyConfig structure size "
 "(Python version mismatch?)");
}

I wrote a proof-of-concept to check if it would be doable to support
multiple versions (sizes) of PyConfig: it's doable and it's quite easy
to implement, a few lines of code. For example, support Python 3.8
PyConfig in Python 3.9.

--

https://bugs.python.org/issue2506 "Add mechanism to disable
optimizations" (-X noopt) would be good first candidate to modify
PyConfig in Python 3.9: https://github.com/python/cpython/pull/13600

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GOD2HMHTBLSVOGSHMLY3WJLOZTKZPI7L/

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Mon., 30 Sep. 2019, 7:13 am Victor Stinner,  wrote:

> Hi Nick,
>
> Le dim. 29 sept. 2019 à 08:47, Nick Coghlan  a écrit :
> > I don't quite understand the purpose of this change, as there's no
> > stable ABI for applications embedding CPython.
>
> Well, I would like to prepare Python to provide a stable ABI for
> embedded Python. While it's not a design goal yet
> (Include/cpython/initconfig.h is currently excluded from
> Py_LIMITED_API), this change is a step towards that.
>
>
> > As a result, updating
> > to a new X.Y.0 release always requires rebuilding the entire
> > application, not just building and relinking CPython.
>
> In Python 3.8, C extensions are no longer linked to libpython which
> allows to switch between a release build and a debug build of
> libpython.
>
> Can we imagine the same idea for embedded Python? I checked vim on
> Linux: it's linked to libpython3.7m.so.1.0: a specific Python version,
> library built in release mode.
>

Switching between ABI compatible debug 3.8 and release 3.8 builds isn't the
same as allowing switching between ABI incompatible release 3.8 and release
3.9 builds.



>
> > I could understand a change to require passing in an expected Python
> > version so we can fail more gracefully on a bad link where an
> > application that intended to embed Python 3.8 is incorrectly linked
> > against Python 3.9 (for example), but performing that kind of check
> > would require passing in PY_VERSION_HEX, not the size of the config
> > struct.
>
> It seems simpler to me to pass the structure size rather than the
> Python version. It avoids the risk of updating the structure without
> update the Python version. I also avoids to have to change the Python
> version immediately when PyConfig is modified.


We already change the Python version as soon as the maintenance branch gets
created (master is 3.9.0a0, and has been pretty much since 3.8.0b1).

The main risk of
> sizeof(PyConfig) comes if we *remove* a field and add a new field of
> the same size: the structure size doesn't change... But in my
> experience, we only add new ways to configure Pyhon, we never remove
> old ones :-D
>

The main risk I see is some *other* unversioned struct in the full C ABI
changing size.

If the config APIs are only checking for config struct size changes, then
changes to anything else will still segfault.

If they're instead checking "What version was the calling application built
against?" then we can decide how to handle it on the interpreter side (e.g.
require that the "X.Y" part of the version match, and report an init error
otherwise).



> The question is if it's convenient to compute sizeof(PyConfig) in
> programming languages other than C. Providing a "structure version" or
> the structure size from a function call would not work. The value must
> be known a compilation time, not at runtime. The purpose is to compare
> the version/size between build and runtime (they must match).
>

You can also compare the build time value of a public integer macro to
detect build discrepancies.

"sizeof(some_struct)" is just a straightforward way to define such an
integer in a way that will always change when a new field is added to a
particular struct.


> In the first implementation of my PEP, I used an internal "config
> version" provides by a macro. But it was said that macros are not
> convenient.
>

The initialisation macros aren't necessarily convenient (e.g. if you're
using a C++ struct rather than a C one). That's an issue with requiring
C-style initialisation, though, and less with macros in general.

That said, I think the change to make the expected API/ABI version an
explicit part of the config API rather than a hidden part of the struct is
a good idea, since it lets us replace cryptic segfaults with explicit
"Python version mismatch" errors.


> PY_VERSION_HEX is provided as a macro, but we are now trying to avoid
> macros in our C API, no? At least, it's what I understood from the PEP
> 587 discussion.
>

The issue with macros is that their behaviour gets locked in at build time,
so you can't fix bugs in them or otherwise change their behaviour just by
linking against a new version. Instead, you have to recompile the consumer
application or module in addition to recompiling the API provider.

In this case though, that's exactly what we want, as the whole point would
be to detect cases where the CPython runtime library had been recompiled,
but the embedding application hadn't (or vice-versa).


>
> > We don't support that - all our APIs that accept
> > PyObject/PyTypeObject/etc require that the caller pass in structs of
> > the correct size for the version of Python being used.
>
> For PyConfig, it's allocated (on the stack on or on heap) by the
> application. So the application requires to access to the full
> structure.


> Objects (instances) are allocated by Python (on the heap).
> Applications usually don't need to know/access the structure.


> Python is far from being perfect, s

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Victor Stinner]

Hi Nick,

Le dim. 29 sept. 2019 à 08:47, Nick Coghlan  a écrit :
> I don't quite understand the purpose of this change, as there's no
> stable ABI for applications embedding CPython.

Well, I would like to prepare Python to provide a stable ABI for
embedded Python. While it's not a design goal yet
(Include/cpython/initconfig.h is currently excluded from
Py_LIMITED_API), this change is a step towards that.


> As a result, updating
> to a new X.Y.0 release always requires rebuilding the entire
> application, not just building and relinking CPython.

In Python 3.8, C extensions are no longer linked to libpython which
allows to switch between a release build and a debug build of
libpython.

Can we imagine the same idea for embedded Python? I checked vim on
Linux: it's linked to libpython3.7m.so.1.0: a specific Python version,
library built in release mode.


> I could understand a change to require passing in an expected Python
> version so we can fail more gracefully on a bad link where an
> application that intended to embed Python 3.8 is incorrectly linked
> against Python 3.9 (for example), but performing that kind of check
> would require passing in PY_VERSION_HEX, not the size of the config
> struct.

It seems simpler to me to pass the structure size rather than the
Python version. It avoids the risk of updating the structure without
update the Python version. I also avoids to have to change the Python
version immediately when PyConfig is modified. The main risk of
sizeof(PyConfig) comes if we *remove* a field and add a new field of
the same size: the structure size doesn't change... But in my
experience, we only add new ways to configure Pyhon, we never remove
old ones :-D

The question is if it's convenient to compute sizeof(PyConfig) in
programming languages other than C. Providing a "structure version" or
the structure size from a function call would not work. The value must
be known a compilation time, not at runtime. The purpose is to compare
the version/size between build and runtime (they must match).

In the first implementation of my PEP, I used an internal "config
version" provides by a macro. But it was said that macros are not
convenient.

PY_VERSION_HEX is provided as a macro, but we are now trying to avoid
macros in our C API, no? At least, it's what I understood from the PEP
587 discussion.


> We don't support that - all our APIs that accept
> PyObject/PyTypeObject/etc require that the caller pass in structs of
> the correct size for the version of Python being used.

For PyConfig, it's allocated (on the stack on or on heap) by the
application. So the application requires to access to the full
structure.

Objects (instances) are allocated by Python (on the heap).
Applications usually don't need to know/access the structure.

Python is far from being perfect, static types are still supported and
they are an issue for the stable ABI.


> The PyConfig
> and PyPreConfig structs are no different from PyObject in that regard:
> if there's a size mismatch, then the developers of the embedding
> application have somehow messed up their build process.

In short, PyConfig initialization works like that:

* The application allocates a PyConfig object on the stack
* Python calls memset(config, 0, sizeof(PyConfig))

If there is a size mismatch, Python triggers a buffer overflow which
is likely to cause issues.

I prefer to have a clean API which makes buffer overflow impossible.

Embedding Python and handling different Python versions is not
trivial, especially if we start to add new fields to each PyConfig
(which is very likely). If prefer to be extra careful.

I also expect bad surprises even in CPython with Programs/_testembed:
many tests use PyConfig. Depending if _testembed is properly rebuilt
or not, bad thing will happen.

--

To implement my PEP 445 "Add new APIs to customize Python memory
allocators", I added a PyMemAllocator structure which is part of the
API.

Quickly, I had to add a new field to PyMemAllocator.  But it wasn't
possible to detect if a C extension used the old or the new
structure... So I decided to rename the structure to PyMemAllocatorEx
to ensure that the compilation of all C extensions using the API will
fail... :-(

I really dislike this solution. What will happen when we will add
another field to the structure, like a new PyMem_Aligned() (similar to
posix_memalign()) function? PyMem_Aligned() can be implementation on
top of an existing memory allocator which doesn't support it natively.
But the problem is again the API and the PyMemAllocatorEx structure...

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZE5K7MEKN32YGHNA5457AROWUN

[Python-Dev] Re: PEP 587 (Python Initialization Configuration) updated to be future proof again

[Nick Coghlan]

On Sat, 28 Sep 2019 at 12:56, Victor Stinner  wrote:
>
> Hi,
>
> I dislike having to do that, but I had to make a last minute change in
> my PEP 587 "Python Initialization Configuration" to allow to modify
> the structure in the future without breaking the backward
> compatibility. I added a "struct_size" field to PyPreConfig and
> PyConfig:
>
> * https://bugs.python.org/issue38304
> * 
> https://github.com/python/peps/commit/afa38c0bef7738b8fcc3173daee8488e0420833d
>
> The example:
>
>PyConfig config;
>PyStatus status = PyConfig_InitPythonConfig(&config);
>...
>
> must now be written:
>
>PyConfig config;
>config.struct_size = sizeof(PyConfig);
>PyStatus status = PyConfig_InitPythonConfig(&config);
>...
>
> At the beginning, I used a private "_config_version" field which was
> initialized statically by a macro. But it was decided to replace
> macros with functions. I only noticed today that the conversion to
> function broke the API/ABI future compatibility.
>
> PyConfig_InitPythonConfig() got uninitialized memory and didn't know
> the size of the config variable.

I don't quite understand the purpose of this change, as there's no
stable ABI for applications embedding CPython. As a result, updating
to a new X.Y.0 release always requires rebuilding the entire
application, not just building and relinking CPython.

I could understand a change to require passing in an expected Python
version so we can fail more gracefully on a bad link where an
application that intended to embed Python 3.8 is incorrectly linked
against Python 3.9 (for example), but performing that kind of check
would require passing in PY_VERSION_HEX, not the size of the config
struct.

> With my change, the function now requires the "struct_size" field to
> be set, and so it can support internally different versions of the
> PyConfig structure.
>
> Storing the structure size directly in the structure is a common
> practice in the Windows API which is a good example of long term ABI
> compatibility.

This analogy doesn't hold, as Microsoft explicitly support running old
binaries on new versions of Windows, and hence need a way to determine
which version of the structure is being passed in.

We don't support that - all our APIs that accept
PyObject/PyTypeObject/etc require that the caller pass in structs of
the correct size for the version of Python being used. The PyConfig
and PyPreConfig structs are no different from PyObject in that regard:
if there's a size mismatch, then the developers of the embedding
application have somehow messed up their build process.

The stable ABI is a different story, but that's why we try very hard
to avoid exposing any structs in the stable ABI.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncogh...@gmail.com   |   Brisbane, Australia
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/UHJDLWFHSZ6D7WCHCFAVVBDBK52AS3PL/