[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Fri, Apr 22, 2022 at 2:52 PM Fabio Zadrozny  wrote:
> Humm, now I'm a bit worried... the approach the debugger is using gets the 
> PyFrameObject that's about to be executed and changes the 
> PyFrameObject.f_code just before the execution so that the new code is 
> executed instead.

You can already modify _PyInterpreterFrame.f_code using the internal C API.

> From what you're saying the PyFrameObject isn't really used anymore 
> (apparently it's substituted by a _PyInterpreterFrame?)... in this case, will 
> this approach still let the debugger patch the code object in the frame 
> before it's actually executed?

There is no public C API to modify the "f_code" attribute of a
PyFrameObject. There is only PyFrame_GetCode() *getter*.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RTBUDAEPTZBTAHEO5LV77MHEH7URJP3J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 22. 04. 22 14:47, Fabio Zadrozny wrote:



Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin > escreveu:


Hello Fabio,
Let's talk a bit about which API should, exactly, be guaranteed to not
change across minor releases.
So far it looks like:
- PyEval_RequestCodeExtraIndex
- PyCode_GetExtra
- PyCode_SetExtra
- PyFrameEvalFunction
- PyInterpreterState_GetEvalFrameFunc
- PyInterpreterState_SetEvalFrameFunc

Do any more come to mind?

The issue with this set is that in 3.11, _PyFrameEvalFunction changes
its signature to take _PyInterpreterFrame rather than PyFrameObject.
Exposing _PyInterpreterFrame would be quite problematic. For example,
since it's not a PyObject, it has its own lifetime management that's
controlled by the interpreter itself,. And it includes several
pointers whose lifetime and semantics also isn't guaranteed (they
might be borrowed, cached or filled on demand). I don't think we can
make any guarantees on these, so the info needs to be accessed using
getter functions.

There is the function _PyFrame_GetFrameObject, which returns a
PyFrameObject.
I think it would be best to only expose _PyInterpreterFrame as an
opaque structure, and expose PyFrame_GetFrameObject so debuggers can
get a PyFrameObject from it.
Does that sound reasonable?



Humm, now I'm a bit worried... the approach the debugger is using gets 
the PyFrameObject that's about to be executed and changes the 
PyFrameObject.f_code just before the execution so that the new code is 
executed instead.


 From what you're saying the PyFrameObject isn't really used anymore 
(apparently it's substituted by a _PyInterpreterFrame?)... in this case, 
will this approach still let the debugger patch the code object in the 
frame before it's actually executed?


PyFrameObject is a fairly thin wrapper around _PyInterpreterFrame -- it 
adds PyObject metadata (type & refcount), and not much else. It's 
allocated at most once for each _PyInterpreterFrame -- once it's created 
it stays attached to the frame.
So, for the most heavily optimized code paths a PyFrameObject is not 
allocated, but it's trivial to get it whenever it's needed.


-- i.e.: the debugger changes the state.interp.eval_frame to its own 
custom evaluation function, but _PyEval_EvalFrameDefault is still what 
ends up being called afterwards (it works more as a hook to change the 
PyFrameObject.f_code prior to execution than as an alternate interpreter).


Ah, you also need PyEval_EvalFrameDefault exposed. The public version 
would take PyFrameObject and pass its _PyInterpreterFrame to the 
internal _PyEval_EvalFrameDefault.




On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny mailto:fabi...@gmail.com>> wrote:
 >
 >
 > Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny
mailto:fabi...@gmail.com>> escreveu:
 >>>
 >>> PEP 523 API added more private functions for code objects:
 >>>
 >>> * _PyEval_RequestCodeExtraIndex()
 >>> * _PyCode_GetExtra()
 >>> * _PyCode_SetExtra()
 >>>
 >>> The _PyEval_RequestCodeExtraIndex() function seems to be used
by the
 >>> pydevd debugger. The two others seem to be unused in the wild.
I'm not
 >>> sure if these ones should be moved to the internal C API. They
can be
 >>> left unchanged, since they don't use a type only defined by the
 >>> internal C API.
 >>
 >> Just to note, the pydevd/debugpy debuggers actually uses all of
those APIs.
 >>
 >> i.e.:
 >>
 >>

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187


 >>

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232


 >>

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311


 >>
 >> The debugger already has workarounds because of changes to
evaluation api over time (see:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491

)
and I know 3.11 won't be different.
 >>
 >> I'm ok with changes as I understand that this is a special API
-- as long as there's still a way to use it and get the information
needed (the debugger already goes through many hops because it needs
to use many internals of CPython -- in every new rele

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Fabio Zadrozny]

Em sex., 22 de abr. de 2022 às 09:02, Petr Viktorin 
escreveu:

> Hello Fabio,
> Let's talk a bit about which API should, exactly, be guaranteed to not
> change across minor releases.
> So far it looks like:
> - PyEval_RequestCodeExtraIndex
> - PyCode_GetExtra
> - PyCode_SetExtra
> - PyFrameEvalFunction
> - PyInterpreterState_GetEvalFrameFunc
> - PyInterpreterState_SetEvalFrameFunc
>
> Do any more come to mind?
>
> The issue with this set is that in 3.11, _PyFrameEvalFunction changes
> its signature to take _PyInterpreterFrame rather than PyFrameObject.
> Exposing _PyInterpreterFrame would be quite problematic. For example,
> since it's not a PyObject, it has its own lifetime management that's
> controlled by the interpreter itself,. And it includes several
> pointers whose lifetime and semantics also isn't guaranteed (they
> might be borrowed, cached or filled on demand). I don't think we can
> make any guarantees on these, so the info needs to be accessed using
> getter functions.
>
> There is the function _PyFrame_GetFrameObject, which returns a
> PyFrameObject.
> I think it would be best to only expose _PyInterpreterFrame as an
> opaque structure, and expose PyFrame_GetFrameObject so debuggers can
> get a PyFrameObject from it.
> Does that sound reasonable?
>


Humm, now I'm a bit worried... the approach the debugger is using gets the
PyFrameObject that's about to be executed and changes the
PyFrameObject.f_code just before the execution so that the new code is
executed instead.

>From what you're saying the PyFrameObject isn't really used anymore
(apparently it's substituted by a _PyInterpreterFrame?)... in this case,
will this approach still let the debugger patch the code object in the
frame before it's actually executed?

-- i.e.: the debugger changes the state.interp.eval_frame to its own custom
evaluation function, but _PyEval_EvalFrameDefault is still what ends up
being called afterwards (it works more as a hook to change the
PyFrameObject.f_code prior to execution than as an alternate interpreter).



On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny  wrote:
> >
> >
> > Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny 
> escreveu:
> >>>
> >>> PEP 523 API added more private functions for code objects:
> >>>
> >>> * _PyEval_RequestCodeExtraIndex()
> >>> * _PyCode_GetExtra()
> >>> * _PyCode_SetExtra()
> >>>
> >>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> >>> pydevd debugger. The two others seem to be unused in the wild. I'm not
> >>> sure if these ones should be moved to the internal C API. They can be
> >>> left unchanged, since they don't use a type only defined by the
> >>> internal C API.
> >>
> >> Just to note, the pydevd/debugpy debuggers actually uses all of those
> APIs.
> >>
> >> i.e.:
> >>
> >>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
> >>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
> >>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
> >>
> >> The debugger already has workarounds because of changes to evaluation
> api over time (see:
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
> and I know 3.11 won't be different.
> >>
> >> I'm ok with changes as I understand that this is a special API -- as
> long as there's still a way to use it and get the information needed (the
> debugger already goes through many hops because it needs to use many
> internals of CPython -- in every new release it's a **really** big task to
> update to the latest version as almost everything that the debugger relies
> to make debugging fast changes across versions and I never really know if
> it'll be possible to support it until I really try to do the port -- I
> appreciate having less things in a public API so it's easier to have
> extensions work in other interpreters/not recompiling on newer versions,
> but please keep it possible to use private APIs which provides the same
> access that CPython has to access things internally for special cases such
> as the debugger).
> >>
> >> Maybe later on that PEP from mark which allows a better debugger API
> could alleviate that (but until then, if possible I appreciate it if
> there's some effort not to break things unless really needed -- ideally
> with instructions on how to port).
> >>
> >> Anyways, to wrap up, the debugger already needs to be built with
> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API
> (as long as it's still accessible in that case) is probably not a big issue
> for the debugger and having setters/getters to set it instead of relying on
> `state.interp.eval_frame` seems good to me.
> >>
> >> Cheers,
> >>
> >> Fabio
> >>
> >
> >
> > I think the main issue here is the compatibility across the same version
> though... is it po

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

Hello Fabio,
Let's talk a bit about which API should, exactly, be guaranteed to not
change across minor releases.
So far it looks like:
- PyEval_RequestCodeExtraIndex
- PyCode_GetExtra
- PyCode_SetExtra
- PyFrameEvalFunction
- PyInterpreterState_GetEvalFrameFunc
- PyInterpreterState_SetEvalFrameFunc

Do any more come to mind?

The issue with this set is that in 3.11, _PyFrameEvalFunction changes
its signature to take _PyInterpreterFrame rather than PyFrameObject.
Exposing _PyInterpreterFrame would be quite problematic. For example,
since it's not a PyObject, it has its own lifetime management that's
controlled by the interpreter itself,. And it includes several
pointers whose lifetime and semantics also isn't guaranteed (they
might be borrowed, cached or filled on demand). I don't think we can
make any guarantees on these, so the info needs to be accessed using
getter functions.

There is the function _PyFrame_GetFrameObject, which returns a PyFrameObject.
I think it would be best to only expose _PyInterpreterFrame as an
opaque structure, and expose PyFrame_GetFrameObject so debuggers can
get a PyFrameObject from it.
Does that sound reasonable?


On Thu, Mar 24, 2022 at 8:13 PM Fabio Zadrozny  wrote:
>
>
> Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny  
> escreveu:
>>>
>>> PEP 523 API added more private functions for code objects:
>>>
>>> * _PyEval_RequestCodeExtraIndex()
>>> * _PyCode_GetExtra()
>>> * _PyCode_SetExtra()
>>>
>>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
>>> pydevd debugger. The two others seem to be unused in the wild. I'm not
>>> sure if these ones should be moved to the internal C API. They can be
>>> left unchanged, since they don't use a type only defined by the
>>> internal C API.
>>
>> Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.
>>
>> i.e.:
>>
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
>>
>> The debugger already has workarounds because of changes to evaluation api 
>> over time (see: 
>> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
>>  and I know 3.11 won't be different.
>>
>> I'm ok with changes as I understand that this is a special API -- as long as 
>> there's still a way to use it and get the information needed (the debugger 
>> already goes through many hops because it needs to use many internals of 
>> CPython -- in every new release it's a **really** big task to update to the 
>> latest version as almost everything that the debugger relies to make 
>> debugging fast changes across versions and I never really know if it'll be 
>> possible to support it until I really try to do the port -- I appreciate 
>> having less things in a public API so it's easier to have extensions work in 
>> other interpreters/not recompiling on newer versions, but please keep it 
>> possible to use private APIs which provides the same access that CPython has 
>> to access things internally for special cases such as the debugger).
>>
>> Maybe later on that PEP from mark which allows a better debugger API could 
>> alleviate that (but until then, if possible I appreciate it if there's some 
>> effort not to break things unless really needed -- ideally with instructions 
>> on how to port).
>>
>> Anyways, to wrap up, the debugger already needs to be built with 
>> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API (as 
>> long as it's still accessible in that case) is probably not a big issue for 
>> the debugger and having setters/getters to set it instead of relying on 
>> `state.interp.eval_frame` seems good to me.
>>
>> Cheers,
>>
>> Fabio
>>
>
>
> I think the main issue here is the compatibility across the same version 
> though... is it possible to have some kind of guarantee on private APIs that 
> something won't change across micro-releases?
>
> I.e.: having the frame evaluation function change across major releases and 
> having them be reworked seems reasonable, but then having the frame 
> evaluation be changed across micro-releases wouldn't be.
>
> So, I'm ok in pushing things to the internal API, but then I still would like 
> guarantees about the compatibility of that API in the same major release 
> (otherwise those setters/getters/frame evaluation should probably remain on 
> the public API if the related structure was moved to the internal API).
>
> Cheers,
>
> Fabio
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/a

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

Here's the issue with the plan (including Nick's suggestions):
https://github.com/python/cpython/issues/91744

On Sun, Apr 10, 2022 at 5:43 AM Nick Coghlan  wrote:
>
> On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin,  wrote:
>>
>> So here's my proposal:
>>
>> - This API stays with the regular public API (Include/cpython/), but to
>> use it you'll need to #define Py_USING_UNSTABLE_API (name up for
>> bikeshedding).
>
>
> I'm fine with the rest of what you suggest, but I don't think this is the 
> right mechanical approach:
>
> * "unstable" is the wrong term. We already have an unstable API tier: the 
> internal API, which can change even in maintenance releases. The value of the 
> new tier is that it is "semi stable": stable in maintenance releases, 
> unstable in feature releases.
> * the lesson I take from our stable ABI experience is that mixing two tiers 
> of the API in a single header file is hard to maintain, as it's too easy to 
> add a new API to the wrong section. A separate file that gets included 
> automatically from the relevant header file(s) when the new definition is 
> used makes the split much clearer.
>
> Cheers,
> Nick.
>
>
>>
>>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGQN7HOPLAV6FDVXGLBTXOXSAQ75XM2S/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin,  wrote:

>
> This applies to:
>
> - Functions added in PEP 523
> - PyCode_New, PyCode_NewWithPosOnlyArgs
> - Ideally anything documented as subject to change between minor
> releases. (To be kind to users, if something is added later we should
> again have one release of compiler warnings before requiring the opt-in.
> Unless that API just changed and users would get errors anyway.)
>


Other candidate items for this tier:


* non-opaque access to frame structs and any other key APIs needed to
implement alternate eval loops with comparable performance to the default
eval loop (unless & until we can figure out stable public APIs that can
deliver equivalent performance)

> * C APIs that provide access to compiled code whether in AST or opcode
form (the API itself may be stable, but the compiled code isn't, so this is
kinda covered by your last point)

Cheers,
Nick.


>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CW543RWJU6EWAPTWCDDHEUS37UHSAU5I/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Thu, 7 Apr 2022, 8:02 pm Petr Viktorin,  wrote:

> So here's my proposal:
>
> - This API stays with the regular public API (Include/cpython/), but to
> use it you'll need to #define Py_USING_UNSTABLE_API (name up for
> bikeshedding).
>

I'm fine with the rest of what you suggest, but I don't think this is the
right mechanical approach:

* "unstable" is the wrong term. We already have an unstable API tier: the
internal API, which can change even in maintenance releases. The value of
the new tier is that it is "semi stable": stable in maintenance releases,
unstable in feature releases.
* the lesson I take from our stable ABI experience is that mixing two tiers
of the API in a single header file is hard to maintain, as it's too easy to
add a new API to the wrong section. A separate file that gets included
automatically from the relevant header file(s) when the new definition is
used makes the split much clearer.

Cheers,
Nick.



>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CTKKTHUV5R2A2RRN5DM32UQFNC42DDGJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 07. 04. 22 17:10, Victor Stinner wrote:

On Thu, Apr 7, 2022 at 12:02 PM Petr Viktorin  wrote:

- This API stays with the regular public API (Include/cpython/), but to
use it you'll need to #define Py_USING_UNSTABLE_API (name up for
bikeshedding).


Since there is already something similar called "Py_LIMITED", I
suggest dropping "USING_" for just: "Py_UNSTABLE_API".


But I really like the explicit “USING” :)
I don't think there's too much value in these two being consistent.
(Yay, bikeshedding!)



- The functions will be renamed to drop the leading underscore. The old
names will be available as aliases (using #define) and may be removed
whenever the API changes. (Ideally, the underscore should always mark
API that's fully private with no guarantees at all.)


Should functions entering the "unstable API" be documented and tested?

For example, _PyEval_EvalFrameDefault() and
_PyInterpreterState_SetEvalFrameFunc() have no test nor doc.


Yes.
I'll add docs, for tests I'll at least open an issue.



This applies to:

- PyCode_New, PyCode_NewWithPosOnlyArgs


It would be nice to update Cython to define the Py_UNSTABLE_API macro
before the macro is required to get the function, since Cython still
uses PyCode_New().


That (for any project, not just Cython) is why I propose warnings in 
3.11, before requiring the opt-in in 3.12.




Should we deprecate types.CodeType constructor in the Python API,
since types.CodeType.replace() exists and seems to be a better API
("more stable")?


I don't know. But it's for a different discussion.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MBDJH4J5SA2GJENCBWTHA53KURPX2S4J/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Thu, Apr 7, 2022 at 12:02 PM Petr Viktorin  wrote:
> - This API stays with the regular public API (Include/cpython/), but to
> use it you'll need to #define Py_USING_UNSTABLE_API (name up for
> bikeshedding).

Since there is already something similar called "Py_LIMITED", I
suggest dropping "USING_" for just: "Py_UNSTABLE_API".


> - The functions will be renamed to drop the leading underscore. The old
> names will be available as aliases (using #define) and may be removed
> whenever the API changes. (Ideally, the underscore should always mark
> API that's fully private with no guarantees at all.)

Should functions entering the "unstable API" be documented and tested?

For example, _PyEval_EvalFrameDefault() and
_PyInterpreterState_SetEvalFrameFunc() have no test nor doc.


> This applies to:
>
> - PyCode_New, PyCode_NewWithPosOnlyArgs

It would be nice to update Cython to define the Py_UNSTABLE_API macro
before the macro is required to get the function, since Cython still
uses PyCode_New().

Should we deprecate types.CodeType constructor in the Python API,
since types.CodeType.replace() exists and seems to be a better API
("more stable")?

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/Q6W6G4DNC4JB4YWRCEY32RWZHIAKJSU5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

So here's my proposal:

- This API stays with the regular public API (Include/cpython/), but to 
use it you'll need to #define Py_USING_UNSTABLE_API (name up for 
bikeshedding).


- Since we're nearing Beta and there's no rush to break things, in 3.11 
you only get a warning if you try to use it without the opt-in #define. 
In 3.12 it'll fail.


- The functions will be renamed to drop the leading underscore. The old 
names will be available as aliases (using #define) and may be removed 
whenever the API changes. (Ideally, the underscore should always mark 
API that's fully private with no guarantees at all.)


- The API will be stable during a minor release. (As usual, for extreme 
cases, exceptions are possible with SC approval.)


- Docs will be updated:
  - https://devguide.python.org/c-api/
  - Individual reference entries for the API and the new opt-in macro


This applies to:

- Functions added in PEP 523
- PyCode_New, PyCode_NewWithPosOnlyArgs
- Ideally anything documented as subject to change between minor 
releases. (To be kind to users, if something is added later we should 
again have one release of compiler warnings before requiring the opt-in. 
Unless that API just changed and users would get errors anyway.)



(Technically, this proposal needs SC approval -- PEP 387 exception for 
PyCode_New*. I'll play by the rules, of course.)




On 06. 04. 22 17:21, Nick Coghlan wrote:



On Wed, 6 Apr 2022, 7:05 am Victor Stinner, > wrote:


On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan mailto:ncogh...@gmail.com>> wrote:
 > The changes you've made have been excellent, and the existing 3
categories (stable public ABI, stable public API, unstable internal
API) cover the vast majority of cases.
 >
 > The final case that isn't quite covered yet is to offer a
"semi-stable" API category for use cases that are intrinsically
coupled to implementation details that may change between feature
releases, but should remain stable within a release series.
 >
 > The concrete motivating example for the new category is the extra
APIs you need in order to provide an alternative eval loop
implementation.
 >
 > The internal API category doesn't properly cover that case, as
the APIs there are free to change even in maintenance releases, and
setting Py_BUILD_CORE exposes a lot more than what an alternative
eval loop would need.
 >
 > Regular public functions may work in some cases, but aren't
necessarily practical in others (such as exposing the internal frame
details for use in alternative eval loops).
 >
 > From an implementation PoV, my own suggestion would be to define
a new API tier with an opt-in macro rather than relying solely on
documentation or naming conventions.
 >
 > For example, define "Py_SEMI_STABLE_API" to opt in, with the
headers under "Include/cpython/semi_stable/" (I don't like
"unstable" as potential terminology here, since the internal API is
already unstable - we're splitting the difference between that and
the long term stability of the full public API)

For me an API is either stable (remains the same forever) or unstable
(change time to time).

Public API means: stable, documented, tested.

Internal API means: unstable, not documented, not tested.

I'm not convinced that it's worth it to create something in the
middle. If you want to add doc and tests, it should become a public
stable API.


The middle semi-stable tier formalises a concept that we already have: 
no guarantees across feature releases, but both API and ABI stable 
within a release series.


It's useful for tightly coupled projects like Cython, as it means they 
can get core level performance without the risk of API compatibility 
breaks in maintenance releases.


Without defining this tier, effectively the *entire* internal API 
becomes semi-stable, as any changes made will risk breaking the third 
party projects that we've told to define Py_BUILD_CORE when compiling.


Cheers,
Nick.




___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7P3ENW56DFR5BOUFSOQUSXPFQNQ5MF56/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ELBMRSGLBJ6LBMA4DMEOFZ2LQCWAGTGP/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Wed, 6 Apr 2022, 7:05 am Victor Stinner,  wrote:

> On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan  wrote:
> > The changes you've made have been excellent, and the existing 3
> categories (stable public ABI, stable public API, unstable internal API)
> cover the vast majority of cases.
> >
> > The final case that isn't quite covered yet is to offer a "semi-stable"
> API category for use cases that are intrinsically coupled to implementation
> details that may change between feature releases, but should remain stable
> within a release series.
> >
> > The concrete motivating example for the new category is the extra APIs
> you need in order to provide an alternative eval loop implementation.
> >
> > The internal API category doesn't properly cover that case, as the APIs
> there are free to change even in maintenance releases, and setting
> Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
> need.
> >
> > Regular public functions may work in some cases, but aren't necessarily
> practical in others (such as exposing the internal frame details for use in
> alternative eval loops).
> >
> > From an implementation PoV, my own suggestion would be to define a new
> API tier with an opt-in macro rather than relying solely on documentation
> or naming conventions.
> >
> > For example, define "Py_SEMI_STABLE_API" to opt in, with the headers
> under "Include/cpython/semi_stable/" (I don't like "unstable" as potential
> terminology here, since the internal API is already unstable - we're
> splitting the difference between that and the long term stability of the
> full public API)
>
> For me an API is either stable (remains the same forever) or unstable
> (change time to time).
>
> Public API means: stable, documented, tested.
>
> Internal API means: unstable, not documented, not tested.
>
> I'm not convinced that it's worth it to create something in the
> middle. If you want to add doc and tests, it should become a public
> stable API.
>

The middle semi-stable tier formalises a concept that we already have: no
guarantees across feature releases, but both API and ABI stable within a
release series.

It's useful for tightly coupled projects like Cython, as it means they can
get core level performance without the risk of API compatibility breaks in
maintenance releases.

Without defining this tier, effectively the *entire* internal API becomes
semi-stable, as any changes made will risk breaking the third party
projects that we've told to define Py_BUILD_CORE when compiling.

Cheers,
Nick.



>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/7P3ENW56DFR5BOUFSOQUSXPFQNQ5MF56/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 4/5/2022 11:52 PM, Victor Stinner wrote:

On Fri, Apr 1, 2022 at 12:36 PM Steve Dower  wrote:

I don't see any additional discussion on the bug, and the prevailing
opinion from actual users of this API is that it probably shouldn't
change,


 From what I understood my change basically only impacts "pydevd"
users. Fabio who works on that project said that he was fine with
that change ("I'm ok with changes"):
https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55ANVKHGG74D62HDBOFLC4EXWJ26/


"I'm okay with" isn't really a sign of support, particularly in this 
kind of relationship where *we* have all the power. Fabio doesn't have a 
choice but to be okay with whatever we decide, and he obviously accepts 
that.


I'm still going to strongly advocate for not going out of our way to 
break users like Fabio. Just because we have a reputation for doing it 
doesn't mean we should keep doing it.



For me, all impacted users were made aware and joined the discussion.
It's very rare to be able to reach *all* users impacted by an
incompatible C API change!


There's no way we reached all the users :) But we did reach more than usual.


On Fri, Apr 1, 2022 at 12:36 PM Steve Dower  wrote:

and certainly shouldn't become internal without additional
guarantees about stability.


The discussed API is not stable, Brett repeated that. The internal API
is not stable on purpose. I missed the discussion proposing to design
a stable API for PEP 523.


I can't tell whether you missed the discussion or if you're deliberately 
misrepresenting it as a persuasive technique.


The API may not be "stable" according to our typical rules around public 
C APIs, but it has explicit rules about its own stability (well, had, 
though once the revert goes in it will have them again).


Nobody suggested making it *more* stable than it already was. We just 
want to treat its existing stability rules with respect and either 
transition out of them slowly, or leave them as they are.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/226B54YXCYX4JNYTVGNSOVN6ONJZX7PC/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

Hi,

Steve, Petr: sorry if you feel that I didn't take your feedback in
account, it's not case. Thanks for your valuable feedback. It seems
like there was some misunderstanding.


On Tue, Apr 5, 2022 at 2:49 AM Gregory P. Smith  wrote:
> Thanks for bringing this up on python-dev, Victor. That was good. But the 
> point of the discussion should've been to continue working with people based 
> on the replies rather than proceeding to merge removals of the APIs after 
> people said they used them.  (echoing Steve and Petr here...)
>
> We discussed this on the steering council today. These APIs were in a weird 
> state and despite past decisions at the time of PEP-523 in 2016 they should 
> be treated as public-ish rather than entirely private. Because we published a 
> document saying "here they are, use them!" and multiple projects have done so 
> to good effect.
>
> For 3.11 we'd like those PRs reverted.

Ok, I created https://github.com/python/cpython/pull/32343 to revert
these two PRs. I will merge it as soon as the Python 3.11 release
manager (Pablo) unblocks the main branch (he is currently preparing
the alpha7 release).


On Fri, Apr 1, 2022 at 12:36 PM Steve Dower  wrote:
> I don't see any additional discussion on the bug, and the prevailing
> opinion from actual users of this API is that it probably shouldn't
> change,

>From what I understood my change basically only impacts "pydevd"
users. Fabio who works on that project said that he was fine with
that change ("I'm ok with changes"):
https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55ANVKHGG74D62HDBOFLC4EXWJ26/

debugpy, ptvsd, PyDev, PyCharm and VSCode Python use the same code
base: https://github.com/fabioz/PyDev.Debugger

Jason Ansel of TorchDynamo was worried that the API couldn't be used
anymore. I replied that the API remains available. It's just about
adding a few lines of code (that I provided).

For me, all impacted users were made aware and joined the discussion.
It's very rare to be able to reach *all* users impacted by an
incompatible C API change!

If I understood correctly, all questions were replied. For example,
yes, the API remains accessible (with additional code), but no, sadly
the API is not stable (can change at each 3.x.0 major release, but
should not change in 3.x.y bugfix release).


On Fri, Apr 1, 2022 at 12:36 PM Steve Dower  wrote:
> and certainly shouldn't become internal without additional
> guarantees about stability.

The discussed API is not stable, Brett repeated that. The internal API
is not stable on purpose. I missed the discussion proposing to design
a stable API for PEP 523.

PEP 523 users are that the API is unstable and seem to be used to
update their code at each major (3.x) Python releases. I'm not
surprised that a debugger which requires a fast low-level access to
Python internals require that.

Or are you talking about not breaking the API in 3.x.y bugfix
releases? Currently, it's an unwritten rule. IMO it's well respected
by all core devs who understand that it's important to not break any
API, including private and internal APIs, in minor bugfix releases. My
changes are unrelated to that.


On Fri, Apr 1, 2022 at 1:24 PM Petr Viktorin  wrote:
> Now, the people who'd like a non-breaking solution will now need rush to
> develop and merge one until the next release, or the API breaks for
> users and it'll be too late to do anything about it.

We are talking about two projects (pydevd and TorchDynamo) and there
are 6 months until Python 3.11 final release (PEP 664). We are still
at the alpha phase, no? I don't see why a change only affecting two
projects is a big deal, knowing that these two projects have been made
aware, and I offered a patch that they can apply right now. They don't
need to wait for next October to apply my patch.


Anyway, I will revert my changes (once the main branch is unblocked)
to apply the SC's decision.

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TUKA6R7VFIKAWBA2XA7QRNXKGKXDH3WG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Sun, Apr 3, 2022 at 3:29 PM Nick Coghlan  wrote:
> The changes you've made have been excellent, and the existing 3 categories 
> (stable public ABI, stable public API, unstable internal API) cover the vast 
> majority of cases.
>
> The final case that isn't quite covered yet is to offer a "semi-stable" API 
> category for use cases that are intrinsically coupled to implementation 
> details that may change between feature releases, but should remain stable 
> within a release series.
>
> The concrete motivating example for the new category is the extra APIs you 
> need in order to provide an alternative eval loop implementation.
>
> The internal API category doesn't properly cover that case, as the APIs there 
> are free to change even in maintenance releases, and setting Py_BUILD_CORE 
> exposes a lot more than what an alternative eval loop would need.
>
> Regular public functions may work in some cases, but aren't necessarily 
> practical in others (such as exposing the internal frame details for use in 
> alternative eval loops).
>
> From an implementation PoV, my own suggestion would be to define a new API 
> tier with an opt-in macro rather than relying solely on documentation or 
> naming conventions.
>
> For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under 
> "Include/cpython/semi_stable/" (I don't like "unstable" as potential 
> terminology here, since the internal API is already unstable - we're 
> splitting the difference between that and the long term stability of the full 
> public API)

For me an API is either stable (remains the same forever) or unstable
(change time to time).

Public API means: stable, documented, tested.

Internal API means: unstable, not documented, not tested.

I'm not convinced that it's worth it to create something in the
middle. If you want to add doc and tests, it should become a public
stable API.

For example, IMO PyCode_New() (C API) and types.CodeType constructor
(Python API) should be moved to the internal C API, likely with a
deprecation period. Cython should not use it but a new stable API.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/LFE4OV3NYWR4GBMCISZ3H7JH3SEFMX2E/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Gregory P. Smith]

On Fri, Apr 1, 2022 at 2:06 AM Victor Stinner  wrote:

> Hi,
>
> Update on this issue: I merged my 2 PRs.
> https://bugs.python.org/issue46850
>
> The following APIs have been moved to the internal C API:
>
> - _PyFrameEvalFunction type
> - _PyInterpreterState_GetEvalFrameFunc()
> - _PyInterpreterState_SetEvalFrameFunc()
> - _PyEval_EvalFrameDefault()
>
> If you use any of these API in your debugger/profiler project, you
> have do add something like the code below to your project:
> ---
> #ifndef Py_BUILD_CORE_MODULE
> #  define Py_BUILD_CORE_MODULE
> #endif
> #include 
> #if PY_VERSION_HEX >= 0x030B00A7
> #  include  //
> _PyInterpreterState_SetEvalFrameFunc()
> #  include   // _PyEval_EvalFrameDefault()
> #endif
> ---
>
> Contact me if you need help to update your affected projects.
>
> IMO PEP 523 doesn't have to be updated since it already says that the
> APIs are private.
>

Thanks for bringing this up on python-dev, Victor. That was good. But the
point of the discussion should've been to continue working with people
based on the replies rather than proceeding to merge removals of the APIs
after people said they used them.  (echoing Steve and Petr here...)

We discussed this on the steering council today. These APIs were in a weird
state and despite past decisions at the time of PEP-523
 in 2016 they should be treated as
public-ish rather than entirely private. Because we published a document
saying "here they are, use them!" and multiple projects have done so to
good effect.

For 3.11 we'd like those PRs reverted.  We see the following as the better
way forward for these APIs:

Add a new #define that can be set before the #include  that
exposes non-limited but stable within a bugfix/patch releases APIs (ie:
Petr's earlier suggestion).
These would be the first to fall within. To do so we should give these,
behind that #define, non _-prefixed "public style" names as these are
quasi-public and cannot be changed as readily as other true internals.  We
still, per the PEP, reserve the right to turn these into no-op potentially
warning setting APIs in a future release (likely 3.12?) as they are at
least documented as being unstable/private in the PEP.

So in 3.11 these should continue to exist as in 3.6-3.10:
- _PyFrameEvalFunction type
- _PyInterpreterState_GetEvalFrameFunc()
- _PyInterpreterState_SetEvalFrameFunc()
- _PyEval_EvalFrameDefault()

AND in 3.11:
 - #define protected versions of those without the leading _ become
available.
 - (i'm intentionally not suggesting a #define name, y'all can pick
something)

In 3.12:
 - the _ prefixed versions can go away.  People using the APIs should've
updated their code to use the new #define and new names when building
against >=3.11 by then.
 - Whether the APIs continue to be as useful and act as originally claimed
in PEP 523 is up to the 3.12 implementors (out of scope for this thread).
They occupy a newly defined middle ground between the "forever" style
limited API and the "can break even on bugfix/patch release" internal API
that wasn't a concept for us in 2016 when PEP 523 was written.

Why? Being conservative with things in active use that weren't *really*
private, and similar to what Mark Shannon and Petr said, we *do not* want
people to #define Py_BUILD_CORE_MODULE and start poking at internals in
arbitrary ways. That exposes a whole pile of other things for (ab)use that
are even more unstable. Avoiding that helps avoid temptation to go wild and
helps us identify users.

-gps (steering council hat on)


> Since these APIs were added by PEP 523, I documented these changes in
> What's New in Python 3.11 > C API > Porting to Python 3.11,even if
> these APIs are private.
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/DNJC6U36CDA7S7ATEGAMUPABBSEIYHC4/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GFOMU7LP63JUVFMWNJNZJLUMZDRPTUYJ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Itamar O]

On Sun, Apr 3, 2022 at 6:32 AM Nick Coghlan  wrote:

> On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:
>
>> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum 
>> wrote:
>>
>> I'm not convinced that advertising an API as being Unstable (in the
>> documentation?) is going to solve any kind of problem. People love to
>> use private APIs, and they do it simply because it's technically
>> possible :-) At the end of the day, we have to help them updating
>> their code, otherwise we (at least my Red Hat team) cannot update
>> Python.
>>
>> I designed the internal C API to be more annoying to be used (need to
>> define a macro, need more specific #include) in the hope that people
>> will think twice before using it :-)
>>
>
>
> The changes you've made have been excellent, and the existing 3 categories
> (stable public ABI, stable public API, unstable internal API) cover the
> vast majority of cases.
>
> The final case that isn't quite covered yet is to offer a "semi-stable"
> API category for use cases that are intrinsically coupled to implementation
> details that may change between feature releases, but should remain stable
> within a release series.
>
> The concrete motivating example for the new category is the extra APIs you
> need in order to provide an alternative eval loop implementation.
>
> The internal API category doesn't properly cover that case, as the APIs
> there are free to change even in maintenance releases, and setting
> Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
> need.
>
> Regular public functions may work in some cases, but aren't necessarily
> practical in others (such as exposing the internal frame details for use in
> alternative eval loops).
>
> From an implementation PoV, my own suggestion would be to define a new API
> tier with an opt-in macro rather than relying solely on documentation or
> naming conventions.
>
> For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
> "Include/cpython/semi_stable/" (I don't like "unstable" as potential
> terminology here, since the internal API is already unstable - we're
> splitting the difference between that and the long term stability of the
> full public API)
>
> Cheers,
> Nick.
>

+1 for an official "semi stable API tier".
It's already the case that essentially anything in Python can change,
it's just a question of how quickly and with how much friction.
Public APIs need to go through a multi-version deprecation cycle
(https://peps.python.org/pep-0387/#making-incompatible-changes).
Private internal APIs can (theoretically) change without notice between
patch releases.
There's a missing tier for APIs that can change without notice between
feature releases,
but are guaranteed(*) to be backwards compatible within a feature release,
and the PEP 523 frame evaluation API is an excellent example for this need
(maybe any newly added API should always go through this stage for a few
releases?).

Even though the docs (
https://docs.python.org/3.10/c-api/intro.html#include-files) explicitly
call out that _Py-prefixed APIs are internal and should not be used by
extensions,
this isn't the case in practice, so introducing the 3-tier concept could be
an opportunity
to clean up this situation a bit.
What exactly should be the naming conventions per tier, and the names of
the tiers,
is bikeshedding that should happen after there's agreement about the tiers
concept :-)

(*) "guaranteed" with exceptions of course (e.g. security or other critical
issue)


>
>
>
>> Victor
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZBNJTAXS6TWQY7QH5H5XZS4CP64ZQAUU/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DQ5AUS3UBQGJV6YLQ4CJ5F5M7RWFC7DY/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Guido van Rossum]

+1 to Nick's analysis and proposal. I had been mulling over my own reply
but this just about covers it.

On Sun, Apr 3, 2022 at 6:29 AM Nick Coghlan  wrote:

> On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:
>
>> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum 
>> wrote:
>>
>> I'm not convinced that advertising an API as being Unstable (in the
>> documentation?) is going to solve any kind of problem. People love to
>> use private APIs, and they do it simply because it's technically
>> possible :-) At the end of the day, we have to help them updating
>> their code, otherwise we (at least my Red Hat team) cannot update
>> Python.
>>
>> I designed the internal C API to be more annoying to be used (need to
>> define a macro, need more specific #include) in the hope that people
>> will think twice before using it :-)
>>
>
>
> The changes you've made have been excellent, and the existing 3 categories
> (stable public ABI, stable public API, unstable internal API) cover the
> vast majority of cases.
>
> The final case that isn't quite covered yet is to offer a "semi-stable"
> API category for use cases that are intrinsically coupled to implementation
> details that may change between feature releases, but should remain stable
> within a release series.
>
> The concrete motivating example for the new category is the extra APIs you
> need in order to provide an alternative eval loop implementation.
>
> The internal API category doesn't properly cover that case, as the APIs
> there are free to change even in maintenance releases, and setting
> Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
> need.
>
> Regular public functions may work in some cases, but aren't necessarily
> practical in others (such as exposing the internal frame details for use in
> alternative eval loops).
>
> From an implementation PoV, my own suggestion would be to define a new API
> tier with an opt-in macro rather than relying solely on documentation or
> naming conventions.
>
> For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
> "Include/cpython/semi_stable/" (I don't like "unstable" as potential
> terminology here, since the internal API is already unstable - we're
> splitting the difference between that and the long term stability of the
> full public API)
>
> Cheers,
> Nick.
>
>
>
>> Victor
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TBWSFAGJNCDEVNK4VSQTEUKYX6FRBUYA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Nick Coghlan]

On Fri, 1 Apr 2022, 6:47 pm Victor Stinner,  wrote:

> On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum  wrote:
>
> I'm not convinced that advertising an API as being Unstable (in the
> documentation?) is going to solve any kind of problem. People love to
> use private APIs, and they do it simply because it's technically
> possible :-) At the end of the day, we have to help them updating
> their code, otherwise we (at least my Red Hat team) cannot update
> Python.
>
> I designed the internal C API to be more annoying to be used (need to
> define a macro, need more specific #include) in the hope that people
> will think twice before using it :-)
>


The changes you've made have been excellent, and the existing 3 categories
(stable public ABI, stable public API, unstable internal API) cover the
vast majority of cases.

The final case that isn't quite covered yet is to offer a "semi-stable" API
category for use cases that are intrinsically coupled to implementation
details that may change between feature releases, but should remain stable
within a release series.

The concrete motivating example for the new category is the extra APIs you
need in order to provide an alternative eval loop implementation.

The internal API category doesn't properly cover that case, as the APIs
there are free to change even in maintenance releases, and setting
Py_BUILD_CORE exposes a lot more than what an alternative eval loop would
need.

Regular public functions may work in some cases, but aren't necessarily
practical in others (such as exposing the internal frame details for use in
alternative eval loops).

>From an implementation PoV, my own suggestion would be to define a new API
tier with an opt-in macro rather than relying solely on documentation or
naming conventions.

For example, define "Py_SEMI_STABLE_API" to opt in, with the headers under
"Include/cpython/semi_stable/" (I don't like "unstable" as potential
terminology here, since the internal API is already unstable - we're
splitting the difference between that and the long term stability of the
full public API)

Cheers,
Nick.



> Victor
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZBNJTAXS6TWQY7QH5H5XZS4CP64ZQAUU/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Mark Shannon]

Hi,

On 22/03/2022 6:07 pm, Victor Stinner wrote:

Hi,

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):

* https://github.com/python/cpython/pull/32052
* https://github.com/python/cpython/pull/32054


I just want to say that it is important the API is easy to use and access.

Otherwise there is a temptation to directly set the underlying function 
pointer, and that means we don't get to see when PEP 523 is used.
We might then mis-optimize on the assumption that PEP 523 isn't used.

Cheers,
Mark.



API:

* _PyFrameEvalFunction type
* _PyInterpreterState_GetEvalFrameFunc()
* _PyInterpreterState_SetEvalFrameFunc()
* (undocumented) _PyEval_EvalFrameDefault()

The private API to get/set the eval function *is* documented at:
https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc

I added the Get/Set functions so debuggers don't have to access
directly to the PyInterpreterState structure which has been moved to
the internal C API in Python 3.8.

This API causes me multiple issues:

* It's a private API and I'm trying to remove the private API from the
public C API header files.
* The _PyFrameEvalFunction type is not stable: it got a new "tstate"
parameter in Python 3.9 and the type of the second parameter changed
from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
* These functions use the _PyInterpreterFrame type which is part of
the internal C API.

While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.

I propose to move theses API to the internal header files
(Include/internals/) to clarify that it's not part of the public C API
and that there is no backward compatibility warranty.

The change is being discussed at:
https://bugs.python.org/issue46850

--

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild. I'm not
sure if these ones should be moved to the internal C API. They can be
left unchanged, since they don't use a type only defined by the
internal C API.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/L2E2HLUT757ROREKWIAD7M2HIHNUJ3VE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 01. 04. 22 11:01, Victor Stinner wrote:

Hi,

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850

The following APIs have been moved to the internal C API:

- _PyFrameEvalFunction type
- _PyInterpreterState_GetEvalFrameFunc()
- _PyInterpreterState_SetEvalFrameFunc()
- _PyEval_EvalFrameDefault()


Really?
I haven't seen any support for that in this thread or on the issue, 
except from you.
Now, the people who'd like a non-breaking solution will now need rush to 
develop and merge one until the next release, or the API breaks for 
users and it'll be too late to do anything about it. Meanwhile, you're 
off to make more changes. That is, frankly, very frustrating.


Could you please stop unilaterally breaking documented API?



If you use any of these API in your debugger/profiler project, you
have do add something like the code below to your project:
---
#ifndef Py_BUILD_CORE_MODULE
#  define Py_BUILD_CORE_MODULE
#endif
#include 
#if PY_VERSION_HEX >= 0x030B00A7
#  include  // _PyInterpreterState_SetEvalFrameFunc()
#  include   // _PyEval_EvalFrameDefault()
#endif
---


IMO, this is a terrible suggestion that undermines the whole point of 
having a private API.




Contact me if you need help to update your affected projects.

IMO PEP 523 doesn't have to be updated since it already says that the
APIs are private.

Since these APIs were added by PEP 523, I documented these changes in
What's New in Python 3.11 > C API > Porting to Python 3.11,even if
these APIs are private.

Victor

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TGBYB2WKOUZUA6LXNLMMLY7PKEQG3FZW/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 4/1/2022 10:01 AM, Victor Stinner wrote:

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850


So what was the point of this discussion then?

I don't see any additional discussion on the bug, and the prevailing 
opinion from actual users of this API is that it probably shouldn't 
change, and certainly shouldn't become internal without additional 
guarantees about stability.


Did we all just waste a whole lot of electrons discussing a foregone 
conclusion?


(My apologies to the people I invited into this thread. I genuinely 
thought it would help to have outside perspective on this potential change.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/IZGWG3KKGGWI7HLYSJUDQVTXFUNGZKD3/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

Hi,

Update on this issue: I merged my 2 PRs.
https://bugs.python.org/issue46850

The following APIs have been moved to the internal C API:

- _PyFrameEvalFunction type
- _PyInterpreterState_GetEvalFrameFunc()
- _PyInterpreterState_SetEvalFrameFunc()
- _PyEval_EvalFrameDefault()

If you use any of these API in your debugger/profiler project, you
have do add something like the code below to your project:
---
#ifndef Py_BUILD_CORE_MODULE
#  define Py_BUILD_CORE_MODULE
#endif
#include 
#if PY_VERSION_HEX >= 0x030B00A7
#  include  // _PyInterpreterState_SetEvalFrameFunc()
#  include   // _PyEval_EvalFrameDefault()
#endif
---

Contact me if you need help to update your affected projects.

IMO PEP 523 doesn't have to be updated since it already says that the
APIs are private.

Since these APIs were added by PEP 523, I documented these changes in
What's New in Python 3.11 > C API > Porting to Python 3.11,even if
these APIs are private.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DNJC6U36CDA7S7ATEGAMUPABBSEIYHC4/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Chris Angelico]

On Fri, 1 Apr 2022 at 19:51, Victor Stinner  wrote:
> In Python, sadly the types.CodeType type also has a public constructor
> and many projects break at each Python release because the API
> changes. Hopefully, it seems like the new CodeType.replace() method
> added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
> better abstraction and closer to what developers need in practice.

It certainly has been for me. When I want to do bytecode hackery, I
usually start by creating a function with def/lambda, then construct a
modified function using f.__code__.replace(). It's the easiest way to
ensure that all the little details are correct.

ChrisA
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JJO6LDWA4BE7SFTEF2VP6NLSIX6ZBCGD/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Wed, Mar 30, 2022 at 5:42 PM Guido van Rossum  wrote:
> In the not so distant past I have proposed to introduce a new category, 
> "Unstable APIs". These are public but are not guaranteed to be backwards 
> compatible in feature releases (though I feel they should remain so in bugfix 
> releases).
>
> I'm not sure whether those should have a leading underscore or not. Perhaps 
> (like some other languages do and like maybe we've used in a few places) the 
> name could just include the word "Unstable"?

I recall discussions about PyCode_New(). IMO this API should not be
public at all. It leaks way too many implementation details: cell
variables, optimization for bytecode offset to line and column
numbers, exception table, etc. This API changed often and will
continue to change.

It's not the right abstraction level. We just exposed the function
because it was technically possible and it was convenient since Python
consumes its own C API. The internal C API was created to draw a line
between what API can be consumed outside Python (public) and what API
must not be used outside Python (internal) unless you're writing a
debugger or other uncommon very specific use case. The main difference
is the warranties provided (public) or not (internal) by Python:
tests, documentation, backward compatibility.

In Python, sadly the types.CodeType type also has a public constructor
and many projects break at each Python release because the API
changes. Hopefully, it seems like the new CodeType.replace() method
added to Python 3.8 mitigated the issue. IMO CodeType.replace() is a
better abstraction and closer to what developers need in practice.

I'm not convinced that advertising an API as being Unstable (in the
documentation?) is going to solve any kind of problem. People love to
use private APIs, and they do it simply because it's technically
possible :-) At the end of the day, we have to help them updating
their code, otherwise we (at least my Red Hat team) cannot update
Python.

I designed the internal C API to be more annoying to be used (need to
define a macro, need more specific #include) in the hope that people
will think twice before using it :-)

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YCHLFQ5KW6XF5C5CFWF4MRTZEXVBZBMA/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Mon, Mar 28, 2022 at 1:44 PM Petr Viktorin  wrote:
> Perhaps we need a new "tier" of C API for debuggers -- API that's
> guaranteed stable for a major release, and if it's changed it should
> always break with compile errors (e.g. the function gets a new
> argument), rather than silently change semantics.
> The internal API Cython & greenlet need might go it this category too.

We should provide public C API functions for Cython and greenlet
needs: well tested and documented functions. There is an on-going work
in Python 3.11:

* https://docs.python.org/dev/c-api/frame.html
* https://github.com/faster-cpython/ideas/issues/309
* https://bugs.python.org/issue40421

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/H64KULVN5O4MXPWWYMO2VBJRTLNNWBYX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Wed, Mar 30, 2022 at 9:26 PM Sebastian Berg
 wrote:
> That is fair, although there are documented underscored ones:
> https://docs.python.org/3/search.html?q=_Py
>
> I suppose that means all bets are off _unless_ it is documented or
> later adopted as stable API (e.g. `_PyObject_Vectorcall`).
>
> With that, the only "not obviously OK" use in NumPy that I am aware of
> is `_Py_HashDouble` (it seems undocumented).
>
> Maybe "unless documented" is just a clear enough distinction in
> practice.
> Although, to some degree, I think it would be clearer if symbols that
> have a realistic chance of changing in bug-fix releases had an
> additional safe-guard.

Since Python 3.7, there is a work-in-progress to (1) better hide
internal C API functions and to (2) promote *private* C API functions
being used by 3rd party projects as documented and well tested
*public* functions.

An example of (2) is the addition of float pack/unpack public
functions, like PyFloat_Pack8() and PyFloat_Unpack8():
https://docs.python.org/dev/c-api/float.html#pack-and-unpack-functions

There were previously known as private _PyFloat_Pack8() and
_PyFloat_Unpack8() functions. They are used by a few serialization
projects like msgpack. When they were made public, tests were added
and the existing comments were converted to documentation and
enhanced.

I discovered that these private functions were used when I moved them
to the internal C API in bpo-46906 (1) and it broke a few projects.

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DT2PUGWPOSZOJZLR4VOMPS6QOS3PDRYR/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Sebastian Berg]

On Wed, 2022-03-30 at 17:51 +0200, Petr Viktorin wrote:
> On 30. 03. 22 17:42, Guido van Rossum wrote:
> > In the not so distant past I have proposed to introduce a new
> > category, 
> > "Unstable APIs". These are public but are not guaranteed to be
> > backwards 
> > compatible in feature releases (though I feel they should remain so
> > in 
> > bugfix releases).
> > 
> > I'm not sure whether those should have a leading underscore or not.
> > Perhaps (like some other languages do and like maybe we've used in
> > a few 
> > places) the name could just include the word "Unstable"?
> 
> IMO, the underscore should mark an API as internal: it can change at
> any 
> time (though in practice it often doesn't, e.g. to accommodate
> projects 
> that used it before a policy is written down).
> 

That is fair, although there are documented underscored ones:
https://docs.python.org/3/search.html?q=_Py

I suppose that means all bets are off _unless_ it is documented or
later adopted as stable API (e.g. `_PyObject_Vectorcall`).

With that, the only "not obviously OK" use in NumPy that I am aware of
is `_Py_HashDouble` (it seems undocumented).

Maybe "unless documented" is just a clear enough distinction in
practice.
Although, to some degree, I think it would be clearer if symbols that
have a realistic chance of changing in bug-fix releases had an
additional safe-guard.

- Sebastian



> This is useful e.g. for macros/static functions that wrap access to 
> something private, where the definition needs to be available but
> marked 
> "keep off".
> 
> 
> > On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner
> >  > > wrote:
> > 
> >     The internal C API can be used on purpose. But there is no
> > backward
> >     compatibility warranty and it can change anytime. In practice,
> > usually
> >     it only changes in 3.x.0 releases. For example, these private C
> > API
> >     changed in Python 3.9 and Python 3.11 (see my first email in
> > the other
> >     PEP 523 thread).
> > 
> >     To use the internal C API, you have to declare the
> > Py_BUILD_CORE macro
> >     and include an internal C API header file. For
> >     _PyInterpreterState_SetEvalFrameFunc(), it should be:
> > 
> >     #ifndef Py_BUILD_CORE_MODULE
> >     #  define Py_BUILD_CORE_MODULE
> >     #endif
> >     #include 
> >     #include  //
> >     _PyInterpreterState_SetEvalFrameFunc()
> >     #include   // _PyEval_EvalFrameDefault
> > 
> >     Victor
> > 
> >     On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
> >     mailto:python-dev@python.org>> wrote:
> >  >
> >  > The PyTorch team plans to use PEP 523 as a part of PyTorch
> > 2.0,
> >     so this proposal may break the next major release of PyTorch.
> >  >
> >  > The related project is TorchDynamo, which can be found here:
> >  > https://github.com/facebookresearch/torchdynamo
> >     
> >  >
> >  > We will likely move this into the core of PyTorch closer to
> > release.
> >  >
> >  > If the changed happens, would PyTorch still be able to use
> > the
> >     eval frame API?  Or would it prevent from being used entirely?
> >  > ___
> >  > Python-Dev mailing list -- python-dev@python.org
> >     
> >  > To unsubscribe send an email to python-dev-le...@python.org
> >     
> >  >
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> >     
> >  > Message archived at
> >    
> > https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> >    
> >  > e/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/>
> >  > Code of Conduct: http://python.org/psf/codeofconduct/
> >     
> > 
> > 
> > 
> >     --
> >     Night gathers, and now my watch begins. It shall not end until
> > my death.
> >     ___
> >     Python-Dev mailing list -- python-dev@python.org
> >     
> >     To unsubscribe send an email to python-dev-le...@python.org
> >     
> >     https://mail.python.org/mailman3/lists/python-dev.python.org/
> >     
> >     Message archived at
> >    
> > https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> >    
> >  > e/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/>
> >     Code of Conduct: http://python.org/psf/codeofconduct/
> >     
> > 
> > 
> > 
> > -- 
> > --Guido van Rossum (python.org/~guido )
> > /Pr

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Guido van Rossum]

I don't think such a guarantee (to not vary internal APIs from 3.x.y to
3.x.(y+1)) has ever before been made in writing, although in practice we've
been doing this (more so now than we were in the 2.x timeframe).

I think we should not lightly vary internal APIs between bugfix releases,
but at the same time I am not sure I want to absolutely guarantee that no
internal API ever changes in a bugfix release (because that might prevent
fixing certain bugs).

So I think we need to officially embrace a category of "unstable public
APIs" and set a policy specifically for those, before we can make progress.

I'd like the SC to take some initiative here.

On Wed, Mar 30, 2022 at 9:34 AM Jason Ansel via Python-Dev <
python-dev@python.org> wrote:

> Got it, thanks for the clarifications!
>
> Tracking 3.x Python versions is fine.  We already need to do that to
> support things like new bytecodes, and PyTorch supports an explicit list of
> 3.x Python versions with separate builds for each.
>
> Tracking 3.x.y Python versions would be much more painful, and make us
> need to rethink our approach.
>
> So what Steve Downer described as "remain compatible within a single 3.x
> release", seems like exactly what we want.  I support that level of
> compatibility guarantee.  Could we keep that guarantee with this change?
>
> Thanks,
> Jason
>
>
>
>
>
> 
> From: Victor Stinner 
> Sent: Wednesday, March 30, 2022 7:56 AM
> To: Steve Dower
> Cc: Jason Ansel; python-dev@python.org
> Subject: Re: [Python-Dev] Re: C API: Move PEP 523 "Adding a frame
> evaluation API to CPython" private C API to the internal C API
>
> On Wed, Mar 30, 2022 at 2:26 AM Steve Dower 
> wrote:
> > Right now, the API is allowed/expected to change between 3.x releases
> > (which normally we don't allow without a deprecation period) but it
> > still has to remain compatible within a single 3.x release. Making it
> > fully internal *without adding a stability guarantee* means it could
> > change more frequently, which you wouldn't be able to handle as an
> > installable package.
> >
> > It's *unlikely* that it'll change that often, because there are still
> > other public interfaces that cannot. But, the plan behind this is to
> > make more stuff internal so that it can be modified more freely, so we
> > may see that rate of change increase.
>
> Well, my plan is not break these internal C API just for fun. It's
> only to better "advertise" the separation between the "stable" public
> C API (backward compatibility warranty) and the "unstable"
> private/internal C API (can change any time, changes are not
> documented).
>
> Victor
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/U7M65SSDZMM7LNAKEDZZ4KKQIFTCOQ2M/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/J3G5RO4FUO3Y3WJ2AJAUMJNS2BXYZ7J7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Jason Ansel via Python-Dev]

Got it, thanks for the clarifications!

Tracking 3.x Python versions is fine.  We already need to do that to support 
things like new bytecodes, and PyTorch supports an explicit list of 3.x Python 
versions with separate builds for each.

Tracking 3.x.y Python versions would be much more painful, and make us need to 
rethink our approach.

So what Steve Downer described as "remain compatible within a single 3.x 
release", seems like exactly what we want.  I support that level of 
compatibility guarantee.  Could we keep that guarantee with this change?

Thanks,
Jason






From: Victor Stinner 
Sent: Wednesday, March 30, 2022 7:56 AM
To: Steve Dower
Cc: Jason Ansel; python-dev@python.org
Subject: Re: [Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation 
API to CPython" private C API to the internal C API

On Wed, Mar 30, 2022 at 2:26 AM Steve Dower  wrote:
> Right now, the API is allowed/expected to change between 3.x releases
> (which normally we don't allow without a deprecation period) but it
> still has to remain compatible within a single 3.x release. Making it
> fully internal *without adding a stability guarantee* means it could
> change more frequently, which you wouldn't be able to handle as an
> installable package.
>
> It's *unlikely* that it'll change that often, because there are still
> other public interfaces that cannot. But, the plan behind this is to
> make more stuff internal so that it can be modified more freely, so we
> may see that rate of change increase.

Well, my plan is not break these internal C API just for fun. It's
only to better "advertise" the separation between the "stable" public
C API (backward compatibility warranty) and the "unstable"
private/internal C API (can change any time, changes are not
documented).

Victor
--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/U7M65SSDZMM7LNAKEDZZ4KKQIFTCOQ2M/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[John Ehresman]

As someone who maintains a debugger that uses private api’s, I’d like to see 
some commitment to seeing them not change in micro releases such as 3.11.1 -> 
3.11.2. Micro releases should be compatible with other micro releases of the 
same major.minor release such as 3.11 so that an extension compiled against 
3.11.1 will work with 3.11.2.

Compatibility was broken (at least) once in a 2.x series (maybe 2.4?) and we 
started getting reports of people using 2.x.0 were having problems. 
Fortunately, in that case compiling against 2.x.0 produced an extension that 
worked against all micro releases.

Note that I’m not asking for private api’s not change in major.micro releases 
such as 3.10 -> 3.11. I know that there can be very good reasons to change 
private api's because that I probably will have work to do in order to support 
a new major.micro release.

I also don’t think that exposing everything that every extension needs with a 
non-private api is a good idea because then the internals will be more 
difficult to change — you’d be signing up to support that api for all future 
major.micro versions until there’s a compatibility break.

John

> On Mar 30, 2022, at 11:01 AM, Victor Stinner  wrote:
> 
> The internal C API can be used on purpose. But there is no backward
> compatibility warranty and it can change anytime. In practice, usually
> it only changes in 3.x.0 releases. For example, these private C API
> changed in Python 3.9 and Python 3.11 (see my first email in the other
> PEP 523 thread).
> 
> To use the internal C API, you have to declare the Py_BUILD_CORE macro
> and include an internal C API header file. For
> _PyInterpreterState_SetEvalFrameFunc(), it should be:
> 
> #ifndef Py_BUILD_CORE_MODULE
> #  define Py_BUILD_CORE_MODULE
> #endif
> #include 
> #include  // _PyInterpreterState_SetEvalFrameFunc()
> #include   // _PyEval_EvalFrameDefault
> 
> Victor
> 
> On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
>  wrote:
>> 
>> The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this 
>> proposal may break the next major release of PyTorch.
>> 
>> The related project is TorchDynamo, which can be found here:
>> https://github.com/facebookresearch/torchdynamo
>> 
>> We will likely move this into the core of PyTorch closer to release.
>> 
>> If the changed happens, would PyTorch still be able to use the eval frame 
>> API?  Or would it prevent from being used entirely?
>> ___
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-le...@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at 
>> https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
>> Code of Conduct: http://python.org/psf/codeofconduct/
> 
> 
> 
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/TK27HEVDUWQQZQ4XIKTQLDHWK52ZLHWN/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/30/2022 4:42 PM, Guido van Rossum wrote:
In the not so distant past I have proposed to introduce a new category, 
"Unstable APIs". These are public but are not guaranteed to be backwards 
compatible in feature releases (though I feel they should remain so in 
bugfix releases).


Agreed. This is definitely a new category, and it seems the only thing 
we're debating now is whether or not to add/omit the underscore.


I'm not sure whether those should have a leading underscore or not. 
Perhaps (like some other languages do and like maybe we've used in a few 
places) the name could just include the word "Unstable"?


I don't think we have "Unstable" anywhere, though we do have "Unsafe" 
(which is more about threading than stability, so not right for this). 
But I'd be okay with that as a compromise.


I'd prefer to not have public-unstable APIs hidden behind the same 
preprocessor directive as internal APIs. That's a big switch to throw 
that may also activate other settings - for example, on Windows we will 
set the minimum Windows version in our headers if you enable internal 
APIs, and disable automatic linking of the Python DLL. Easy enough 
things to work around, but they probably need to be explicitly 
documented as well if we're going to document public APIs as requiring 
Py_BUILD_CORE (and I don't want to have to document that kind of stuff).


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/ZEPJSIJNXQKXTOE2MRNS6GJRP52WLDEF/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 30. 03. 22 17:42, Guido van Rossum wrote:
In the not so distant past I have proposed to introduce a new category, 
"Unstable APIs". These are public but are not guaranteed to be backwards 
compatible in feature releases (though I feel they should remain so in 
bugfix releases).


I'm not sure whether those should have a leading underscore or not. 
Perhaps (like some other languages do and like maybe we've used in a few 
places) the name could just include the word "Unstable"?


IMO, the underscore should mark an API as internal: it can change at any 
time (though in practice it often doesn't, e.g. to accommodate projects 
that used it before a policy is written down).


This is useful e.g. for macros/static functions that wrap access to 
something private, where the definition needs to be available but marked 
"keep off".



On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner > wrote:


The internal C API can be used on purpose. But there is no backward
compatibility warranty and it can change anytime. In practice, usually
it only changes in 3.x.0 releases. For example, these private C API
changed in Python 3.9 and Python 3.11 (see my first email in the other
PEP 523 thread).

To use the internal C API, you have to declare the Py_BUILD_CORE macro
and include an internal C API header file. For
_PyInterpreterState_SetEvalFrameFunc(), it should be:

#ifndef Py_BUILD_CORE_MODULE
#  define Py_BUILD_CORE_MODULE
#endif
#include 
#include  //
_PyInterpreterState_SetEvalFrameFunc()
#include   // _PyEval_EvalFrameDefault

Victor

On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
mailto:python-dev@python.org>> wrote:
 >
 > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0,
so this proposal may break the next major release of PyTorch.
 >
 > The related project is TorchDynamo, which can be found here:
 > https://github.com/facebookresearch/torchdynamo

 >
 > We will likely move this into the core of PyTorch closer to release.
 >
 > If the changed happens, would PyTorch still be able to use the
eval frame API?  Or would it prevent from being used entirely?
 > ___
 > Python-Dev mailing list -- python-dev@python.org

 > To unsubscribe send an email to python-dev-le...@python.org

 > https://mail.python.org/mailman3/lists/python-dev.python.org/

 > Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/


 > Code of Conduct: http://python.org/psf/codeofconduct/




--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org

To unsubscribe send an email to python-dev-le...@python.org

https://mail.python.org/mailman3/lists/python-dev.python.org/

Message archived at

https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/


Code of Conduct: http://python.org/psf/codeofconduct/




--
--Guido van Rossum (python.org/~guido )
/Pronouns: he/him //(why is my pronoun here?)/ 



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EIK3XLPV7A7WVB4FA47PM2G2SJ42RERO/
Code of Conduct: http://python.org/psf/codeofconduct/

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MBHN5AR5C7S3SP3OBSFV3ES26PDM746R/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Guido van Rossum]

In the not so distant past I have proposed to introduce a new category,
"Unstable APIs". These are public but are not guaranteed to be backwards
compatible in feature releases (though I feel they should remain so in
bugfix releases).

I'm not sure whether those should have a leading underscore or not. Perhaps
(like some other languages do and like maybe we've used in a few places)
the name could just include the word "Unstable"?

On Wed, Mar 30, 2022 at 8:08 AM Victor Stinner  wrote:

> The internal C API can be used on purpose. But there is no backward
> compatibility warranty and it can change anytime. In practice, usually
> it only changes in 3.x.0 releases. For example, these private C API
> changed in Python 3.9 and Python 3.11 (see my first email in the other
> PEP 523 thread).
>
> To use the internal C API, you have to declare the Py_BUILD_CORE macro
> and include an internal C API header file. For
> _PyInterpreterState_SetEvalFrameFunc(), it should be:
>
> #ifndef Py_BUILD_CORE_MODULE
> #  define Py_BUILD_CORE_MODULE
> #endif
> #include 
> #include  //
> _PyInterpreterState_SetEvalFrameFunc()
> #include   // _PyEval_EvalFrameDefault
>
> Victor
>
> On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
>  wrote:
> >
> > The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this
> proposal may break the next major release of PyTorch.
> >
> > The related project is TorchDynamo, which can be found here:
> > https://github.com/facebookresearch/torchdynamo
> >
> > We will likely move this into the core of PyTorch closer to release.
> >
> > If the changed happens, would PyTorch still be able to use the eval
> frame API?  Or would it prevent from being used entirely?
> > ___
> > Python-Dev mailing list -- python-dev@python.org
> > To unsubscribe send an email to python-dev-le...@python.org
> > https://mail.python.org/mailman3/lists/python-dev.python.org/
> > Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> > Code of Conduct: http://python.org/psf/codeofconduct/
>
>
>
> --
> Night gathers, and now my watch begins. It shall not end until my death.
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/EIK3XLPV7A7WVB4FA47PM2G2SJ42RERO/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

The internal C API can be used on purpose. But there is no backward
compatibility warranty and it can change anytime. In practice, usually
it only changes in 3.x.0 releases. For example, these private C API
changed in Python 3.9 and Python 3.11 (see my first email in the other
PEP 523 thread).

To use the internal C API, you have to declare the Py_BUILD_CORE macro
and include an internal C API header file. For
_PyInterpreterState_SetEvalFrameFunc(), it should be:

#ifndef Py_BUILD_CORE_MODULE
#  define Py_BUILD_CORE_MODULE
#endif
#include 
#include  // _PyInterpreterState_SetEvalFrameFunc()
#include   // _PyEval_EvalFrameDefault

Victor

On Tue, Mar 29, 2022 at 12:26 AM Jason Ansel via Python-Dev
 wrote:
>
> The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this 
> proposal may break the next major release of PyTorch.
>
> The related project is TorchDynamo, which can be found here:
> https://github.com/facebookresearch/torchdynamo
>
> We will likely move this into the core of PyTorch closer to release.
>
> If the changed happens, would PyTorch still be able to use the eval frame 
> API?  Or would it prevent from being used entirely?
> ___
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-le...@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at 
> https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
> Code of Conduct: http://python.org/psf/codeofconduct/



--
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OQTAF6CQRKHQPYUY5HWVOTUAEXKHI5WE/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Wed, Mar 30, 2022 at 2:26 AM Steve Dower  wrote:
> Right now, the API is allowed/expected to change between 3.x releases
> (which normally we don't allow without a deprecation period) but it
> still has to remain compatible within a single 3.x release. Making it
> fully internal *without adding a stability guarantee* means it could
> change more frequently, which you wouldn't be able to handle as an
> installable package.
>
> It's *unlikely* that it'll change that often, because there are still
> other public interfaces that cannot. But, the plan behind this is to
> make more stuff internal so that it can be modified more freely, so we
> may see that rate of change increase.

Well, my plan is not break these internal C API just for fun. It's
only to better "advertise" the separation between the "stable" public
C API (backward compatibility warranty) and the "unstable"
private/internal C API (can change any time, changes are not
documented).

Victor
-- 
Night gathers, and now my watch begins. It shall not end until my death.
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OMGUI5N33BG3EU4OG3IUZXJQF6XU7X2B/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/28/2022 10:44 PM, Jason Ansel via Python-Dev wrote:

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this 
proposal may break the next major release of PyTorch.

The related project is TorchDynamo, which can be found here:
https://github.com/facebookresearch/torchdynamo

We will likely move this into the core of PyTorch closer to release.

If the changed happens, would PyTorch still be able to use the eval frame API?  
Or would it prevent from being used entirely?


You'd be able to use it, but if we don't nail down the compatibility 
guarantees then you might find PyTorch doesn't work properly against 
3.11.(n+1) while 3.11.n was fine.


Right now, the API is allowed/expected to change between 3.x releases 
(which normally we don't allow without a deprecation period) but it 
still has to remain compatible within a single 3.x release. Making it 
fully internal *without adding a stability guarantee* means it could 
change more frequently, which you wouldn't be able to handle as an 
installable package.


It's *unlikely* that it'll change that often, because there are still 
other public interfaces that cannot. But, the plan behind this is to 
make more stuff internal so that it can be modified more freely, so we 
may see that rate of change increase.


In this world, your best bet is for TorchDynamo to become a full CPython 
fork. And when that's your best bet, it should set off alarms everywhere 
;) But that is what relying on internal APIs implies, and is certainly 
what people would have to do if we were to remove the existing PEP's 
interface. (It's what Pyjion was doing before the PEP was written, and 
it's why Pyjion doesn't have to be a full fork these days.)


So you probably want to state explicit support for either keeping the 
APIs public and slightly-flexible, or making them internal but stable. 
(Public and stable won't work at all for us, and normal internal won't 
work at all for you.)


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/MU56YBMPWSVMK3HAUJOBRW4KEN2ZLVCG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Terry Reedy]

On 3/28/2022 5:44 PM, Jason Ansel via Python-Dev wrote:

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this 
proposal may break the next major release of PyTorch.

The related project is TorchDynamo, which can be found here:
https://github.com/facebookresearch/torchdynamo

We will likely move this into the core of PyTorch closer to release.

If the changed happens, would PyTorch still be able to use the eval frame API?  
Or would it prevent from being used entirely?


I believe that you will just have to add a new or different #include 
statement.  I recommend that you add a comment to the issue, 
https://bugs.python.org/issue46850,

and even more, that you create a branch with the diffs in

https://github.com/python/cpython/pull/32052
https://github.com/python/cpython/pull/32054

applied so that you can experiment and verify what I believe.


--
Terry Jan Reedy

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/JS7YMJGEWCY5PBF4ATMZPWXVY6PYLPKK/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Jason Ansel via Python-Dev]

The PyTorch team plans to use PEP 523 as a part of PyTorch 2.0, so this 
proposal may break the next major release of PyTorch.

The related project is TorchDynamo, which can be found here:
https://github.com/facebookresearch/torchdynamo

We will likely move this into the core of PyTorch closer to release.

If the changed happens, would PyTorch still be able to use the eval frame API?  
Or would it prevent from being used entirely?
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/RVQ7LDIJ2OYAN4QMIPTI3A3PODGBLNN7/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 24. 03. 22 20:06, Fabio Zadrozny wrote:


Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny > escreveu:


PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild.
I'm not
sure if these ones should be moved to the internal C API. They
can be
left unchanged, since they don't use a type only defined by the
internal C API.

Just to note, the pydevd/debugpy debuggers actually uses all of
those APIs.

i.e.:


https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187



https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232



https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311



The debugger already has workarounds because of changes to
evaluation api over time (see:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491

)
and I know 3.11 won't be different.

I'm ok with changes as I understand that this is a special API -- as
long as there's still a way to use it and get the information needed
(the debugger already goes through many hops because it needs to use
many internals of CPython -- in every new release it's a **really**
big task to update to the latest version as almost everything that
the debugger relies to make debugging fast changes across versions
and I never really know if it'll be possible to support it until I
really try to do the port -- I appreciate having less things in a
public API so it's easier to have extensions work in other
interpreters/not recompiling on newer versions, but please keep it
possible to use private APIs which provides the same access that
CPython has to access things internally for special cases such as
the debugger).

Maybe later on that PEP from mark which allows a better debugger API
could alleviate that (but until then, if possible I appreciate it if
there's some effort not to break things unless really needed --
ideally with instructions on how to port).

Anyways, to wrap up, the debugger already needs to be built with
`Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private
API (as long as it's still accessible in that case) is probably not
a big issue for the debugger and having setters/getters to set it
instead of relying on `state.interp.eval_frame` seems good to me.

Cheers,

Fabio


I think the main issue here is the compatibility across the same version 
though... is it possible to have some kind of guarantee on private APIs 
that something won't change across micro-releases?


Currently we don't really have that (except that in practice, bugfix 
releases tend to not need internal API changes.)



I.e.: having the frame evaluation function change across major releases 
and having them be reworked seems reasonable, but then having the frame 
evaluation be changed across micro-releases wouldn't be.


So, I'm ok in pushing things to the internal API, but then I still would 
like guarantees about the compatibility of that API in the same major 
release (otherwise those setters/getters/frame evaluation should 
probably remain on the public API if the related structure was moved to 
the internal API).


Perhaps we need a new "tier" of C API for debuggers -- API that's 
guaranteed stable for a major release, and if it's changed it should 
always break with compile errors (e.g. the function gets a new 
argument), rather than silently change semantics.

The internal API Cython & greenlet need might go it this category too.


___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OF4UNE22IDDJDJVQZMOUXHELAYEFCJ45/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Fabio Zadrozny]

Em qui., 24 de mar. de 2022 às 15:39, Fabio Zadrozny 
escreveu:

> PEP 523 API added more private functions for code objects:
>>
>> * _PyEval_RequestCodeExtraIndex()
>> * _PyCode_GetExtra()
>> * _PyCode_SetExtra()
>>
>> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
>> pydevd debugger. The two others seem to be unused in the wild. I'm not
>> sure if these ones should be moved to the internal C API. They can be
>> left unchanged, since they don't use a type only defined by the
>> internal C API.
>>
> Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.
>
> i.e.:
>
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
>
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311
>
> The debugger already has workarounds because of changes to evaluation api
> over time (see:
> https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
> and I know 3.11 won't be different.
>
> I'm ok with changes as I understand that this is a special API -- as long
> as there's still a way to use it and get the information needed (the
> debugger already goes through many hops because it needs to use many
> internals of CPython -- in every new release it's a **really** big task to
> update to the latest version as almost everything that the debugger relies
> to make debugging fast changes across versions and I never really know if
> it'll be possible to support it until I really try to do the port -- I
> appreciate having less things in a public API so it's easier to have
> extensions work in other interpreters/not recompiling on newer versions,
> but please keep it possible to use private APIs which provides the same
> access that CPython has to access things internally for special cases such
> as the debugger).
>
> Maybe later on that PEP from mark which allows a better debugger API could
> alleviate that (but until then, if possible I appreciate it if there's some
> effort not to break things unless really needed -- ideally with
> instructions on how to port).
>
> Anyways, to wrap up, the debugger already needs to be built with
> `Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API
> (as long as it's still accessible in that case) is probably not a big issue
> for the debugger and having setters/getters to set it instead of relying on
> `state.interp.eval_frame` seems good to me.
>
> Cheers,
>
> Fabio
>
>

I think the main issue here is the compatibility across the same version
though... is it possible to have some kind of guarantee on private APIs
that something won't change across micro-releases?

I.e.: having the frame evaluation function change across major releases and
having them be reworked seems reasonable, but then having the frame
evaluation be changed across micro-releases wouldn't be.

So, I'm ok in pushing things to the internal API, but then I still would
like guarantees about the compatibility of that API in the same major
release (otherwise those setters/getters/frame evaluation should probably
remain on the public API if the related structure was moved to the internal
API).

Cheers,

Fabio
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/DHKE7LVN4R7NQFTBJJHGXI3AJOK6OYIV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Fabio Zadrozny]

>
> PEP 523 API added more private functions for code objects:
>
> * _PyEval_RequestCodeExtraIndex()
> * _PyCode_GetExtra()
> * _PyCode_SetExtra()
>
> The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> pydevd debugger. The two others seem to be unused in the wild. I'm not
> sure if these ones should be moved to the internal C API. They can be
> left unchanged, since they don't use a type only defined by the
> internal C API.
>
Just to note, the pydevd/debugpy debuggers actually uses all of those APIs.

i.e.:

https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L187
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L232
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L311

The debugger already has workarounds because of changes to evaluation api
over time (see:
https://github.com/fabioz/PyDev.Debugger/blob/main/_pydevd_frame_eval/pydevd_frame_evaluator.template.pyx#L491)
and I know 3.11 won't be different.

I'm ok with changes as I understand that this is a special API -- as long
as there's still a way to use it and get the information needed (the
debugger already goes through many hops because it needs to use many
internals of CPython -- in every new release it's a **really** big task to
update to the latest version as almost everything that the debugger relies
to make debugging fast changes across versions and I never really know if
it'll be possible to support it until I really try to do the port -- I
appreciate having less things in a public API so it's easier to have
extensions work in other interpreters/not recompiling on newer versions,
but please keep it possible to use private APIs which provides the same
access that CPython has to access things internally for special cases such
as the debugger).

Maybe later on that PEP from mark which allows a better debugger API could
alleviate that (but until then, if possible I appreciate it if there's some
effort not to break things unless really needed -- ideally with
instructions on how to port).

Anyways, to wrap up, the debugger already needs to be built with
`Py_BUILD_CORE_MODULE=1` anyways, so, I guess having it in a private API
(as long as it's still accessible in that case) is probably not a big issue
for the debugger and having setters/getters to set it instead of relying on
`state.interp.eval_frame` seems good to me.

Cheers,

Fabio
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/XPDT55ANVKHGG74D62HDBOFLC4EXWJ26/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Brett Cannon]

On Wed, Mar 23, 2022 at 2:48 AM Petr Viktorin  wrote:

> On 22. 03. 22 19:07, Victor Stinner wrote:
> > Hi,
> >
> > I proposed two PRs to move the private C API (Include/cpython/) of PEP
> > 523 "Adding a frame evaluation API to CPython" to the internal C API
> > (Include/internals/):
> >
> > * https://github.com/python/cpython/pull/32052
> > * https://github.com/python/cpython/pull/32054
> >
> > API:
> >
> > * _PyFrameEvalFunction type
> > * _PyInterpreterState_GetEvalFrameFunc()
> > * _PyInterpreterState_SetEvalFrameFunc()
> > * (undocumented) _PyEval_EvalFrameDefault()
> >
> > The private API to get/set the eval function *is* documented at:
> >
> https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc
> >
> > I added the Get/Set functions so debuggers don't have to access
> > directly to the PyInterpreterState structure which has been moved to
> > the internal C API in Python 3.8.
> >
> > This API causes me multiple issues:
> >
> > * It's a private API and I'm trying to remove the private API from the
> > public C API header files.
> > * The _PyFrameEvalFunction type is not stable: it got a new "tstate"
> > parameter in Python 3.9 and the type of the second parameter changed
> > from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
> > * These functions use the _PyInterpreterFrame type which is part of
> > the internal C API.
> >
> > While Pyston didn't bring a JIT compiler to Python with PEP 523,
> > debuggers were made faster by using this API. Debuggers like pydevd,
> > debugpy and ptvsd use it.
> >
> > I propose to move theses API to the internal header files
> > (Include/internals/) to clarify that it's not part of the public C API
> > and that there is no backward compatibility warranty.
> >
> > The change is being discussed at:
> > https://bugs.python.org/issue46850
> >
> > --
> >
> > PEP 523 API added more private functions for code objects:
> >
> > * _PyEval_RequestCodeExtraIndex()
> > * _PyCode_GetExtra()
> > * _PyCode_SetExtra()
> >
> > The _PyEval_RequestCodeExtraIndex() function seems to be used by the
> > pydevd debugger. The two others seem to be unused in the wild. I'm not
> > sure if these ones should be moved to the internal C API. They can be
> > left unchanged, since they don't use a type only defined by the
> > internal C API.
>
> PEP 523 clearly meant for these to be used by external tools, but made
> them private. That sounds like a contradiction.
>
> Brett/Dino, what was the intent here?
>

>From the PEP :
"The API is purposefully listed as private to communicate the fact that
there are no semantic guarantees of the API between Python releases."

-Brett


>
> IMO, if external code should use these, they should lose the leading
> underscore.
>
>
>
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/OI73PLAILLG75IKKZAADIH4GSOIXFWIQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/22/2022 11:28 PM, Victor Stinner wrote:

On Tue, Mar 22, 2022 at 7:33 PM Steve Dower  wrote:

After a normal deprecation period, yes?


There is no backward compatibility warranty and no deprecation process
for private APIs.


And yet you're asking the question, which means you know these are 
special ;)


The PEP says: "The API is purposefully listed as private to communicate 
the fact that there are no semantic guarantees of the API between Python 
releases."


Absence/presence isn't a semantic guarantee, it's an availability 
guarantee. Code using them should be able to rely on their presence, and 
ideally their prototype (though it seems we've messed that up in the 
past), but shouldn't expect code that worked against 3.8 to also work 
against 3.9 or 3.10.


Perhaps in hindsight, we could have not used the underscore and just 
explicitly described them as being behaviorally unstable between major 
versions. I guess that would have raised exactly the same question though.


The point is, it's a documented API that we've told people they can use. 
We can't simply revoke that without telling people that it's going to 
happen, even if we covered ourselves for there being version changes 
that affect how they need to be used.


Cheers,
Steve

___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/YF75IDAHS66BKJZDAAQDGOW2IGU2KME5/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Petr Viktorin]

On 22. 03. 22 19:07, Victor Stinner wrote:

Hi,

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):

* https://github.com/python/cpython/pull/32052
* https://github.com/python/cpython/pull/32054

API:

* _PyFrameEvalFunction type
* _PyInterpreterState_GetEvalFrameFunc()
* _PyInterpreterState_SetEvalFrameFunc()
* (undocumented) _PyEval_EvalFrameDefault()

The private API to get/set the eval function *is* documented at:
https://docs.python.org/dev/c-api/init.html#c._PyInterpreterState_GetEvalFrameFunc

I added the Get/Set functions so debuggers don't have to access
directly to the PyInterpreterState structure which has been moved to
the internal C API in Python 3.8.

This API causes me multiple issues:

* It's a private API and I'm trying to remove the private API from the
public C API header files.
* The _PyFrameEvalFunction type is not stable: it got a new "tstate"
parameter in Python 3.9 and the type of the second parameter changed
from PyFrameObject* to _PyInterpreterFrame* in Python 3.11.
* These functions use the _PyInterpreterFrame type which is part of
the internal C API.

While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.

I propose to move theses API to the internal header files
(Include/internals/) to clarify that it's not part of the public C API
and that there is no backward compatibility warranty.

The change is being discussed at:
https://bugs.python.org/issue46850

--

PEP 523 API added more private functions for code objects:

* _PyEval_RequestCodeExtraIndex()
* _PyCode_GetExtra()
* _PyCode_SetExtra()

The _PyEval_RequestCodeExtraIndex() function seems to be used by the
pydevd debugger. The two others seem to be unused in the wild. I'm not
sure if these ones should be moved to the internal C API. They can be
left unchanged, since they don't use a type only defined by the
internal C API.


PEP 523 clearly meant for these to be used by external tools, but made 
them private. That sounds like a contradiction.


Brett/Dino, what was the intent here?

IMO, if external code should use these, they should lose the leading 
underscore.



___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/CW3EKBNZCPYPCQMNEXPBV2BKTKI3Y6TX/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Victor Stinner]

On Tue, Mar 22, 2022 at 7:33 PM Steve Dower  wrote:
> After a normal deprecation period, yes?

There is no backward compatibility warranty and no deprecation process
for private APIs.

Victor
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/GTHWKT3QPPMU4VEC6SNO5HLRNNAZI6US/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-Dev] Re: C API: Move PEP 523 "Adding a frame evaluation API to CPython" private C API to the internal C API

[Steve Dower]

On 3/22/2022 6:07 PM, Victor Stinner wrote:

I proposed two PRs to move the private C API (Include/cpython/) of PEP
523 "Adding a frame evaluation API to CPython" to the internal C API
(Include/internals/):


After a normal deprecation period, yes?


While Pyston didn't bring a JIT compiler to Python with PEP 523,
debuggers were made faster by using this API. Debuggers like pydevd,
debugpy and ptvsd use it.


Pyjion (https://github.com/tonybaloney/Pyjion) uses it, and ptvsd never 
did. I also understand that pydevd - and implicitly debugpy - disabled 
its use of the API by default because of reliability, but that was a few 
years back so maybe they fixed it. So it's likely only being used by the 
project that requested it, though I don't think that's enough to justify 
skipping normal deprecation.


Cheers,
Steve
___
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-le...@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at 
https://mail.python.org/archives/list/python-dev@python.org/message/X6EC7KW4B5CB4R5CS4WSQYWSXIYFV4J3/
Code of Conduct: http://python.org/psf/codeofconduct/