[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: > Posted > https://mail.python.org/archives/list/python-...@python.org/thread/4UZJYAZL3NHRAGN5WAMJC4IHAHEXF3QF/ > to see if anyone else wants to weigh in. Mark Shannon listed flaws in the PEP 532 and suggest to withdraw this PEP. Honestly, I'm open to any

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: I plan to merge my PR 17340 at the end of week to not miss Python 3.9 feature freeze deadline, unless someone speaks up and find a good reason to not merge this PR. The PR adds a public C API PyInterpreterState_SetEvalFrameFunc() and now pass tstate to the

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: I marked bpo-38818 "Modify PyInterpreterState.eval_frame to pass tstate (PyThreadState)" as a duplicate of this issue. It has been said earlier that if we choose to add a public C API to set the frame evaluation function, it's better to design it properly

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by Paulo Henrique Silva : -- nosy: +phsilva ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: Fabio: "I don't really see a problem in breaking the API in major releases (so, having access for it in the internal API but without a backwards-compatibility guarantee seems like a good fit for me)." In Python, the internal C API now means that you have to

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by Ned Deily : -- keywords: +3.8regression nosy: +lukasz.langa priority: normal -> critical ___ Python tracker ___ ___

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: > @victor if we are going to ask folks to start using a setter and getter I say > we might as well get it right the first time and start taking the tstate now. Done: New PR 17352 combines my two other PRs. -- ___

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by STINNER Victor : -- pull_requests: +16836 pull_request: https://github.com/python/cpython/pull/17352 ___ Python tracker ___

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: @greg I hope this goes into 3.8.1 since it was a breaking change. @victor if we are going to ask folks to start using a setter and getter I say we might as well get it right the first time and start taking the tstate now. --

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: New question: if we add a public API to Python 3.8.1, would it make sense to directly change eval_frame type to add a new "tstate" parameter? See bpo-38818 for the rationale. I already proposed a change for the master branch:

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: I wrote PR 17340 to add Add PyInterpreterState_GetEvalFrameFunc() and PyInterpreterState_SetEvalFrameFunc() functions. Since the PEP 523 has been approved, for me, these functions must be public, not private. My implementation adds these functions to

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by STINNER Victor : -- pull_requests: +16824 pull_request: https://github.com/python/cpython/pull/17340 ___ Python tracker ___

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Gregory P. Smith added the comment: +1 to re-exposing a way to do PEP-523. PEP-523 added a public API, we unintentionally hid it behind the mask of Py_BUILD_CORE_MODULE in 3.8. We shouldn't remove PEP-523's abilities without a deprecation cycle. But given the use cases tend to be "deep"

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by Gregory P. Smith : -- nosy: +gregory.p.smith ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: Posted https://mail.python.org/archives/list/python-...@python.org/thread/4UZJYAZL3NHRAGN5WAMJC4IHAHEXF3QF/ to see if anyone else wants to weigh in. -- ___ Python tracker

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: @Mark > PEP 523 makes no mention of adding a getter or setter. > Adding them is a big change to Python semantics and shouldn't, IMO, be done > without a PEP that explicit states they are going to be added. Adding getters or setters for something that was

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Fabio Zadrozny added the comment: @Mark First you have to explain to me how you envision changing the method code reliably in the debugger... Import hooks don't work (they'd break with something as simple as the code below) def method(): a = 10 mod = reload(old_mod)

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: Fabio, Can you give me a specific example where changing the bytecode via the `__code__` attribute of a function does not work as expected? I am assuming that adding a breakpoint on a line is equivalent to adding `breakpoint();` at the beginning of that

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: Brett, PEP 523 makes no mention of adding a getter or setter. Adding them is a big change to Python semantics and shouldn't, IMO, be done without a PEP that explicit states they are going to be added. -- ___ Python

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: OK, then my vote is to provide getter and setter methods that are underscore-prefixed to mark them as "internal" with clear comments specifying that they are part of PEP 523 and thus should not be needlessly broken in a bugfix release. --

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Fabio Zadrozny added the comment: @Bret I don't really see a problem in breaking the API in major releases (so, having access for it in the internal API but without a backwards-compatibility guarantee seems like a good fit for me). -- ___ Python

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: I think the real question is whether this is part of the CPython public API or the CPython internal API. @Fabio how burdensome would it be if we placed this in the internal API that you can get access to but we don't make backwards-compatibility guarantees

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Fabio Zadrozny added the comment: @Mark @Brett Well, PEP 523 still works (it's just more inconvenient to use now). Still, if PEP 523 will still be supported I think that having the setter/getter makes sense. If it is to be deprecated as @Mark is suggesting it doesn't really make sense to

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by STINNER Victor : -- keywords: +patch pull_requests: +16739 stage: -> patch review pull_request: https://github.com/python/cpython/pull/17187 ___ Python tracker ___

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: I would like to bring this issue back on topic as this about how to expose PEP 523 support in Python 3.8, not whether Fabio should be using a different approach when he has been something sanctioned in an accepted PEP and was previously doable that was

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: Fabio, If the user changes the `__code__` attribute of a function then, AFAICT, your debugger does the wrong thing, but bytecode modification does the right thing. Suppose we have two functions `spam` and `eggs`. Set a break point in `eggs`, set

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Fabio Zadrozny added the comment: @Mark I can think of many use-cases which may break if the function code is changed (users can change the code in real-use cases and when they do that they'd loose debugging). So, as long as the function code is part of the public API of Python, the

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: It sounds to me like `PyInterpreterState.eval_frame` is being used to lazily modify the bytecode to support breakpoints. I can see no reason why changing the bytecode can't be done via `function.__code__`. Suppose the code-object with the breakpoint add is

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: I heard that for debuggers, the PEP 523 is really useful. I recall a huge speedup thanks to this PEP in Jetbrain: https://blog.jetbrains.com/pycharm/2017/03/inside-the-debugger-interview-with-elizaveta-shashkova/ --

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: PEP 523 was to give user code the ability to change the eval function. While the work was motivated by our JIT work, supporting debugging was another motivating factor: https://www.python.org/dev/peps/pep-0523/#debugging. There's no C API because at the time

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: By the way, it's still possible to access directly ts->interp if you include "pycore_pystate.h" header, which requires to define Py_BUILD_CORE_MODULE. This header is an internal header. -- ___ Python tracker

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Fabio Zadrozny added the comment: @Mark I don't want to change the original function code, I just want to change the code to be executed in the frame (i.e.: as breakpoints change things may be different). Changing the actual function code is a no-go since changing the real function code

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: Victor, I don't think this is a regression. `PyThreadState` is an internal opaque data structure, which means we are free to change it. That the `eval_frame` is hard to access is a feature not a bug, as it discourages misuse and enables us to remove it easily,

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

STINNER Victor added the comment: > IMO this punches a big hole in the Python execution model, but provides no > benefit. This PEP is about fixing a Python 3.8 regression. In Python 3.7, it was possible to get and set frame_eval. In Python 3.8, it's no longer possible. One option to fix

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Brett Cannon added the comment: @Mark are you strongly opposed because we're providing an API for changing the eval function in the CPython API and you think it should be in the private API? Or you objecting to PEP 523 all-up (based on your list of objections)? Either way the PEP was

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Fabio Zadrozny added the comment: @Mark Shannon what I do is change the code object of the frame about to be evaluated to add a programmatic breakpoint, to avoid the need to have the trace function set at contexts that would need to be traced (after changing the frame.f_code it goes on to

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: Fabio, OOI what are you trying achieve by changing the frame evaluator? -- ___ Python tracker ___

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Mark Shannon added the comment: I'm strongly opposed to this change. PEP 523 does not specify what the semantics of changing the interpreter frame evaluator actually is: Is the VM obliged to call the new interpreter? What happens if the custom evaluator leaves the VM in a inconsistent state?

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Dino Viehland added the comment: Adding the getter/setters seems perfectly reasonable to me, and I agree they should be underscore prefixed as well. -- ___ Python tracker

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Eric Snow added the comment: It depends on how you look at the degree to which you are interacting with the runtime. This is a fairly low-level hook into the runtime. So arguably if you are using this API then you should specify being a "core" extension. That said, getting that clever

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

Change by STINNER Victor : -- nosy: +eric.snow ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe:

[issue38500] Provide a way to get/set PyInterpreterState.frame_eval without needing to access interpreter internals

New submission from Fabio Zadrozny : In CPython 3.7 it was possible to do: #include "pystate.h" ... PyThreadState *ts = PyThreadState_Get(); PyInterpreterState *interp = ts->interp; interp->eval_frame = my_frame_eval_func; This is no longer possible because in 3.8 the PyInterpreterState is