[Python-ideas] Re: Add a way to execute code in the context of different frame

2020-02-18 Thread Guido van Rossum 
On Tue, Feb 18, 2020 at 11:09 AM Serhiy Storchaka 
wrote:

> 18.02.20 20:13, Guido van Rossum пише:
> > I am a little confused how you get from "there are extra frames in the
> > traceback" to "modify exec() to run code in another frame".
>
> Sorry for skipping several steps. I'll try to expand them here. PEP 558
> does not have relation to this, as frame is not the part of locals().
>

Ah, fair.

exec() is used by the import machinery to execute the module code.
> Currently frames look like this:
>
>  import foo  # user code
>  # many frames in the import machinery
>  # ...
>  # ...
>  exec(code, module.__dict__)  # still in the import machinery
>  # user code in foo.py
>
> I propose to add a parameter to exec() which will allow to execute the
> code in the imported mode as exec() was called immediately from the
> importer code:
>
>  import foo  # user code
>  # user code in foo.py
>
> As it was when the import machinery was implemented in C.
>

This makes sense (except when debugging the import machinery, which is also
a valid use case. :-).

The caller would still have to provide a dict for the globals (the module
dict), and I guess it would have to point to the *previous* frame (the one
executing `import foo`) so that exec() can create a new frame pointing back
there. And somehow we would need to set things up so that after exec()
completes or raises we automatically return or raise to the parent frame,
rather than passing control to exec()'s caller. Or alternatively, we'd need
exec()'s caller (deep down inside importlib) to be able to say "pretend to
return or raise from this (new) frame".


> I did not propose to modify exec() to run code in another frame, I
> proposde to modify exec() to run code in new frame whose parent is a
> specified frame (either arbitrary frame or on of ancestors of the caller
> frame).
>

I understand that now.

I think it may be not difficult to do. I ask whether there are other
> applications of this or similar feature and what form of the feature
> will be more convenient.
>

My gut feeling says that the only other use cases will be alternatives to
importlib, perhaps other kinds of import hooks.


> It just occurred to me that it may be not a feature of exec(), but a
> separate function which will allow to execute an arbitrary function, not
> just a Python code, as it was called from the different frame.
>
>  apply_in_frame(frame, func, /, *args, **kwargs)
>
> or
>
>  apply_with_frames_skipped(stacklevel, func, /, *args, **kwargs)
>
> The question is how to call this function and where to put it. In the
> first variant it could be a method of frame.
>

So either way it would create a new frame, right? And it would not pass
control to the caller, but to the indicated frame.

I'm not that keen on passing a stacklevel -- that could easily cause
confusion, and to calculate the right level the caller would have to crawl
up the stack anyway.

A problem I see with either solution is that it looks like except and
finally blocks in the intervening frames would be skipped. Or perhaps not,
but at that point I suspect that the implementation won't be all that easy
after all.

So *if* we end up doing this optimization I think it may be best to make it
something that's clearly meant for the importlib use case. Sine we already
have sys._getframe(), maybe it could be another magic thing in sys?

-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/JY2RCOZKES5RESIZBFTKWDPXAXTHUUPV/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

2020-02-18 Thread Serhiy Storchaka 

18.02.20 20:13, Guido van Rossum пише:
I am a little confused how you get from "there are extra frames in the 
traceback" to "modify exec() to run code in another frame".


Sorry for skipping several steps. I'll try to expand them here. PEP 558 
does not have relation to this, as frame is not the part of locals().


exec() is used by the import machinery to execute the module code. 
Currently frames look like this:


import foo  # user code
# many frames in the import machinery
# ...
# ...
exec(code, module.__dict__)  # still in the import machinery
# user code in foo.py

I propose to add a parameter to exec() which will allow to execute the 
code in the imported mode as exec() was called immediately from the 
importer code:


import foo  # user code
# user code in foo.py

As it was when the import machinery was implemented in C.

I did not propose to modify exec() to run code in another frame, I 
proposde to modify exec() to run code in new frame whose parent is a 
specified frame (either arbitrary frame or on of ancestors of the caller 
frame).


I think it may be not difficult to do. I ask whether there are other 
applications of this or similar feature and what form of the feature 
will be more convenient.


It just occurred to me that it may be not a feature of exec(), but a 
separate function which will allow to execute an arbitrary function, not 
just a Python code, as it was called from the different frame.


apply_in_frame(frame, func, /, *args, **kwargs)

or

apply_with_frames_skipped(stacklevel, func, /, *args, **kwargs)

The question is how to call this function and where to put it. In the 
first variant it could be a method of frame.


___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/LEYVARMREKR74BKUOPSSIKN7ZF674WDQ/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

2020-02-18 Thread Guido van Rossum 
I am a little confused how you get from "there are extra frames in the
traceback" to "modify exec() to run code in another frame".

Also, with PEP 558 (Defined semantics for locals(), by Nick Coghlan) we
might be able to just pass the frame's globals and locals to exec() or
eval() without further changes (or perhaps just with changes to allow
passing a proxy instead of a true dict).

On Mon, Feb 17, 2020 at 11:36 PM Serhiy Storchaka 
wrote:

> The idea is inspired by the following StackOverflow question:
> https://stackoverflow.com/questions/40945752/inspect-who-imported-me and
> the corresponding BPO issue: https://bugs.python.org/issue39643.
>
> In the older Python versions the f_back attribute of the frame in which
> the module code is executed pointed to the frame which executes the
> import statement or calls __import__() or import_module(). But after
> rewriting the import machinery in Python, there are other frames between
> the caller frame and the calling frame. This caused problems with
> tracebacks which was worked around by removing intermediate frames
> related to the import machinery when the exception floats to the caller.
> But there is still unresolved problem with warnings, and all frames of
> the import machinery are visible from the imported code.
>
> I propose to add possibility to execute the code in the context of
> different frame. Either add a frame argument in exec() and eval(), which
> will allow to pass an arbitrary frame. Or add a stacklevel argument in
> exec() and eval() (similar to warnings.warn()), which will limit
> possible frames to the parent frames of the current frame. I do not know
> what is more feasible.
>
> This will automatically fix problems with warnings. This will allow to
> get rid the workaround for exceptions, and may make the code simpler.
> This will allow the imported code to do miscellaneous magic things with
> the code which performs the import.
>
> What do you think about this?
> ___
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/ETBOLQKHTZFH7ELNOJG3TWRAXCOGYX2D/
> Code of Conduct: http://python.org/psf/codeofconduct/
>


-- 
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/FUUDB734UEXB6B5G3FXPBZOPSBHG3ZVG/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

2020-02-18 Thread Serhiy Storchaka 

18.02.20 11:28, M.-A. Lemburg пише:

This sounds like a nice idea, but it may make sense to limit the
frame to one already on the stack. Otherwise, the code you're
executing may never return to your current stack...


I do not think there will be such problem. exec() will temporary replace 
the stack of frames before executing the code and restore the original 
stack after it be finished. Since the stack is implemented as a 
single-linked list, and multiple lists can share the same tail, it all 
should be zero-overhead.



OTOH, perhaps that could be regarded and used as a feature, similar
to what PyPy exposes:

https://doc.pypy.org/en/latest/stackless.html


Thank you for the link. It is interesting. The difference with PyPy is 
that I do not propose to create cycles. And since CPython uses the 
single C stack it has nothing with stackless and do not allow to get rid 
of the recursion depth limit. But there is something common in these 
ideas -- both work with f_back. Maybe more.

___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/MQQ547VS24A6TQ5MGX4CP2HKYIT74PXM/
Code of Conduct: http://python.org/psf/codeofconduct/

[Python-ideas] Re: Add a way to execute code in the context of different frame

2020-02-18 Thread M.-A. Lemburg 
On 18.02.2020 08:33, Serhiy Storchaka wrote:
> The idea is inspired by the following StackOverflow question:
> https://stackoverflow.com/questions/40945752/inspect-who-imported-me and
> the corresponding BPO issue: https://bugs.python.org/issue39643.
> 
> In the older Python versions the f_back attribute of the frame in which
> the module code is executed pointed to the frame which executes the
> import statement or calls __import__() or import_module(). But after
> rewriting the import machinery in Python, there are other frames between
> the caller frame and the calling frame. This caused problems with
> tracebacks which was worked around by removing intermediate frames
> related to the import machinery when the exception floats to the caller.
> But there is still unresolved problem with warnings, and all frames of
> the import machinery are visible from the imported code.
> 
> I propose to add possibility to execute the code in the context of
> different frame. Either add a frame argument in exec() and eval(), which
> will allow to pass an arbitrary frame. Or add a stacklevel argument in
> exec() and eval() (similar to warnings.warn()), which will limit
> possible frames to the parent frames of the current frame. I do not know
> what is more feasible.
> 
> This will automatically fix problems with warnings. This will allow to
> get rid the workaround for exceptions, and may make the code simpler.
> This will allow the imported code to do miscellaneous magic things with
> the code which performs the import.
> 
> What do you think about this?

This sounds like a nice idea, but it may make sense to limit the
frame to one already on the stack. Otherwise, the code you're
executing may never return to your current stack...

OTOH, perhaps that could be regarded and used as a feature, similar
to what PyPy exposes:

https://doc.pypy.org/en/latest/stackless.html

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, Feb 18 2020)
>>> Python Projects, Coaching and Support ...https://www.egenix.com/
>>> Python Product Development ...https://consulting.egenix.com/


::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
   Registered at Amtsgericht Duesseldorf: HRB 46611
   https://www.egenix.com/company/contact/
 https://www.malemburg.com/
___
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/XNFL4FBRSXJL2MGNIWILP4HHGECRGOSH/
Code of Conduct: http://python.org/psf/codeofconduct/