[issue33918] Hooking into pause/resume of iterators/coroutines

[Andrew Svetlov]

Change by Andrew Svetlov :


--
resolution:  -> rejected
stage:  -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Andrew Svetlov]

Andrew Svetlov  added the comment:

decimal was changed from threading.local to contextvar usage.
The module is "safe" not only for asyncio but for threading, trio etc.

unittest.mock doesn't use explicit context all for patching.
It changes global interpreter-wide objects instead.

So, mock.patch fails not only if two async tasks are executed in parallel but 
two threads also.

I doubt if thread-local (or contextvar) can be applied to mock because it 
changes the current behavior -- but this is a different story.

*Any* library that needs to modify a global state, e.g. your MyLogger.enabled 
can use contextvars for handling it.

Say again, contextvars is not for asyncio-only but a generic instrument for 
handling context-aware variables.

I'm going to close the issue.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Liran Nuna]

Liran Nuna  added the comment:

> That's still doable with contextvars. You just need a custom mock-like object 
> (or library) that stores its settings/state in a context variable.

contextvars only work with asyncio, what about the iterator case?

In addition, you can't possibly expect authors to re-implement a library just 
because it may or may not be used with asyncio. In my example, re-implementing 
mock/patch is quite a task just to get such basic functionality.

In other words, contextvars don't solve this issue, it just creates new issues 
to solve and causes code duplication.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Yury Selivanov]

Yury Selivanov  added the comment:

> Imagine the context manager is mock.patch used in testing and you want to run 
> two tests in "parallel", each with a different mocked method. mock.patch 
> isn't aware of `await` so patching will be incorrect.

That's still doable with contextvars. You just need a custom mock-like object 
(or library) that stores its settings/state in a context variable.

Now, the "mock" module doesn't provide this functionality out of the box, but I 
hope that somebody will come up with a new mock library that will work that way 
(or with a new mock primitive) after 3.7.0 is released.

Adding __pause__ and __resume__ was considered in PEP 521, and it was decided 
that the actual implementation will be too slow and complex.  It's very 
unlikely that PEP 521 is ever accepted.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Liran Nuna]

Liran Nuna  added the comment:

> You should try to use the contextvars module that was specifically created to 
> handle local context state (for tasks & coroutines).

Yury, from my original report:

> I'm aware that this particular problem could be solved with the new context 
> variables introduced with python3.7, however it is just a simplification of 
> our actual issue.

Not everything can use context managers. Imagine the context manager is 
mock.patch used in testing and you want to run two tests in "parallel", each 
with a different mocked method. mock.patch isn't aware of `await` so patching 
will be incorrect.

Those are just some behaviors where context variables don't solve the issue I'm 
describing.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Yury Selivanov]

Yury Selivanov  added the comment:

You should try to use the contextvars module that was specifically created to 
handle local context state (for tasks & coroutines).

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Liran Nuna]

Liran Nuna  added the comment:

I would like to stress this issue happens with iterators as well, and this 
isn't a unique issue to asyncio only. 

I would like to propose four new magic methods for context managers to solve 
this: __pause__, __resume__, __apause__ and __aresume__ which will be called 
before/after a pause/resume happen before the coroutine/iterator continues.

I'm not sure however if this is the correct venue for such discussion.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Ned Deily]

Change by Ned Deily :


--
components: +asyncio -Interpreter Core
nosy: +asvetlov, yselivanov
versions:  -Python 3.4, Python 3.5

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33918] Hooking into pause/resume of iterators/coroutines

[Liran Nuna]

New submission from Liran Nuna :

An interesting property of async programming is that execution order is 
nondeterministic and async function "pause" and "resume" execution as events 
come in.

This can play havok with context managers, especially ones that wrap a global 
state change. I can best explain it with code - see attached file.

If you were to run this, you'd notice that "Should be logged" does not get 
logged - this is because the execution order runs the context manager 
immediately and that affects the entire batch (created by asyncio.gather).

Is there a way to hook into a pause/resume handling of coroutines so this kind 
of thing could be done correctly? I'm aware that this particular problem could 
be solved with the new context variables introduced with python3.7, however it 
is just a simplification of our actual issue.

Iterators also suffer from this issue, as `yield` pauses and resumes execution.

--
components: Interpreter Core
files: async_context_managers.py
messages: 320101
nosy: Liran Nuna
priority: normal
severity: normal
status: open
title: Hooking into pause/resume of iterators/coroutines
type: behavior
versions: Python 3.4, Python 3.5, Python 3.6, Python 3.7, Python 3.8
Added file: https://bugs.python.org/file47646/async_context_managers.py

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com