[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Blaze Spinnaker]

Blaze Spinnaker  added the comment:

I know this issue is closed, but can you folks review:

https://github.com/ipython/ipykernel/issues/548#issuecomment-713637954

The nest_asyncio patch is becoming a defacto contract to fixing this amongst a 
lot of people.  But the patch doesn't really event work (IMHO).

It would be ideal to address this in a more central manner, either with change 
of code, better errors, more clear APIs, better documentation, etc.   Lots of 
options, but hopefully it will be addressed.

--
nosy: +blazespinnaker

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Yury Selivanov]

Yury Selivanov  added the comment:



--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Let's close the issue as "won't fix".

Third-party loop implementation *can* be reentrant but we don't want to 
encourage people to use this pattern in stdlib.

--
resolution:  -> wont fix
stage: patch review -> resolved
status: open -> closed

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Jason Fried]

Jason Fried  added the comment:

Yeah I have to agree at this point, from a naive point of view it looks
awesome, but its just as bad as loosing gil guarantees.

On Mon, Aug 26, 2019 at 03:04 Andrew Svetlov  wrote:

>
> Andrew Svetlov  added the comment:
>
> The solution produces subtle and error-prone code.  It can live in third
> party library but not good enough for stdlib I think.
>
> --
>
> ___
> Python tracker 
> 
> ___
>

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Andrew Svetlov]

Andrew Svetlov  added the comment:

The solution produces subtle and error-prone code.  It can live in third party 
library but not good enough for stdlib I think.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[flying sheep]

flying sheep  added the comment:

There’s this monkeypatching solution: https://pypi.org/project/nest-asyncio/

But yes, it’s a very practical problem that you can’t call async code from sync 
code that’s being called from async code.

--
nosy: +flying sheep

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Jason Fried]

Jason Fried  added the comment:

For loops not supporting this throwing NotImplmentedError from the method to 
enable reentrancy seems appropriate. 

"You should convert all your stack to async functions..."

That may not be practical for large code bases in transition to asyncio. The 
fixes for reentrancy that I find in reality are not adding async logic through 
out the call stack but instead its one of the two I listed.

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Andrew Svetlov]

Andrew Svetlov  added the comment:

Sorry, no.
The feature was requested many times but was constantly rejected.
By this, you are adding a BLOCKING call to your async function.
At least it leads to log warning about too long callback execution.

Moreover, I suspect that `run_until_complete` reentrancy requirement breaks the 
existing third-party loop implementations, not all loops can be fixed easily.

The last: calling blocking code from async code is the anti-pattern, asyncio 
explicitly discourages it.

You should convert all your stack to async functions and add sync stubs
when needed like

def sync_call(arg):
asyncio.get_event_loop().run_until_complete(async_call(arg))

Yuri, do you agree with me?

--

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Jason Fried]

Change by Jason Fried :


--
keywords: +patch
pull_requests: +6540
stage:  -> patch review

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Jason Fried]

Change by Jason Fried :


--
nosy: +lukasz.langa

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com

[issue33523] loop.run_until_complete re-entrancy to support more complicated codebases in transition to asyncio

[Jason Fried]

New submission from Jason Fried :

At Facebook and Instagram we have large interconnected codebases without clear 
boundaries of ownership. As we move more and more services to utilize asyncio 
we are finding that once blocking (but fast) code paths, are now cropping up 
with asyncio code using run_until_complete().  Now this is fine for all the 
blocking callers, but some times we have async callers to that blocking code 
path and now it doesn't work.  

So we have two options revert the change to not use asyncio deep in the dep 
tree or Convert all functions in the stack to be asyncio.  Both are not 
possible and engineers have solved them in two crazy ways.

1. Nested Event Loops, when you hit a run_until_complete, create a new 
eventloop and do the async and return the result.
2. Like the first, but each library creates its own eventloop, and swaps it 
with the running loop for the duration of the run_until_complete, restoring the 
original loop when its done. 

Both of these ultimately have the same problem, everything on the primary event 
loop stops running until the new loop is complete. What if we could instead 
start servicing the existing eventloop from the run_until_complete. This would 
insure that tasks don't timeout.

This would allow us to convert things to asyncio faster without having to have 
absolute knowledge of a codebase and its call graph, and not have to have all 
engineers completely synchronized.

--
components: asyncio
messages: 316678
nosy: asvetlov, fried, yselivanov
priority: normal
severity: normal
status: open
title: loop.run_until_complete re-entrancy to support more complicated 
codebases in transition to asyncio
type: enhancement
versions: Python 3.8

___
Python tracker 

___
___
Python-bugs-list mailing list
Unsubscribe: 
https://mail.python.org/mailman/options/python-bugs-list/archive%40mail-archive.com