I would like to clarify that asyncio *does* most certainly provide support
for programs that use multiple event loops and interact with a number of
different threads. So, asyncio can definitely be used outside of a purely
"single-process, single-threaded, single-event-loop
approach". However, a key point that should be followed is that __async
objects__ created in a single event loop are intended to be used only in
that event loop. If you try to go outside of the bounds of that, you'll
likely find yourself in the territory of undefined behavior (which is a
part of why we're deprecating the loop argument throughout the high-level
API, to discourage that).

The asyncio part of your program can use a thread to run a function and get
the result without blocking the event loop (using
loop.run_in_executor(None, func, *args) or the upcoming asyncio.to_thread()
in 3.9), schedule callbacks from another thread with
loop.call_soon_threadsafe(), or submit a coroutine to be run in a specific
event loop with asyncio.run_coroutine_threadsafe() from a different thread.
But it does require some thinking about the architecture of your program,
and can't just be used to replace an existing threaded program without any
considerations.

Fundamentally, OS threads and coroutines are two entirely different models
of concurrency; though, we do have interoperability tools in place and are
actively working on making them easier to utilize. Also, with adequate
arguments for specific real-world use cases, those interoperability tools
can be expanded upon as needed. However, these should be separate from
existing parts of asyncio rather than changing methods like
loop.run_until_complete(), IMO. There's something to be said about having
too much purpose crammed into a single API member -- it tends to make the
behavior needlessly complex and hard to follow.

On Tue, Jun 9, 2020 at 2:37 AM Stephen J. Turnbull <
turnbull.stephen...@u.tsukuba.ac.jp> wrote:

> Fortunately, Guido has jumped in.  He's the authority on asyncio.  I
> recommend you consider his advice very carefully.  Here are a couple
> more commments that I had already written when I saw his post.
>
> Celelibi writes:
>
>  > But that doesn't mean we can't try to design asyncio to be more
>  > thread-friendly.
>
> True, lack of an example to emulate doesn't mean we can't try ... but
> the fact that asyncio and threads are based on completely different
> models of concurrency likely does, in combination with that lack.
> Strong suggestion that "here be dragons".
>
>  > I'm working on switching out a thread-based client protocol lib for
>  > an asyncio-based lib instead.
>
> Have you tried to do this with an alternative to asyncio such as curio
> (by David Beazley, IIRC) or trio (by Nathaniel Smith)?  They provide
> simpler APIs, and Nathaniel claims that they also provide a
> concurrency model that's far easier to understand.  They're still
> async def/await-based, but they have a property Nathaniel calls
> "respect for causality" that seems very useful in understanding
> programs.
>
>  > But if it did, I'd end up calling call_soon_threadsafe from the thread
>  > that runs the event loop and wait for the result. Which would just lock
>  > up.
>
> If I understand correctly, this is the kind of thing that "respect for
> causality" helps to identify and to some extent prevent.
>
>  > Another issue I found is that the client protocol lib class isn't always
>  > instanciated by the same thread, and its constructor indirectly
>  > instanciate asyncio objects (like Event).
>
> The relations among the threading library, the asyncio library, and
> the client protocol library in your software are quite unclear to me.
> But this sounds very bad for your "divide and conquer" strategy.
> Asyncio is fundamentally a single-process, single-threaded,
> single-event-loop approach to handling concurrency.  If the asyncio
> part of your program can ignore what other threads are doing, you
> might be OK, but this kind of coupling means you can't, and will need
> to do things like:
>
>  > In the end, its start method will start the event loop in its own
>  > thread.  With the deprecation of the loop argument, Event() finds
>  > the loop it should use by calling get_event_loop() in its
>  > constructor. I therefore had to add a call set_event_loop(), even
>  > though this event loop will never run in this thread. That's a
>  > pretty ugly hack.
>
> But it's intended to be a temporary one, no?  This is the kind of
> thing you need to do when you have multiple incompatible models of
> computation running in the same system.
>
>  > The deprecation of the loop arguments looks like an incentive to create
>  > the asyncio objects in the threads that use them.
>
> It is.  A major point (perhaps the whole point) of asyncio is to
> provide that "single-process, single-threaded, single-event-loop
> approach" which nevertheless is extremely fast and scalable for
> I/O-bound programs.
>
> If you can't get to the point where all of your interactions with the
> threaded code can be done in a single thread, asyncio is very unlikely
> to be a good model for your system.
>
>  > Which seems pretty crazy to me as a whole.
>
> You should think carefully about that feeling.  If it's just because
> you're temporarily in concurrency model hell, you'll get over that
> once the port to asyncio is done.  But if you really need multiple
> threads because some code exposes them and you can't port that code to
> asyncio, you need to consider whether asyncio can do what you need.
>
> Steve
> _______________________________________________
> Python-ideas mailing list -- python-ideas@python.org
> To unsubscribe send an email to python-ideas-le...@python.org
> https://mail.python.org/mailman3/lists/python-ideas.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-ideas@python.org/message/CFSVWPMXSNP6ABLGBSOVVKZMBKNB2ER3/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-le...@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at 
https://mail.python.org/archives/list/python-ideas@python.org/message/WQHM7HV754NAIAQ4J6LKPK2VPUGAYP4I/
Code of Conduct: http://python.org/psf/codeofconduct/

Reply via email to