Re: Asynchronous execution of synchronous functions
On Tue, Sep 27, 2022 at 3:19 AM Axy via Python-list wrote: > > > Did you check the ThreadPoolExecutor or the ProcessPoolExecutor? They > > won't give you atomic writes unless you add a Lock or a Condition, but > > they will execute your code in another thread or process. > > Yes, I did, but they are too complicated to use. I'd like something for > humans, such as > > asynchronizer = InThreadExecutor() > > result = await asynchronizer.run(myfunc, myargs, mykwargs) > There are two convenient APIs in asyncio. asyncio.to_thread and run_in_executor. to_thread creates and destroy a dedicated thread for the function. run_in_executor uses threadpool owned by loop. https://docs.python.org/3/library/asyncio-task.html?highlight=to_thread#asyncio.to_thread https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.loop.run_in_executor > > and I almost implemented that borrowing code from asyncsqlite (doves fly > slowly, electric mail is even slower :)), but... > > > Keep in mind that Python's threads have a global interpreter lock > > (GIL) that prevents full parallelism. Processes work as expected, but > > require IPC and pickable objects in and out. > > yes, that became a problem. > > So, I revoke my question. Went out to redesign the whole approach. > > Thanks for reply! > > Axy. > -- > https://mail.python.org/mailman/listinfo/python-list -- Inada Naoki -- https://mail.python.org/mailman/listinfo/python-list
Re: Asynchronous execution of synchronous functions
On 2022-09-26, Stefan Ram wrote: > So, I wanted to try to download all pages in parallel with > processes to avoid any GIL effect, while I don't understand > what the GIL actuall is. But processes didn't work here, so > I tried threads. This worked and now the total run time is > down to about 50 seconds. Downloading things from the network is *extremely* I/O-bound. So, as you have discovered, the GIL is going to make essentially no difference whatsoever. -- https://mail.python.org/mailman/listinfo/python-list
Re: Asynchronous execution of synchronous functions
Did you check the ThreadPoolExecutor or the ProcessPoolExecutor? They won't give you atomic writes unless you add a Lock or a Condition, but they will execute your code in another thread or process. Yes, I did, but they are too complicated to use. I'd like something for humans, such as asynchronizer = InThreadExecutor() result = await asynchronizer.run(myfunc, myargs, mykwargs) and I almost implemented that borrowing code from asyncsqlite (doves fly slowly, electric mail is even slower :)), but... Keep in mind that Python's threads have a global interpreter lock (GIL) that prevents full parallelism. Processes work as expected, but require IPC and pickable objects in and out. yes, that became a problem. So, I revoke my question. Went out to redesign the whole approach. Thanks for reply! Axy. -- https://mail.python.org/mailman/listinfo/python-list
RE: Asynchronous execution of synchronous functions
Did you check the ThreadPoolExecutor or the ProcessPoolExecutor? They won't give you atomic writes unless you add a Lock or a Condition, but they will execute your code in another thread or process. https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor https://docs.python.org/3/library/concurrent.futures.html#processpoolexecutor Keep in mind that Python's threads have a global interpreter lock (GIL) that prevents full parallelism. Processes work as expected, but require IPC and pickable objects in and out. -- Diego Souza Wespa Intelligent Systems Rio de Janeiro - Brasil On Mon, Sep 26, 2022 at 1:01 PM wrote: > From: Axy > To: Python List > Date: Mon, 26 Sep 2022 12:17:47 +0100 > Subject: Asynchronous execution of synchronous functions > Hi there, > > is there a library to call functions in context of a thread? For > example, as in asyncsqlite which has a thread and a queue I mean has > anyone generalized such an approach already? > > If not, I'll do it myself, no problem. > > It's a kind of tiny stuff, like atomicwrites, which is quite difficult > to dig out with modern search engines that have already rolled down to > hell a decade ago. > > Axy. -- https://mail.python.org/mailman/listinfo/python-list
Asynchronous execution of synchronous functions
Hi there, is there a library to call functions in context of a thread? For example, as in asyncsqlite which has a thread and a queue I mean has anyone generalized such an approach already? If not, I'll do it myself, no problem. It's a kind of tiny stuff, like atomicwrites, which is quite difficult to dig out with modern search engines that have already rolled down to hell a decade ago. Axy. -- https://mail.python.org/mailman/listinfo/python-list
