kylebarron commented on issue #6587: URL: https://github.com/apache/arrow-rs/issues/6587#issuecomment-2429692681
I'm not fully clear on all the details on how the pyo3-async-runtimes integration works. [From its readme](https://github.com/PyO3/pyo3-async-runtimes#managing-event-loops): > in pyo3-asyncio, we decided the best way to handle Rust/Python interop was to just surrender the main thread to Python and run Rust's event loops in the background. And I know there's revamped async work going on in https://github.com/PyO3/pyo3/issues/1632 and https://github.com/wyfo/pyo3-async, which is suppose to remove some overhead of integrating rust and python async. To answer your questions: > Ok so if I follow, this would mean that in order for tokio to poll the stream, the python async code would need to invoke it in a timely manner? Or to put it differently, if the python code is busy doing something else, the tokio work would get starved. I _think_ so, yes. At least in my current implementation of this, if `__anext__` is never called, then the tokio future's `next()` is never called. So it relies on Python to drive this async. > I wonder if this is going to lead to the same sorts of issues we've run into multiplexing CPU bound tasks on the same threadpool? Potentially, but I'd say this is up to the user to ensure the Python async runtime doesn't get blocked, and that they're not running CPU bound tasks on the main thread. > Perhaps there needs to be some sort of buffering between the two regardless, which would obviate any lifetime shenanigans? I'm not sure how to implement this 😅 , or else I'd try to implement it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
