Can we use a lower default timeout to mitigate this issue in the short term (I'd imagine one second or possibly smaller would be sufficient for our use), and get a fix upstream in the long term?
On Fri, Oct 11, 2019 at 9:38 AM Luke Cwik <lc...@google.com> wrote: > > I'm looking for a thread pool that re-uses threads that are idle before > creating new ones and has an API that is compatible with the > concurrent.futures ThreadPoolExecutor[1]. > > To my knowledge, the concurrent.futures ThreadPool creates new threads for > tasks up until the thread pool limit before re-using existing ones for all > Python versions prior to 3.8. > > I tried using CollapsingThreadPoolExecutor within pr/9477[2] but after > testing it with Apache Beam, I found that it has some pool shutdown issues[3]. > > Does anyone have any suggestions for a good Python library that contains a > stable thread pool implementation? > > Preferably the library that provides the thread pool would have no > dependencies and be compatible with the same Python versions that Apache Beam > is compatible with today. > > 1: > https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor > 1: https://github.com/apache/beam/pull/9477 > 2: https://github.com/ftpsolutions/collapsing-thread-pool-executor/issues/3
