Hi, I'm using flight to receive streams from client and write to the storage with python `pa.dataset.write_dataset` API. The whole data is 1 Billion rows, over 40GB with one partition column ranges from 0~63. The container runs at 8-cores CPU and 4GB ram resources. It can be done about 160s (6M rows/s, each record batch is about 32K rows) for completing transferring and writing almost synchronously, after setting 128 for io_thread_count. Then I'd like to find out the bottleneck of the system, CPU or RAM or storage? 1. I extend the ram into 32GB, then the server consumes more ram, the writing progress works at the beginning, then suddenly slow down and the data accumulated into ram until OOM. 2. Then I set the ram to 64GB, so that the server will not killed by OOM. Same happens, also, after all the data is transferred (in memory), the server consumes all CPU shares (800%), but still very slow on writing (not totally stopped, but about 100MB/minute). 2.1 I'm wondering if the io thread is stuck, or the computation task is stuck. After setting both io_thread_count and cpu_count to 32, I wrapped the input stream of write_dataset with a callback on each record batch, I can tell that all the record batches are consumed into write_dataset API. 2.2 I dumped all threads stack traces and grab a flamegraph. See https://gist.github.com/hu6360567/e21ce04e7f620fafb5500cd93d44d3fb.
It seems that all threads stucks at ThrottledAsyncTaskSchedulerImpl. -- --------------------- Best Regards, Wenbo Hu,