skywalker0618 commented on issue #18424: URL: https://github.com/apache/hudi/issues/18424#issuecomment-4160048048
> > To fix this, we can potentially change the flushDisruptor function NOT to call disruptorQueue.close() directly, but implementing a function to spin-wait until the disruptor buffer is fully consumed. In other words, just does the "spin-wait part" but not the "terminate thread" part. > > Looks reasonable as long as we have a good way to collect the finished threads. Does it mean the disrupt thread is closed after the buffer is fully consumed? Yes this is true. Basically disruptorQueue.close() closes disruptor thread after the distruptor buffer is fully consumed. Then the GCS upload thread found that "writer thread is not alive" so it will exit as well. Then the following step to flush parquet buffer to GCS failed. The proposed solution is actually not killing the disruptor thread. Is there a reason we want to kill and re-init that thread? If we kill that, the GCS upload thread will also exit, then we need to re-init 2 threads. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
