lidavidm commented on pull request #12099: URL: https://github.com/apache/arrow/pull/12099#issuecomment-1007726131
Ah, the fundamental issue is S3FS implements writes asynchronously (unless background_writes=False), but our file interfaces are still mostly synchronous, and the dataset writer is asynchronous, using the thread pool to manage parallelism…so we have nested parallelism. Setting background_writes=False fixes it by breaking this loop, the other way will be to have at least CloseAsync(). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
