svalaskevicius commented on PR #15312: URL: https://github.com/apache/iceberg/pull/15312#issuecomment-4351980799
Hi @rdblue, Just to clarify: > Next, I see newExistingScheduledPool has 2 outside uses, S3FileIO and GCSFileIO, and newExitingWorkerPool has 3 outside uses in HadoopFileIO, RESTMetricsReporter, and AuthSessionCache. The problem covers all thread pools currently managed by MoreExecutors - including both worker pools here. > 2. A global shutdown for all thread pools created by ThreadPools A global shutdown exists already via MoreExecutors, just that it is deferred to the JVM shutdown hook and there is no control as to when exactly will it be invoked. --- > I think those are both poor solutions. For the first one, if you don't want a threadpool that is configured like the static pools, the right answer is to use your own threadpool and manage its lifecycle <..> I agree with this. If iceberg had thread pools injected via constructors (and without relying on the global JVM shutdown hooks) it would address the issue cleanly. At this point, however, I was looking for a pragmatic solution. We have an application that uses the iceberg library, and cannot commit/flush files on exit because iceberg has already killed its threads and is producing a lot of errors how the ThreadPools have been stopped. Given that, and that there is an open issue related to thread stopping, my reasoning was that these can be combined to a single change to solve both problems - this is the resulting PR. I suppose the question is - is this an acceptable interim solution until the threadpool (factory?) injection is implemented, or should this PR be closed and converted to an open issue (for the uncontrolled/non-deterministic shutdown behaviour)? Do you have any thoughts as to when could the proper fix be expected? Regards, Sarunas -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
