[ https://issues.apache.org/jira/browse/TEZ-1187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15834119#comment-15834119 ]
Harish Jaiprakash commented on TEZ-1187: ---------------------------------------- Please do tell me if this makes sense: Define TezSharedExecutors as follows: {code} public interface TezSharedExecutors { ListeningScheduledExecutorService createScheduledExecutorService(); void shutdownAll(); } {code} The implementation would be roughly like this: * We have an shared executorService, mostly like a ForkJoinExecutor service with parallelism set to number vcores available for the container. * createScheduledExecutorService returns an internal implementation of ListeningScheduledExecutorService, delegating the responsibility to the SharedExecutorService for actual execution. * We track all the Futures returned by shared executor service, to be used in awaitTermination and shutdown calls. * For shutdownNow we can cancel all these futures. * shutdown will just prevent more jobs from getting submitted via the current object. * For the execute method, since it does not return any future and the runnable cannot be removed once submitted. We'll wrap it in a different runnable which will cleanup any references and prevent execution, in case shutdownNow is invoked. Does this implementation sound fine? > Share Thread pools between different tasks > ------------------------------------------ > > Key: TEZ-1187 > URL: https://issues.apache.org/jira/browse/TEZ-1187 > Project: Apache Tez > Issue Type: Sub-task > Reporter: Siddharth Seth > Assignee: Harish Jaiprakash > > Thread pools are used all over for fetchers, sort etc. When running a single > task - this is already a problem, and gets worse when running multiple tasks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)