[ https://issues.apache.org/jira/browse/HIVE-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16139848#comment-16139848 ]
Hive QA commented on HIVE-16949: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12883454/HIVE-16949.1.patch {color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified. {color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11003 tests executed *Failed tests:* {noformat} org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_windowing2] (batchId=10) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_vectorized_dynamic_partition_pruning] (batchId=169) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query14] (batchId=235) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=235) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema (batchId=180) org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation (batchId=180) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6514/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6514/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6514/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 7 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12883454 - PreCommit-HIVE-Build > Leak of threads from Get-Input-Paths thread pool when more than 1 used in > query > ------------------------------------------------------------------------------- > > Key: HIVE-16949 > URL: https://issues.apache.org/jira/browse/HIVE-16949 > Project: Hive > Issue Type: Bug > Components: HiveServer2 > Reporter: Birger Brunswiek > Assignee: Sahil Takiar > Attachments: HIVE-16949.1.patch > > > The commit > [20210de|https://github.com/apache/hive/commit/20210dec94148c9b529132b1545df3dd7be083c3] > which was part of HIVE-15546 [introduced a thread > pool|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3109] > which is not shutdown upon completion of its threads. This leads to a leak > of threads for each query which uses more than 1 partition. They are not > removed automatically. When queries spanning multiple partitions are made the > number of threads increases and is never reduced. On my machine hiveserver2 > starts to get slower and slower once 10k threads are reached. > Thread pools only shutdown automatically in special circumstances (see > [documentation section > _Finalization_|https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/ThreadPoolExecutor.html]). > This is not currently the case for the Get-Input-Paths thread pool. I would > add a _pool.shutdown()_ in a finally block just before returning the result > to make sure the threads are really shutdown. > My current workaround is to set {{hive.exec.input.listing.max.threads = 1}}. > This prevents the the thread pool from being spawned > [\[1\]|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2118] > > [\[2\]|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3107]. > The same issue probably also applies to the [Get-Input-Summary thread > pool|https://github.com/apache/hive/blob/824b9c80b443dc4e2b9ad35214a23ac756e75234/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L2193]. -- This message was sent by Atlassian JIRA (v6.4.14#64029)