[
https://issues.apache.org/jira/browse/HIVE-13374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15613273#comment-15613273
]
Thejas M Nair commented on HIVE-13374:
--------------------------------------
[~pxiong] If this is resulting in longer compile phase time, HS2 would block
other query compiles before 2.0 (HIVE-4239) , as there was a lock around
compile phase.
Thats likely one part of the issue.
The other part seems to be memory consumption is very high when very large
number (10s of thousands) of partitions are queried.
> HiveServer2 hangs up if query to scan too many partition is submitted
> ---------------------------------------------------------------------
>
> Key: HIVE-13374
> URL: https://issues.apache.org/jira/browse/HIVE-13374
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
> Affects Versions: 1.2.1
> Environment: HDP2.4.0
> JDK1.8
> Reporter: Wataru Yukawa
>
> If query to scan too many partition(for example, more than 40,000 partitions)
> is submitted, hiveserver2 hangs up
> hiveserver2 java heap status is the following.
> https://gyazo.com/e0a35bd628a0dc904bc24f19cdb2261e
> https://gyazo.com/65f12cd3875330995508ae11afd394a2
> OutOfMemoryError doesn't occur. but other queries fails.
> Maybe it is necessary to spill to disk...
> hiveserver2.log
> {code}
> ....
> 2016-03-28 13:44:28,923 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getSplits(517)) -
> Total number of paths: 47409, launching 50 threads to check non-combinable
> ones.
> 2016-03-28 13:44:29,444 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(439))
> - CombineHiveInputSplit creating pool for
> hdfs://.../aaa=.../yyyymmdd=20150721; using filter path
> hdfs://.../aaa=.../yyyymmdd=20150721
> ....
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444))
> - CombineHiveInputSplit: pool is already created for
> hdfs://.../aaa=.../yyyymmdd=20160326; using filter path
> hdfs://.../aaa=.../yyyymmdd=20160326
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444))
> - CombineHiveInputSplit: pool is already created for
> hdfs://.../aaa=.../yyyymmdd=20160327; using filter path
> hdfs://.../aaa=.../yyyymmdd=20160327
> 2016-03-28 13:44:34,811 INFO [HiveServer2-Handler-Pool: Thread-766923]:
> io.CombineHiveInputFormat (CombineHiveInputFormat.java:getCombineSplits(444))
> - CombineHiveInputSplit: pool is already created for
> hdfs://.../aaa=.../yyyymmdd=20160328; using filter path
> hdfs://.../aaa=.../yyyymmdd=20160328
> 2016-03-28 13:45:00,285 INFO
> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@6f5d0190]:
> util.JvmPauseMonitor (JvmPauseMonitor.java:run(195)) - Detected pause in JVM
> or host machine (eg GC): pause of approximately 4587ms
> GC pool 'PS MarkSweep' had collection(s): count=1 time=4699ms
> GC pool 'PS Scavenge' had collection(s): count=1 time=63ms
> ....
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
