[
https://issues.apache.org/jira/browse/HIVE-6448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dima machlin updated HIVE-6448:
-------------------------------
Description:
too many union all statements in a single query require massive amount of
memory to process.
Under default HiveServer Xmx (256) we can run ~50 union alls.
It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
The error is :
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at
org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
at
org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
Query example :
select *
from (
select count(*) from default.dual union all
select count(*) from default.dual union all
......
select count(*) from default.dual) z
was:
too many union all statements in a single query require massive amount of
memory to process.
Under default HiveServer Xmx (256) we can run ~50 union alls.
It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
The error is :
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
at java.lang.StringBuilder.append(StringBuilder.java:119)
at
org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
at org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
at
org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Java heap space pasring a query with many union all
> ---------------------------------------------------
>
> Key: HIVE-6448
> URL: https://issues.apache.org/jira/browse/HIVE-6448
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Query Processor
> Affects Versions: 0.11.0
> Environment:
> Reporter: dima machlin
>
> too many union all statements in a single query require massive amount of
> memory to process.
> Under default HiveServer Xmx (256) we can run ~50 union alls.
> It seems to be rather linear as increasing the Xmx to 1024 we can run ~200.
> The error is :
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2882)
> at
> java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
> at
> java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:390)
> at java.lang.StringBuilder.append(StringBuilder.java:119)
> at
> org.apache.hadoop.hive.ql.QueryPlan.getJSONKeyValue(QueryPlan.java:453)
> at
> org.apache.hadoop.hive.ql.QueryPlan.getJSONQuery(QueryPlan.java:598)
> at org.apache.hadoop.hive.ql.QueryPlan.toString(QueryPlan.java:609)
> at
> org.apache.hadoop.hive.ql.history.HiveHistory.logPlanProgress(HiveHistory.java:499)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:136)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.run(TaskRunner.java:47)
> Query example :
> select *
> from (
> select count(*) from default.dual union all
> select count(*) from default.dual union all
> ......
> select count(*) from default.dual) z
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
