Sergey Shelukhin created HIVE-17642:
---------------------------------------
Summary: union test OOMs
Key: HIVE-17642
URL: https://issues.apache.org/jira/browse/HIVE-17642
Project: Hive
Issue Type: Bug
Reporter: Sergey Shelukhin
Noticed this while investigating some spurious failures due to a combination of
errors.
The reason unionDistinct (or whatever) test fails sporadically is OOM:
{noformat}
2017-09-28T01:07:25,804 ERROR [3d4e3f44-40c5-431a-b3de-801d60c1c579 main]
ql.Driver: FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 61,
vertexId=vertex_1506585924598_0001_53_01, diagnostics=[Vertex
vertex_1506585924598_0001_53_01 [Map 61] killed/failed due
to:ROOT_INPUT_INIT_FAILURE, Vertex Input: src initializer failed,
vertex=vertex_1506585924598_0001_53_01 [Map 61], java.lang.OutOfMemoryError: GC
overhead limit exceeded
at
java.util.concurrent.ConcurrentHashMap.putVal(ConcurrentHashMap.java:1019)
at
java.util.concurrent.ConcurrentHashMap.putAll(ConcurrentHashMap.java:1084)
at
java.util.concurrent.ConcurrentHashMap.<init>(ConcurrentHashMap.java:852)
at org.apache.hadoop.conf.Configuration.<init>(Configuration.java:721)
at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:442)
at
org.apache.hadoop.hive.ql.io.HiveInputFormat.getSplits(HiveInputFormat.java:591)
at
org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.initialize(HiveSplitGenerator.java:196)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:278)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:269)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1807)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:269)
at
org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:253)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
]Invalid event V_INTERNAL_ERROR on Vertex vertex_1506585924598_0001_53_00 [Map
60]
2017-09-28T01:07:25,804 DEBUG [3d4e3f44-40c5-431a-b3de-801d60c1c579 main]
ql.Driver: Shutting down query
SELECT count(1) FROM (
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src UNION DISTINCT
SELECT key, value FROM src) src
{noformat}
Perhaps we should make it smaller.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
