chen cong created HIVE-15097:
--------------------------------
Summary: Error: java.lang.RuntimeException: Error in configuring
object Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
Key: HIVE-15097
URL: https://issues.apache.org/jira/browse/HIVE-15097
Project: Hive
Issue Type: Bug
Environment: hive 1.2.1
Reporter: chen cong
hive> CREATE TABLE user_tag_pre AS
> SELECT A.userid , A.tag , A.single_tag_sumcount/B.sum_tag_sumcount_avg as
pre
> FROM user_tag_sumcount A , user_tag_avg B
> WHERE A.userid = B.userid;
Query ID = ubuntu_20161031134731_bdbfeaa6-5687-460c-b331-86ce3237aa29
Total jobs = 2
Stage-7 is selected by condition resolver.
Stage-1 is filtered out by condition resolver.
Execution log at:
/tmp/ubuntu/ubuntu_20161031134731_bdbfeaa6-5687-460c-b331-86ce3237aa29.log
2016-10-31 13:47:34 Starting to launch local task to process map join;
maximum memory = 477626368
2016-10-31 13:47:37 Processing rows: 200000 Hashtable size: 199999
Memory usage: 134844656 percentage: 0.282
2016-10-31 13:47:38 Processing rows: 300000 Hashtable size: 299999
Memory usage: 121492112 percentage: 0.254
2016-10-31 13:47:38 Processing rows: 400000 Hashtable size: 399999
Memory usage: 122277256 percentage: 0.256
2016-10-31 13:47:38 Processing rows: 500000 Hashtable size: 499999
Memory usage: 151827872 percentage: 0.318
2016-10-31 13:47:40 Processing rows: 600000 Hashtable size: 599999
Memory usage: 179637408 percentage: 0.376
2016-10-31 13:47:40 Dump the side-table for tag: 1 with group count: 675226
into file:
file:/tmp/ubuntu/193dc11b-1e25-46a0-b124-210a7ffde96c/hive_2016-10-31_13-47-31_406_4438237799451830529-1/-local-10003/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
2016-10-31 13:47:41 Uploaded 1 File to:
file:/tmp/ubuntu/193dc11b-1e25-46a0-b124-210a7ffde96c/hive_2016-10-31_13-47-31_406_4438237799451830529-1/-local-10003/HashTable-Stage-4/MapJoin-mapfile01--.hashtable
(22302233 bytes)
2016-10-31 13:47:41 End of local task; Time Taken: 6.116 sec.
Execution completed successfully
MapredLocal task succeeded
Launching Job 2 out of 2
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1477875480008_0012, Tracking URL =
http://master:8088/proxy/application_1477875480008_0012/
Kill Command = /home/ubuntu/cloud/hadoop-2.7.2/bin/hadoop job -kill
job_1477875480008_0012
Hadoop job information for Stage-4: number of mappers: 1; number of reducers: 0
2016-10-31 13:47:49,787 Stage-4 map = 0%, reduce = 0%
2016-10-31 13:48:50,370 Stage-4 map = 0%, reduce = 0%
2016-10-31 13:49:50,768 Stage-4 map = 0%, reduce = 0%
2016-10-31 13:50:35,748 Stage-4 map = 100%, reduce = 0%
Ended Job = job_1477875480008_0012 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1477875480008_0012_m_000000 (and more) from job
job_1477875480008_0012
Task with the most failures(4):
-----
Task ID:
task_1477875480008_0012_m_000000
URL:
http://master:8088/taskdetails.jsp?jobid=job_1477875480008_0012&tipid=task_1477875480008_0012_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:449)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:112)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
... 14 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
... 17 more
Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.apache.hadoop.hive.serde2.objectinspector.primitive.WritableStringObjectInspector.copyObject(WritableStringObjectInspector.java:36)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:311)
at
org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.copyToStandardObject(ObjectInspectorUtils.java:346)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyObject.read(MapJoinKeyObject.java:112)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyObject.read(MapJoinKeyObject.java:107)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinKeyObject.read(MapJoinKeyObject.java:103)
at
org.apache.hadoop.hive.ql.exec.persistence.MapJoinTableContainerSerDe.load(MapJoinTableContainerSerDe.java:83)
at
org.apache.hadoop.hive.ql.exec.mr.HashTableLoader.load(HashTableLoader.java:98)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:288)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:173)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator$1.call(MapJoinOperator.java:169)
at
org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieve(ObjectCache.java:55)
at
org.apache.hadoop.hive.ql.exec.mr.ObjectCache.retrieveAsync(ObjectCache.java:63)
at
org.apache.hadoop.hive.ql.exec.MapJoinOperator.initializeOp(MapJoinOperator.java:166)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:362)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:481)
at
org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java:438)
at org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:375)
at
org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:78)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-4: Map: 1 HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec
Someone knows to solve it？
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
