Bruce Nelson created HIVE-4650: ---------------------------------- Summary: Getting Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask on auto convert to MapJoin after upgrade to Hive-0.11.0.x from hive-0.10.0.x Key: HIVE-4650 URL: https://issues.apache.org/jira/browse/HIVE-4650 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Environment: HortonWorks 1.3 distro on x86_64 Centos 6 Reporter: Bruce Nelson
working from a simple table in Hive hive> desc cmnt > ; OK x1 int None x2 int None x3 int None x4 int None y double None hive> select * from cmnt; OK 7 26 6 60 78.5 1 29 15 52 74.3 11 56 8 20 104.3 11 31 8 47 87.6 7 52 6 33 95.9 11 55 9 22 109.2 3 71 17 6 102.7 1 31 22 44 72.5 2 54 18 22 93.1 21 47 4 26 115.9 1 40 23 34 83.8 11 66 9 12 113.3 10 68 8 12 109.4 A query that joins and transforms against this table : select * from (select VAL001 x1,VAL002 x2,VAL003 x3,VAL004 x4,VAL005 y from ( select /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v3 join (select count(*) c, avg(VAL001) mu1,avg(VAL002) mu2,avg(VAL003) mu3,avg(VAL004) mu4,avg(VAL005) mu5, stddev_pop(VAL001) sd1,stddev_pop(VAL002) sd2,stddev_pop(VAL003) sd3,stddev_pop(VAL004) sd4,stddev_pop(VAL005) sd5 from ( select * from ( select x1 VAL001,x2 VAL002,x3 VAL003,x4 VAL004,y VAL005 from cmnt ) obj1_3 ) v1) v2 ) obj1_7) obj1_6 ; Generates during Stage-3 : setting HADOOP_USER_NAME test Execution log at: /tmp/test/.log 2013-06-03 12:40:55 Starting to launch local task to process map join; maximum memory = 1065484288 2013-06-03 12:40:56 Processing rows: 1 Hashtable size: 1 Memory usage: 7175528 rate: 0.007 2013-06-03 12:40:56 Dump the hashtable into file: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable 2013-06-03 12:40:56 Upload 1 File to: file:/tmp/test/hive_2013-06-03_00-40-21_708_6820064283161196136/-local-10003/HashTable-Stage-3/MapJoin-mapfile00--.hashtable File size: 334 2013-06-03 12:40:56 End of local task; Time Taken: 0.726 sec. Execution completed successfully Mapred Local Task Succeeded . Convert the Join into MapJoin Mapred Local Task Succeeded . Convert the Join into MapJoin Launching Job 2 out of 2 Number of reduce tasks is set to 0 since there's no reduce operator Starting Job = job_201306022123_0045, Tracking URL = http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045 Kill Command = /usr/lib/hadoop/libexec/../bin/hadoop job -kill job_201306022123_0045 Hadoop job information for Stage-3: number of mappers: 1; number of reducers: 0 2013-06-03 00:41:05,895 Stage-3 map = 0%, reduce = 0% 2013-06-03 00:41:40,687 Stage-3 map = 100%, reduce = 100% Ended Job = job_201306022123_0045 with errors Error during job, obtaining debugging information... Job Tracking URL: http://sun1vm3:50030/jobdetails.jsp?jobid=job_201306022123_0045 Examining task ID: task_201306022123_0045_m_000002 (and more) from job job_201306022123_0045 Task with the most failures(4): ----- Task ID: task_201306022123_0045_m_000000 URL: http://sun1vm3:50030/taskdetails.jsp?jobid=job_201306022123_0045&tipid=task_201306022123_0045_m_000000 ----- Diagnostic Messages for this Task: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:365) at org.apache.hadoop.mapred.Child$4.run(Child.java:255) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.mapred.Child.main(Child.java:249) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1377) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) ... 8 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:186) ... 14 more FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask If "hive.auto.convert.join = false" is set then the all the query stages work OK. The same scenario worked OK in Hive-0.10.0.x and Hive-0.9.x with MapJoin working. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira