[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23
[ https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548021#comment-13548021 ] Hudson commented on HIVE-3303: -- Integrated in Hive-trunk-hadoop2 #54 (See [https://builds.apache.org/job/Hive-trunk-hadoop2/54/]) HIVE-3303: Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 (Zhenxiao Luo via Ashutosh Chauhan) (Revision 1367413) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1367413 Files : * /hive/trunk/build-common.xml * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_hadoop20.q * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff_hadoop20.q * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_hadoop20.q.out * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff_hadoop20.q.out Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 - Key: HIVE-3303 URL: https://issues.apache.org/jira/browse/HIVE-3303 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Fix For: 0.10.0 Attachments: HIVE-3303.1.patch.txt when running hive on hadoop23, mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs: [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script. [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out [junit] 5c5 [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] — [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script The error code 2(which indicates unable to initialize custom script) could not be retrieved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23
[ https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13425913#comment-13425913 ] Hudson commented on HIVE-3303: -- Integrated in Hive-trunk-h0.21 #1579 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1579/]) HIVE-3303: Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 (Zhenxiao Luo via Ashutosh Chauhan) (Revision 1367413) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1367413 Files : * /hive/trunk/build-common.xml * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace.q * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_hadoop20.q * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff.q * /hive/trunk/ql/src/test/queries/clientnegative/mapreduce_stack_trace_turnoff_hadoop20.q * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_hadoop20.q.out * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out * /hive/trunk/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff_hadoop20.q.out Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 - Key: HIVE-3303 URL: https://issues.apache.org/jira/browse/HIVE-3303 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Fix For: 0.10.0 Attachments: HIVE-3303.1.patch.txt when running hive on hadoop23, mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs: [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script. [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out [junit] 5c5 [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] — [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script The error code 2(which indicates unable to initialize custom script) could not be retrieved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23
[ https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422916#comment-13422916 ] Zhenxiao Luo commented on HIVE-3303: review request submitted at: https://reviews.facebook.net/D4365 Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 - Key: HIVE-3303 URL: https://issues.apache.org/jira/browse/HIVE-3303 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-3303.1.patch.txt when running hive on hadoop23, mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs: [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script. [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out [junit] 5c5 [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] — [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script The error code 2(which indicates unable to initialize custom script) could not be retrieved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23
[ https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13423329#comment-13423329 ] Ashutosh Chauhan commented on HIVE-3303: +1 Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 - Key: HIVE-3303 URL: https://issues.apache.org/jira/browse/HIVE-3303 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo Attachments: HIVE-3303.1.patch.txt when running hive on hadoop23, mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs: [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script. [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out [junit] 5c5 [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] — [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script The error code 2(which indicates unable to initialize custom script) could not be retrieved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23
[
https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422816#comment-13422816
]
Zhenxiao Luo commented on HIVE-3303:
The problem is, hadoop23 is getting Task-diagnostics differently from hadoop20.
In hadoop20, Task-diagnostics is retrieved via jobSubmitClient in
JobClient.java:
public String[] getTaskDiagnostics(TaskAttemptID id) throws IOException {
return jobSubmitClient.getTaskDiagnostics(id);
}
And in JobTracker.java, all the related logs are put into diagnostic info:
public synchronized String[] getTaskDiagnostics(TaskAttemptID taskId)
throws IOException {
JobID jobId = taskId.getJobID();
TaskID tipId = taskId.getTaskID();
JobInProgress job = jobs.get(jobId);
if (job == null) {
throw new IllegalArgumentException(Job + jobId + not found.);
}
TaskInProgress tip = job.getTaskInProgress(tipId);
if (tip == null) {
throw new IllegalArgumentException(TIP + tipId + not found.);
}
ListString taskDiagnosticInfo = tip.getDiagnosticInfo(taskId);
return ((taskDiagnosticInfo == null) ? null
: taskDiagnosticInfo.toArray(new String[0]));
}
Here is the diagnostic info in hadoop20:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row {key:238,value:val_238}
[junit] at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
[junit] at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
[junit] at
org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358)
[junit] at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307)
[junit] at org.apache.hadoop.mapred.Child.main(Child.java:170)
[junit] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row {key:238,value:val_238}
[junit] at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:548)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
[junit] ... 4 more
[junit] Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: [Error
2]: Unable to initialize custom script.
[junit] at
org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:346)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
[junit] at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
[junit] at
org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
[junit] at
org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:762)
[junit] at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:529)
[junit] ... 5 more
[junit] Caused by: java.io.IOException: Cannot run program
script_does_not_exist: java.io.IOException: error=2, No such file or directory
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:475)
[junit] at
org.apache.hadoop.hive.ql.exec.ScriptOperator.processOp(ScriptOperator.java:305)
[junit] ... 14 more
[junit] Caused by: java.io.IOException: java.io.IOException: error=2, No
such file or directory
[junit] at java.lang.UNIXProcess.init(UNIXProcess.java:164)
[junit] at java.lang.ProcessImpl.start(ProcessImpl.java:81)
[junit] at java.lang.ProcessBuilder.start(ProcessBuilder.java:468)
[junit] ... 15 more
The error code [2] appears in the diagnostic info, and could be retrieved
by Hive.
While, in hadoop23, in Job.java, a different execution path is:
public String[] getTaskDiagnostics(final TaskAttemptID taskid)
throws IOException, InterruptedException {
ensureState(JobState.RUNNING);
return ugi.doAs(new PrivilegedExceptionActionString[]() {
@Override
public String[] run() throws IOException, InterruptedException {
return cluster.getClient().getTaskDiagnostics(taskid);
}
});
}
Here is the diagnostic info in hadoop23:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row {key:238,value:val_238}
[junit] at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
[junit] at
org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
[junit] at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
[junit] at
[jira] [Commented] (HIVE-3303) Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23
[ https://issues.apache.org/jira/browse/HIVE-3303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13422856#comment-13422856 ] Carl Steinbach commented on HIVE-3303: -- We should use the [INCLUDE|EXCLUDE]_HADOOP_MAJOR_VERSIONS macros to fix this. The 0.23 behavior should be the standard going forward, so please create mapreduce_stack_trace_h20.q and use the INCLUDE macro, and EXCLUDE 0.20 from mapreduce_stack_trace.q Fix error code inconsistency bug in mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q when running hive on hadoop23 - Key: HIVE-3303 URL: https://issues.apache.org/jira/browse/HIVE-3303 Project: Hive Issue Type: Bug Reporter: Zhenxiao Luo Assignee: Zhenxiao Luo when running hive on hadoop23, mapreduce_stack_trace.q and mapreduce_stack_trace_turnoff.q are having inconsistent error code diffs: [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace.q.out [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script. [junit] diff -a /home/cloudera/Code/hive/build/ql/test/logs/clientnegative/mapreduce_stack_trace_turnoff.q.out /home/cloudera/Code/hive/ql/src/test/results/clientnegative/mapreduce_stack_trace_turnoff.q.out [junit] 5c5 [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask [junit] — [junit] FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask. Unable to initialize custom script The error code 2(which indicates unable to initialize custom script) could not be retrieved. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
