Hi Nitin, I have a M/R job which gets executed as soon as a hive query runs. This hive query has many complex joins.
Both the Map/Reduce jobs complete with status at 100%, after which I get the below error 2014-06-27 23:42:06,396 INFO org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printInfo(377)) - 2014-06-27 23:42:06,395 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 532.7 sec 2014-06-27 23:42:07,429 INFO org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printInfo(377)) - 2014-06-27 23:42:07,428 Stage-2 map = 100%, reduce = 100%, Cumulative CPU 532.7 sec 2014-06-27 23:42:07,430 INFO org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printInfo(377)) - MapReduce Total cumulative CPU time: 8 minutes 52 seconds 700 msec 2014-06-27 23:42:07,445 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printError(386)) - Ended Job = job_1403907716709_0021 with errors 2014-06-27 23:42:07,447 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printError(386)) - Error during job, obtaining debugging information... 2014-06-27 23:42:07,447 INFO org.apache.hadoop.conf.Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(840)) - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address 2014-06-27 23:42:07,452 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printError(386)) - Examining task ID: task_1403907716709_0021_m_000004 (and more) from job job_1403907716709_0021 2014-06-27 23:42:07,453 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,453 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,453 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,453 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,454 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,454 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,454 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,454 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,454 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,454 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,457 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printError(386)) - Examining task ID: task_1403907716709_0021_r_000000 (and more) from job job_1403907716709_0021 2014-06-27 23:42:07,457 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,457 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,458 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,458 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,458 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,479 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,480 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,482 WARN org.apache.hadoop.hive.shims.HadoopShimsSecure (Hadoop23Shims.java:getTaskAttemptLogUrl(56)) - Can't fetch tasklog: TaskLogServlet is not supported in MR2 mode. 2014-06-27 23:42:07,491 ERROR org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printError(386)) - Task with the most failures(4): ----- Task ID: task_1403907716709_0021_r_000004 URL: http://ip-10-40-51-206.ec2.internal:9026/taskdetails.jsp?jobid=job_1403907716709_0021&tipid=task_1403907716709_0021_r_000004 ----- Diagnostic Messages for this Task: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2014-06-27 23:42:07,494 INFO org.apache.hadoop.hive.ql.exec.Task (SessionState.java:printInfo(377)) - Counters: 2014-06-27 23:42:07,501 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl (YarnClientImpl.java:killApplication(182)) - Killing application application_1403907716709_0021 2014-06-27 23:42:07,605 ERROR org.apache.hadoop.hive.ql.Driver (SessionState.java:printError(386)) - FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask 2014-06-27 23:42:07,606 INFO org.apache.hadoop.hive.ql.Driver (PerfLogger.java:PerfLogEnd(127)) - </PERFLOG method=Driver.execute start=1403911523544 end=1403912527606 duration=1004062> 2014-06-27 23:42:07,606 INFO org.apache.hadoop.hive.ql.Driver (SessionState.java:printInfo(377)) - MapReduce Jobs Launched: 2014-06-27 23:42:07,607 INFO org.apache.hadoop.hive.ql.Driver (SessionState.java:printInfo(377)) - Job 0: Map: 25 Reduce: 10 Cumulative CPU: 569.15 sec HDFS Read: 8125 HDFS Write: 483117997 SUCCESS 2014-06-27 23:42:07,607 INFO org.apache.hadoop.hive.ql.Driver (SessionState.java:printInfo(377)) - Job 1: Map: 5 Reduce: 10 Cumulative CPU: 532.7 sec HDFS Read: 483383876 HDFS Write: 100388623 FAIL 2014-06-27 23:42:07,607 INFO org.apache.hadoop.hive.ql.Driver (SessionState.java:printInfo(377)) - Total MapReduce CPU Time Spent: 18 minutes 21 seconds 850 msec 2014-06-27 23:42:07,608 INFO org.apache.hadoop.hive.ql.Driver (PerfLogger.java:PerfLogBegin(100)) - <PERFLOG method=releaseLocks> 2014-06-27 23:42:07,608 INFO org.apache.hadoop.hive.ql.Driver (PerfLogger.java:PerfLogEnd(127)) - </PERFLOG method=releaseLocks start=1403912527608 end=1403912527608 duration=0> 2014-06-27 23:42:07,611 INFO org.apache.hadoop.hive.ql.Driver (PerfLogger.java:PerfLogBegin(100)) - <PERFLOG method=releaseLocks> 2014-06-27 23:42:07,612 INFO org.apache.hadoop.hive.ql.Driver (PerfLogger.java:PerfLogEnd(127)) - </PERFLOG method=releaseLocks start=1403912527611 end=1403912527612 duration=1> 2014-06-27 23:42:07,614 INFO amazon.emr.metrics.MetricsSaver (MetricsSaver.java:performShutdown(423)) - Inside MetricsSaver Shutdown Hook One INFO which might help in your understanding. The two properties (mapred.max.split.size -> 25600000 & hive.merge.size.per.task -> 256000000) are different for "/home/hadoop/hive/conf/hive-default.xml" in PRD. Do they have to be same or they could be different? We have the two values same (256000000) in STG environment, where we don't face this issue. Can you please help? Thanks, Shouvanik ________________________________ This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited. Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy. ______________________________________________________________________________________ www.accenture.com
