[ https://issues.apache.org/jira/browse/TEZ-1984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14285346#comment-14285346 ]
tangjunjie commented on TEZ-1984: --------------------------------- 2015-01-22 00:27:13,603 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 225.7 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:13,621 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 13064 for container-id container_1421076019070_0161_01_000003: 131.8 MB of 1 GB physical memory used; 1.4 GB of 2.1 GB virtual memory used 2015-01-22 00:27:15,138 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1421076019070_0161_01_000003 is : 255 2015-01-22 00:27:15,138 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1421076019070_0161_01_000003 and exit code: 255 ExitCodeException exitCode=255: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1421076019070_0161_01_000003 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 255 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=255: 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:455) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-01-22 00:27:15,139 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2015-01-22 00:27:15,140 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 255 2015-01-22 00:27:15,140 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000003 transitioned from RUNNING to EXITED_WITH_FAILURE 2015-01-22 00:27:15,140 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1421076019070_0161_01_000003 2015-01-22 00:27:15,157 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1421076019070_0161 CONTAINERID=container_1421076019070_0161_01_000003 2015-01-22 00:27:15,157 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/hdfs/appcache/application_1421076019070_0161/container_1421076019070_0161_01_000003 2015-01-22 00:27:15,157 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000003 transitioned from EXITED_WITH_FAILURE to DONE 2015-01-22 00:27:15,157 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1421076019070_0161_01_000003 from application application_1421076019070_0161 2015-01-22 00:27:15,157 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1421076019070_0161_01_000003 for log-aggregation 2015-01-22 00:27:15,157 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1421076019070_0161 2015-01-22 00:27:16,622 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1421076019070_0161_01_000003 2015-01-22 00:27:16,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 225.8 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:18,140 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1421076019070_0161_01_000003] 2015-01-22 00:27:18,543 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1421076019070_0161_01_000004 by user hdfs 2015-01-22 00:27:18,544 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=127.0.0.1 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1421076019070_0161 CONTAINERID=container_1421076019070_0161_01_000004 2015-01-22 00:27:18,544 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1421076019070_0161_01_000004 to application application_1421076019070_0161 2015-01-22 00:27:18,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000004 transitioned from NEW to LOCALIZING 2015-01-22 00:27:18,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1421076019070_0161 2015-01-22 00:27:18,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1421076019070_0161 2015-01-22 00:27:18,548 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle 2015-01-22 00:27:18,549 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1421076019070_0161 2015-01-22 00:27:18,556 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000004 transitioned from LOCALIZING to LOCALIZED 2015-01-22 00:27:18,575 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000004 transitioned from LOCALIZED to RUNNING 2015-01-22 00:27:18,578 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /yarn/nm/usercache/hdfs/appcache/application_1421076019070_0161/container_1421076019070_0161_01_000004/default_container_executor.sh] 2015-01-22 00:27:19,644 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1421076019070_0161_01_000004 2015-01-22 00:27:19,665 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 225.9 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:19,683 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 13200 for container-id container_1421076019070_0161_01_000004: 136.1 MB of 1 GB physical memory used; 1.4 GB of 2.1 GB virtual memory used 2015-01-22 00:27:21,168 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1421076019070_0161_01_000004 is : 255 2015-01-22 00:27:21,168 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1421076019070_0161_01_000004 and exit code: 255 ExitCodeException exitCode=255: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-01-22 00:27:21,169 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1421076019070_0161_01_000004 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 255 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=255: 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:455) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-01-22 00:27:21,170 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2015-01-22 00:27:21,170 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 255 2015-01-22 00:27:21,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000004 transitioned from RUNNING to EXITED_WITH_FAILURE 2015-01-22 00:27:21,171 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1421076019070_0161_01_000004 2015-01-22 00:27:21,192 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/hdfs/appcache/application_1421076019070_0161/container_1421076019070_0161_01_000004 2015-01-22 00:27:21,192 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1421076019070_0161 CONTAINERID=container_1421076019070_0161_01_000004 2015-01-22 00:27:21,192 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000004 transitioned from EXITED_WITH_FAILURE to DONE 2015-01-22 00:27:21,192 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1421076019070_0161_01_000004 from application application_1421076019070_0161 2015-01-22 00:27:21,192 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1421076019070_0161_01_000004 for log-aggregation 2015-01-22 00:27:21,193 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1421076019070_0161 2015-01-22 00:27:22,684 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1421076019070_0161_01_000004 2015-01-22 00:27:22,706 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 227.9 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:24,148 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1421076019070_0161_01_000004] 2015-01-22 00:27:24,555 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl: Start request for container_1421076019070_0161_01_000005 by user hdfs 2015-01-22 00:27:24,557 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1421076019070_0161_01_000005 to application application_1421076019070_0161 2015-01-22 00:27:24,557 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs IP=127.0.0.1 OPERATION=Start Container Request TARGET=ContainerManageImpl RESULT=SUCCESS APPID=application_1421076019070_0161 CONTAINERID=container_1421076019070_0161_01_000005 2015-01-22 00:27:24,560 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000005 transitioned from NEW to LOCALIZING 2015-01-22 00:27:24,560 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_INIT for appId application_1421076019070_0161 2015-01-22 00:27:24,561 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event APPLICATION_INIT for appId application_1421076019070_0161 2015-01-22 00:27:24,561 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got APPLICATION_INIT for service mapreduce_shuffle 2015-01-22 00:27:24,561 INFO org.apache.hadoop.mapred.ShuffleHandler: Added token for job_1421076019070_0161 2015-01-22 00:27:24,568 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000005 transitioned from LOCALIZING to LOCALIZED 2015-01-22 00:27:24,586 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000005 transitioned from LOCALIZED to RUNNING 2015-01-22 00:27:24,589 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: launchContainer: [nice, -n, 0, bash, /yarn/nm/usercache/hdfs/appcache/application_1421076019070_0161/container_1421076019070_0161_01_000005/default_container_executor.sh] 2015-01-22 00:27:25,706 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Starting resource-monitoring for container_1421076019070_0161_01_000005 2015-01-22 00:27:25,724 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 228.0 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:25,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 13332 for container-id container_1421076019070_0161_01_000005: 136.5 MB of 1 GB physical memory used; 1.4 GB of 2.1 GB virtual memory used 2015-01-22 00:27:27,140 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1421076019070_0161_01_000005 is : 255 2015-01-22 00:27:27,140 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1421076019070_0161_01_000005 and exit code: 255 ExitCodeException exitCode=255: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) at org.apache.hadoop.util.Shell.run(Shell.java:455) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch. 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1421076019070_0161_01_000005 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 255 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=255: 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell.run(Shell.java:455) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.FutureTask.run(FutureTask.java:262) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 2015-01-22 00:27:27,141 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: at java.lang.Thread.run(Thread.java:745) 2015-01-22 00:27:27,142 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 255 2015-01-22 00:27:27,142 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000005 transitioned from RUNNING to EXITED_WITH_FAILURE 2015-01-22 00:27:27,142 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Cleaning up container container_1421076019070_0161_01_000005 2015-01-22 00:27:27,158 INFO org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Deleting absolute path : /yarn/nm/usercache/hdfs/appcache/application_1421076019070_0161/container_1421076019070_0161_01_000005 2015-01-22 00:27:27,158 WARN org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=hdfs OPERATION=Container Finished - Failed TARGET=ContainerImpl RESULT=FAILURE DESCRIPTION=Container failed with state: EXITED_WITH_FAILURE APPID=application_1421076019070_0161 CONTAINERID=container_1421076019070_0161_01_000005 2015-01-22 00:27:27,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1421076019070_0161_01_000005 transitioned from EXITED_WITH_FAILURE to DONE 2015-01-22 00:27:27,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Removing container_1421076019070_0161_01_000005 from application application_1421076019070_0161 2015-01-22 00:27:27,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.AppLogAggregatorImpl: Considering container container_1421076019070_0161_01_000005 for log-aggregation 2015-01-22 00:27:27,158 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices: Got event CONTAINER_STOP for appId application_1421076019070_0161 2015-01-22 00:27:28,742 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Stopping resource-monitoring for container_1421076019070_0161_01_000005 2015-01-22 00:27:28,760 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 232.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:30,158 INFO org.apache.hadoop.yarn.server.nodemanager.NodeStatusUpdaterImpl: Removed completed containers from NM context: [container_1421076019070_0161_01_000005] 2015-01-22 00:27:31,784 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.3 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:34,810 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:37,833 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:40,860 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:43,889 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:46,913 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:49,934 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:52,960 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:55,983 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:27:59,010 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 234.2 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:28:02,039 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 248.3 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used 2015-01-22 00:28:05,063 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: Memory usage of ProcessTree 12730 for container-id container_1421076019070_0161_01_000001: 248.3 MB of 1.5 GB physical memory used; 1.7 GB of 3.1 GB virtual memory used > hive0.13.1+tez-0.4.1-incubating+hadoop-2.5.0-cdh5.2.0 run query error > --------------------------------------------------------------------- > > Key: TEZ-1984 > URL: https://issues.apache.org/jira/browse/TEZ-1984 > Project: Apache Tez > Issue Type: Bug > Affects Versions: 0.4.1 > Environment: hive0.13.1+tez-0.4.1-incubating+hadoop-2.5.0-cdh5.2.0 > Reporter: tangjunjie > > hive> select count(*) from u_data; > Total jobs = 1 > Launching Job 1 out of 1 > Status: Running (application id: application_1421076019070_0156) > Map 1: -/- Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Map 1: 0/1 Reducer 2: 0/1 > Status: Failed > Vertex failed, vertexName=Map 1, vertexId=vertex_1421076019070_0156_1_01, > diagnostics=[Task failed, taskId=task_1421076019070_0156_1_01_000000, > diagnostics=[AttemptID:attempt_1421076019070_0156_1_01_000000_0 > Info:Container container_1421076019070_0156_01_000002 COMPLETED with > diagnostics set to [Exception from container-launch. > Container id: container_1421076019070_0156_01_000002 > Exit code: 255 > Stack trace: ExitCodeException exitCode=255: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 255 > ], AttemptID:attempt_1421076019070_0156_1_01_000000_1 Info:Container > container_1421076019070_0156_01_000003 COMPLETED with diagnostics set to > [Exception from container-launch. > Container id: container_1421076019070_0156_01_000003 > Exit code: 255 > Stack trace: ExitCodeException exitCode=255: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 255 > ], AttemptID:attempt_1421076019070_0156_1_01_000000_2 Info:Container > container_1421076019070_0156_01_000004 COMPLETED with diagnostics set to > [Exception from container-launch. > Container id: container_1421076019070_0156_01_000004 > Exit code: 255 > Stack trace: ExitCodeException exitCode=255: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 255 > ], AttemptID:attempt_1421076019070_0156_1_01_000000_3 Info:Container > container_1421076019070_0156_01_000005 COMPLETED with diagnostics set to > [Exception from container-launch. > Container id: container_1421076019070_0156_01_000005 > Exit code: 255 > Stack trace: ExitCodeException exitCode=255: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Container exited with a non-zero exit code 255 > ]], Vertex failed as one or more tasks failed. failedTasks:1] > Vertex killed, vertexName=Reducer 2, vertexId=vertex_1421076019070_0156_1_00, > diagnostics=[Vertex received Kill while in RUNNING state., Vertex killed as > other vertex failed. failedTasks:0] > DAG failed due to vertex failure. failedVertices:1 killedVertices:1 > FAILED: Execution Error, return code 2 from > org.apache.hadoop.hive.ql.exec.tez.TezTask -- This message was sent by Atlassian JIRA (v6.3.4#6332)