Hi Rohit , Thanks for replying . No , I do not see any connection retry attempts to HDFS in the logs .
Also , Namenode and HDFS look healthy in our cluster . PFA latest AM logs for the job . Regards, Ashish On Mon, Jul 20, 2015 at 3:29 PM, Rohith Sharma K S < rohithsharm...@huawei.com> wrote: > Hi > > > > From thread dump, it seems waiting for HDFS operation. Can you attach AM > logs, and do you see any client retry for connecting to HDFS? > > > > "CommitterEvent Processor #4" prio=10 tid=0x000000000199a800 nid=0x18df in > Object.wait() [0x00007f4f12aa4000] > > java.lang.Thread.State: WAITING (on object monitor) > > at java.lang.Object.wait(Native Method) > > at java.lang.Object.wait(Object.java:503) > > ………………………. > > at > org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1864) > > at > org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:575) > > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:345) > > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.mergePaths(FileOutputCommitter.java:362) > > at > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.commitJob(FileOutputCommitter.java:310) > > at > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobCommit(CommitterEventHandler.java:274) > > at > org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:237) > > > > > > May be you can check from HDFS that is it Healthy? > > > > Thanks & Regards > > Rohith Sharma K S > > > > *From:* Ashish Kumar Singh [mailto:ashish23...@gmail.com] > *Sent:* 20 July 2015 14:16 > *To:* user@hadoop.apache.org > *Subject:* Application Master waits a long time after Mapper/Reducers > finish > > > > Hello Users , > > I am facing a problem running Mapreduce jobs on Hadoop 2.6. > I am observing that the Applocation Master waits for a long time after > all the Mappers and Reducers are completed before the job is completed . > > This wait time sometimes exceeds 20-25 mins which is very strange as our > mappers and reducers complete in less than 10 minutes for the job . > > Below are some observations: > a) Job completion status stands at 95% when the wait begins > > b)JOB_COMMIT is initiated just before this wait time ( logs: 2015-07-14 > 01:54:46,636 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: > job_1436854849540_0123Job Transitioned from RUNNING to COMMITTING ) > > c) job success happens after 20-25 minutes ( logs: 2015-07-14 02:15:06,634 > INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: > job_1436854849540_0123Job Transitioned from COMMITTING to SUCCEEDED ) > > > Appreciate any help on this . > > Thread dump while the Application master hangs is attached. > > Regards, > > Ashish >
2015-07-14 04:48:57,898 INFO [IPC Server handler 113 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000875_0 is : 0.8724331 2015-07-14 04:48:57,995 INFO [IPC Server handler 128 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002220_0 is : 1.0 2015-07-14 04:48:57,996 INFO [IPC Server handler 0 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_002220_0 2015-07-14 04:48:57,996 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002220_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:48:57,996 INFO [ContainerLauncher #43] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_002561 taskAttempt attempt_1436854849540_0292_m_002220_0 2015-07-14 04:48:57,996 INFO [ContainerLauncher #43] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002220_0 2015-07-14 04:48:58,001 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002220_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:48:58,001 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_002220_0 2015-07-14 04:48:58,001 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_002220 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:48:58,002 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3485 2015-07-14 04:48:58,364 INFO [IPC Server handler 34 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002712_0 is : 0.6379301 2015-07-14 04:48:58,456 INFO [IPC Server handler 24 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001016_0 is : 0.9816483 2015-07-14 04:48:58,580 INFO [IPC Server handler 60 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000260_0 is : 0.9199145 2015-07-14 04:48:58,604 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:19 AssignedReds:0 CompletedMaps:3485 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:48:58,607 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_001474 2015-07-14 04:48:58,607 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:18 AssignedReds:0 CompletedMaps:3485 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:48:58,607 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_001760_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:48:58,643 INFO [IPC Server handler 87 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001758_0 is : 0.9990311 2015-07-14 04:48:58,681 INFO [IPC Server handler 141 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001758_0 is : 0.9990311 2015-07-14 04:48:58,737 INFO [IPC Server handler 39 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000389_0 is : 0.847498 2015-07-14 04:48:58,797 INFO [IPC Server handler 25 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002201_0 is : 0.9508693 2015-07-14 04:48:58,799 INFO [IPC Server handler 106 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.21215254 2015-07-14 04:48:58,825 INFO [IPC Server handler 100 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000260_0 2015-07-14 04:48:58,825 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000260_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:48:58,825 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000260_0 given a go for committing the task output. 2015-07-14 04:48:58,826 INFO [IPC Server handler 29 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000260_0 2015-07-14 04:48:58,826 INFO [IPC Server handler 29 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000260_0:true 2015-07-14 04:48:58,830 INFO [IPC Server handler 136 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_001016_0 2015-07-14 04:48:58,830 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001016_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:48:58,830 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_001016_0 given a go for committing the task output. 2015-07-14 04:48:58,831 INFO [IPC Server handler 93 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_001016_0 2015-07-14 04:48:58,831 INFO [IPC Server handler 93 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_001016_0:true 2015-07-14 04:48:58,914 INFO [IPC Server handler 33 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_001758_0 2015-07-14 04:48:58,915 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001758_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:48:58,915 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_001758_0 given a go for committing the task output. 2015-07-14 04:48:58,915 INFO [IPC Server handler 120 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_001758_0 2015-07-14 04:48:58,915 INFO [IPC Server handler 120 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_001758_0:true 2015-07-14 04:48:58,974 INFO [IPC Server handler 16 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000874_0 is : 0.72033435 2015-07-14 04:48:58,999 INFO [IPC Server handler 3 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000260_0 is : 1.0 2015-07-14 04:48:59,000 INFO [IPC Server handler 126 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000260_0 2015-07-14 04:48:59,000 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000260_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:48:59,000 INFO [ContainerLauncher #176] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000578 taskAttempt attempt_1436854849540_0292_m_000260_0 2015-07-14 04:48:59,000 INFO [ContainerLauncher #176] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000260_0 2015-07-14 04:48:59,002 INFO [IPC Server handler 54 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001016_0 is : 1.0 2015-07-14 04:48:59,003 INFO [IPC Server handler 112 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_001016_0 2015-07-14 04:48:59,003 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001016_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:48:59,003 INFO [ContainerLauncher #214] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_001140 taskAttempt attempt_1436854849540_0292_m_001016_0 2015-07-14 04:48:59,003 INFO [ContainerLauncher #214] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_001016_0 2015-07-14 04:48:59,005 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000260_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:48:59,005 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000260_0 2015-07-14 04:48:59,005 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000260 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:48:59,005 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3486 2015-07-14 04:48:59,008 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001016_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:48:59,008 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_001016_0 2015-07-14 04:48:59,008 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_001016 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:48:59,008 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3487 2015-07-14 04:48:59,093 INFO [IPC Server handler 62 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001758_0 is : 1.0 2015-07-14 04:48:59,094 INFO [IPC Server handler 32 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_001758_0 2015-07-14 04:48:59,094 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001758_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:48:59,095 INFO [ContainerLauncher #39] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_001819 taskAttempt attempt_1436854849540_0292_m_001758_0 2015-07-14 04:48:59,095 INFO [ContainerLauncher #39] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_001758_0 2015-07-14 04:48:59,099 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001758_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:48:59,099 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_001758_0 2015-07-14 04:48:59,099 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_001758 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:48:59,099 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3488 2015-07-14 04:48:59,607 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:18 AssignedReds:0 CompletedMaps:3488 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:48:59,610 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_002561 2015-07-14 04:48:59,610 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:17 AssignedReds:0 CompletedMaps:3488 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:48:59,610 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002220_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:48:59,648 INFO [IPC Server handler 141 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000730_0 is : 0.7652104 2015-07-14 04:48:59,983 INFO [IPC Server handler 28 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.014224249 2015-07-14 04:49:00,047 INFO [IPC Server handler 121 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.6476498 2015-07-14 04:49:00,069 INFO [IPC Server handler 125 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.5852198 2015-07-14 04:49:00,094 INFO [IPC Server handler 32 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000389_1 is : 0.063531496 2015-07-14 04:49:00,137 INFO [IPC Server handler 118 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002996_0 is : 0.77514726 2015-07-14 04:49:00,187 INFO [IPC Server handler 50 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000258_0 is : 0.62848085 2015-07-14 04:49:00,612 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000578 2015-07-14 04:49:00,612 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_001819 2015-07-14 04:49:00,612 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_001140 2015-07-14 04:49:00,612 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000260_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:00,612 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:14 AssignedReds:0 CompletedMaps:3488 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:49:00,612 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_001758_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:00,612 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_001016_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:00,861 INFO [IPC Server handler 148 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000871_0 is : 0.76870865 2015-07-14 04:49:00,919 INFO [IPC Server handler 17 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000875_0 is : 0.8826311 2015-07-14 04:49:01,386 INFO [IPC Server handler 132 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002712_0 is : 0.7574839 2015-07-14 04:49:01,759 INFO [IPC Server handler 7 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000389_0 is : 0.9815821 2015-07-14 04:49:01,817 INFO [IPC Server handler 100 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002201_0 is : 0.95910144 2015-07-14 04:49:01,820 INFO [IPC Server handler 29 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.33593994 2015-07-14 04:49:01,995 INFO [IPC Server handler 128 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000874_0 is : 0.741774 2015-07-14 04:49:02,178 INFO [IPC Server handler 50 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000389_0 is : 0.9815821 2015-07-14 04:49:02,474 INFO [IPC Server handler 96 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000389_0 2015-07-14 04:49:02,475 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000389_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:02,475 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000389_0 given a go for committing the task output. 2015-07-14 04:49:02,475 INFO [IPC Server handler 79 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000389_0 2015-07-14 04:49:02,476 INFO [IPC Server handler 79 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000389_0:true 2015-07-14 04:49:02,649 INFO [IPC Server handler 141 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000389_0 is : 1.0 2015-07-14 04:49:02,649 INFO [IPC Server handler 138 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000389_0 2015-07-14 04:49:02,650 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000389_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:02,650 INFO [ContainerLauncher #54] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000947 taskAttempt attempt_1436854849540_0292_m_000389_0 2015-07-14 04:49:02,650 INFO [ContainerLauncher #54] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000389_0 2015-07-14 04:49:02,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000389_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:02,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000389_0 2015-07-14 04:49:02,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Issuing kill to other attempt attempt_1436854849540_0292_m_000389_1 2015-07-14 04:49:02,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000389 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:02,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3489 2015-07-14 04:49:02,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000389_1 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP 2015-07-14 04:49:02,654 INFO [ContainerLauncher #175] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_003654 taskAttempt attempt_1436854849540_0292_m_000389_1 2015-07-14 04:49:02,655 INFO [ContainerLauncher #175] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000389_1 2015-07-14 04:49:02,657 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000389_1 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 2015-07-14 04:49:02,657 INFO [CommitterEvent Processor #4] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-07-14 04:49:02,669 INFO [IPC Server handler 11 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000730_0 is : 0.87090003 2015-07-14 04:49:02,702 INFO [IPC Server handler 57 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000389_1 is : 0.063531496 2015-07-14 04:49:02,942 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000389_1 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 2015-07-14 04:49:03,042 INFO [IPC Server handler 6 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.026272796 2015-07-14 04:49:03,068 INFO [IPC Server handler 125 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.65941745 2015-07-14 04:49:03,093 INFO [IPC Server handler 62 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.6060043 2015-07-14 04:49:03,159 INFO [IPC Server handler 50 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002996_0 is : 0.89182484 2015-07-14 04:49:03,209 INFO [IPC Server handler 14 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000258_0 is : 0.7622019 2015-07-14 04:49:03,618 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:14 AssignedReds:0 CompletedMaps:3489 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:49:03,892 INFO [IPC Server handler 113 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000871_0 is : 0.7865482 2015-07-14 04:49:03,941 INFO [IPC Server handler 23 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000875_0 is : 0.9034152 2015-07-14 04:49:04,408 INFO [IPC Server handler 78 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002712_0 is : 0.88199115 2015-07-14 04:49:04,435 INFO [IPC Server handler 1 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002201_0 is : 0.95910144 2015-07-14 04:49:04,623 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000947 2015-07-14 04:49:04,623 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_003654 2015-07-14 04:49:04,623 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:12 AssignedReds:0 CompletedMaps:3489 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:49:04,623 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000389_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:04,623 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000389_1: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:04,750 INFO [IPC Server handler 137 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_002201_0 2015-07-14 04:49:04,750 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002201_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:04,750 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_002201_0 given a go for committing the task output. 2015-07-14 04:49:04,751 INFO [IPC Server handler 88 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_002201_0 2015-07-14 04:49:04,751 INFO [IPC Server handler 88 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_002201_0:true 2015-07-14 04:49:04,839 INFO [IPC Server handler 80 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002201_0 is : 1.0 2015-07-14 04:49:04,840 INFO [IPC Server handler 40 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.46956614 2015-07-14 04:49:04,916 INFO [IPC Server handler 120 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002201_0 is : 1.0 2015-07-14 04:49:04,917 INFO [IPC Server handler 17 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_002201_0 2015-07-14 04:49:04,917 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002201_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:04,917 INFO [ContainerLauncher #200] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_002112 taskAttempt attempt_1436854849540_0292_m_002201_0 2015-07-14 04:49:04,917 INFO [ContainerLauncher #200] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002201_0 2015-07-14 04:49:04,920 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002201_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:04,920 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_002201_0 2015-07-14 04:49:04,920 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_002201 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:04,920 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3490 2015-07-14 04:49:05,015 INFO [IPC Server handler 6 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000874_0 is : 0.80442035 2015-07-14 04:49:05,114 INFO [DefaultSpeculator background processing] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: DefaultSpeculator.addSpeculativeAttempt -- we are speculating task_1436854849540_0292_m_002882 2015-07-14 04:49:05,114 INFO [DefaultSpeculator background processing] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: We launched 1 speculations. Sleeping 15000 milliseconds. 2015-07-14 04:49:05,114 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Scheduling a redundant attempt for task task_1436854849540_0292_m_002882 2015-07-14 04:49:05,123 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved nyc2aggr284.nyc7.pubmatic.com to /default-rack 2015-07-14 04:49:05,123 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved nyc2aggr46.nyc7.pubmatic.com to /default-rack 2015-07-14 04:49:05,123 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved nyc2aggr81.nyc7.pubmatic.com to /default-rack 2015-07-14 04:49:05,123 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_1 TaskAttempt Transitioned from NEW to UNASSIGNED 2015-07-14 04:49:05,624 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:12 AssignedReds:0 CompletedMaps:3490 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:49:05,626 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1436854849540_0292: ask=5 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:22536192, vCores:-269> knownNMs=243 2015-07-14 04:49:05,691 INFO [IPC Server handler 101 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000730_0 is : 0.9780511 2015-07-14 04:49:05,810 INFO [IPC Server handler 100 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002996_0 is : 0.89182484 2015-07-14 04:49:06,089 INFO [IPC Server handler 62 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.04941036 2015-07-14 04:49:06,089 INFO [IPC Server handler 32 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.67903936 2015-07-14 04:49:06,116 INFO [IPC Server handler 59 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.6369837 2015-07-14 04:49:06,184 INFO [IPC Server handler 14 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002996_0 is : 1.0 2015-07-14 04:49:06,224 INFO [IPC Server handler 83 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_002996_0 2015-07-14 04:49:06,225 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002996_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:06,225 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_002996_0 given a go for committing the task output. 2015-07-14 04:49:06,226 INFO [IPC Server handler 73 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_002996_0 2015-07-14 04:49:06,226 INFO [IPC Server handler 73 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_002996_0:true 2015-07-14 04:49:06,230 INFO [IPC Server handler 110 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000258_0 is : 0.89713854 2015-07-14 04:49:06,294 INFO [IPC Server handler 69 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000730_0 is : 0.9780511 2015-07-14 04:49:06,502 INFO [IPC Server handler 134 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000730_0 2015-07-14 04:49:06,502 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000730_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:06,502 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000730_0 given a go for committing the task output. 2015-07-14 04:49:06,503 INFO [IPC Server handler 67 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000730_0 2015-07-14 04:49:06,503 INFO [IPC Server handler 67 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000730_0:true 2015-07-14 04:49:06,505 INFO [IPC Server handler 122 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002996_0 is : 1.0 2015-07-14 04:49:06,506 INFO [IPC Server handler 82 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_002996_0 2015-07-14 04:49:06,506 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002996_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:06,506 INFO [ContainerLauncher #226] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_003272 taskAttempt attempt_1436854849540_0292_m_002996_0 2015-07-14 04:49:06,506 INFO [ContainerLauncher #226] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002996_0 2015-07-14 04:49:06,512 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002996_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:06,512 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_002996_0 2015-07-14 04:49:06,512 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_002996 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:06,512 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3491 2015-07-14 04:49:06,626 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:12 AssignedReds:0 CompletedMaps:3491 CompletedReds:0 ContAlloc:3654 ContRel:149 HostLocal:3418 RackLocal:87 2015-07-14 04:49:06,629 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_002112 2015-07-14 04:49:06,629 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated containers 1 2015-07-14 04:49:06,629 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002201_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:06,631 INFO [RMCommunicator Allocator] org.apache.hadoop.yarn.util.RackResolver: Resolved nyc2aggr101.nyc7.pubmatic.com to /default-rack 2015-07-14 04:49:06,631 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned container container_1436854849540_0292_01_003656 to attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:06,631 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:12 AssignedReds:0 CompletedMaps:3491 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:06,631 INFO [AsyncDispatcher event handler] org.apache.hadoop.yarn.util.RackResolver: Resolved nyc2aggr101.nyc7.pubmatic.com to /default-rack 2015-07-14 04:49:06,631 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_1 TaskAttempt Transitioned from UNASSIGNED to ASSIGNED 2015-07-14 04:49:06,631 INFO [ContainerLauncher #90] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container container_1436854849540_0292_01_003656 taskAttempt attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:06,631 INFO [ContainerLauncher #90] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:06,637 INFO [ContainerLauncher #90] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle port returned by ContainerManager for attempt_1436854849540_0292_m_002882_1 : 13562 2015-07-14 04:49:06,637 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt: [attempt_1436854849540_0292_m_002882_1] using containerId: [container_1436854849540_0292_01_003656 on NM: [nyc2aggr101.nyc7.pubmatic.com:45454] 2015-07-14 04:49:06,637 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_1 TaskAttempt Transitioned from ASSIGNED to RUNNING 2015-07-14 04:49:06,637 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: ATTEMPT_START task_1436854849540_0292_m_002882 2015-07-14 04:49:06,822 INFO [IPC Server handler 136 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000730_0 is : 1.0 2015-07-14 04:49:06,823 INFO [IPC Server handler 93 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000730_0 2015-07-14 04:49:06,823 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000730_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:06,824 INFO [ContainerLauncher #238] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000936 taskAttempt attempt_1436854849540_0292_m_000730_0 2015-07-14 04:49:06,824 INFO [ContainerLauncher #238] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000730_0 2015-07-14 04:49:06,827 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000730_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:06,827 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000730_0 2015-07-14 04:49:06,827 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000730 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:06,828 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3492 2015-07-14 04:49:06,917 INFO [IPC Server handler 17 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000871_0 is : 0.81421363 2015-07-14 04:49:06,962 INFO [IPC Server handler 30 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000875_0 is : 0.96498024 2015-07-14 04:49:07,037 INFO [IPC Server handler 90 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002712_0 is : 0.88199115 2015-07-14 04:49:07,430 INFO [IPC Server handler 38 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002712_0 is : 1.0 2015-07-14 04:49:07,478 INFO [IPC Server handler 104 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_002712_0 2015-07-14 04:49:07,478 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002712_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:07,478 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_002712_0 given a go for committing the task output. 2015-07-14 04:49:07,479 INFO [IPC Server handler 42 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_002712_0 2015-07-14 04:49:07,479 INFO [IPC Server handler 42 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_002712_0:true 2015-07-14 04:49:07,631 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:12 AssignedReds:0 CompletedMaps:3492 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:07,634 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1436854849540_0292: ask=5 release= 0 newContainers=0 finishedContainers=1 resourcelimit=<memory:0, vCores:-254> knownNMs=243 2015-07-14 04:49:07,634 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_003272 2015-07-14 04:49:07,634 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002996_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:07,634 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:11 AssignedReds:0 CompletedMaps:3492 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:07,726 INFO [IPC Server handler 145 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002712_0 is : 1.0 2015-07-14 04:49:07,727 INFO [IPC Server handler 147 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_002712_0 2015-07-14 04:49:07,727 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002712_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:07,727 INFO [ContainerLauncher #232] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_002642 taskAttempt attempt_1436854849540_0292_m_002712_0 2015-07-14 04:49:07,727 INFO [ContainerLauncher #232] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002712_0 2015-07-14 04:49:07,732 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002712_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:07,732 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_002712_0 2015-07-14 04:49:07,732 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_002712 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:07,732 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3493 2015-07-14 04:49:07,825 INFO [IPC Server handler 80 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000875_0 is : 0.96498024 2015-07-14 04:49:07,862 INFO [IPC Server handler 148 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.60310376 2015-07-14 04:49:07,903 INFO [Socket Reader #1 for port 52733] SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for job_1436854849540_0292 (auth:SIMPLE) 2015-07-14 04:49:07,914 INFO [IPC Server handler 33 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID : jvm_1436854849540_0292_m_003656 asked for a task 2015-07-14 04:49:07,914 INFO [IPC Server handler 33 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID: jvm_1436854849540_0292_m_003656 given task: attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:08,036 INFO [IPC Server handler 90 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000874_0 is : 0.93850434 2015-07-14 04:49:08,143 INFO [IPC Server handler 50 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000875_0 2015-07-14 04:49:08,144 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000875_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:08,144 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000875_0 given a go for committing the task output. 2015-07-14 04:49:08,145 INFO [IPC Server handler 14 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000875_0 2015-07-14 04:49:08,145 INFO [IPC Server handler 14 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000875_0:true 2015-07-14 04:49:08,308 INFO [IPC Server handler 124 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000875_0 is : 1.0 2015-07-14 04:49:08,309 INFO [IPC Server handler 109 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000875_0 2015-07-14 04:49:08,309 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000875_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:08,309 INFO [ContainerLauncher #166] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000551 taskAttempt attempt_1436854849540_0292_m_000875_0 2015-07-14 04:49:08,309 INFO [ContainerLauncher #166] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000875_0 2015-07-14 04:49:08,313 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000875_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:08,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000875_0 2015-07-14 04:49:08,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000875 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:08,314 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3494 2015-07-14 04:49:08,634 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:11 AssignedReds:0 CompletedMaps:3494 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:08,637 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000936 2015-07-14 04:49:08,637 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:10 AssignedReds:0 CompletedMaps:3494 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:08,637 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000730_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:09,111 INFO [IPC Server handler 2 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.6982545 2015-07-14 04:49:09,134 INFO [IPC Server handler 118 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.11071447 2015-07-14 04:49:09,138 INFO [IPC Server handler 50 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.7512751 2015-07-14 04:49:09,251 INFO [IPC Server handler 61 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000258_0 is : 0.99649924 2015-07-14 04:49:09,339 INFO [IPC Server handler 18 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000258_0 is : 0.99649924 2015-07-14 04:49:09,400 INFO [IPC Server handler 78 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000874_0 is : 0.93850434 2015-07-14 04:49:09,640 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_002642 2015-07-14 04:49:09,640 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000551 2015-07-14 04:49:09,640 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002712_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:09,640 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:8 AssignedReds:0 CompletedMaps:3494 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:09,640 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000875_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:09,648 INFO [IPC Server handler 141 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000258_0 2015-07-14 04:49:09,649 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000258_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:09,649 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000258_0 given a go for committing the task output. 2015-07-14 04:49:09,650 INFO [IPC Server handler 138 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000258_0 2015-07-14 04:49:09,650 INFO [IPC Server handler 138 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000258_0:true 2015-07-14 04:49:09,735 INFO [IPC Server handler 89 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000874_0 2015-07-14 04:49:09,735 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000874_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:09,735 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000874_0 given a go for committing the task output. 2015-07-14 04:49:09,736 INFO [IPC Server handler 39 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000874_0 2015-07-14 04:49:09,736 INFO [IPC Server handler 39 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000874_0:true 2015-07-14 04:49:09,942 INFO [IPC Server handler 30 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000871_0 is : 0.9226574 2015-07-14 04:49:09,980 INFO [IPC Server handler 105 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000258_0 is : 1.0 2015-07-14 04:49:09,981 INFO [IPC Server handler 28 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000258_0 2015-07-14 04:49:09,981 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000258_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:09,981 INFO [ContainerLauncher #65] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000565 taskAttempt attempt_1436854849540_0292_m_000258_0 2015-07-14 04:49:09,981 INFO [ContainerLauncher #65] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000258_0 2015-07-14 04:49:09,984 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000258_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:09,984 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000258_0 2015-07-14 04:49:09,984 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000258 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:09,984 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3495 2015-07-14 04:49:10,096 INFO [IPC Server handler 2 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000874_0 is : 1.0 2015-07-14 04:49:10,097 INFO [IPC Server handler 15 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000874_0 2015-07-14 04:49:10,097 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000874_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:10,097 INFO [ContainerLauncher #187] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000740 taskAttempt attempt_1436854849540_0292_m_000874_0 2015-07-14 04:49:10,097 INFO [ContainerLauncher #187] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000874_0 2015-07-14 04:49:10,101 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000874_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:10,101 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000874_0 2015-07-14 04:49:10,101 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000874 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:10,101 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3496 2015-07-14 04:49:10,640 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:8 AssignedReds:0 CompletedMaps:3496 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:10,882 INFO [IPC Server handler 114 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.7319748 2015-07-14 04:49:11,646 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000565 2015-07-14 04:49:11,646 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000740 2015-07-14 04:49:11,646 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000258_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:11,646 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:6 AssignedReds:0 CompletedMaps:3496 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:11,646 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000874_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:11,777 INFO [IPC Server handler 13 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000871_0 is : 0.9226574 2015-07-14 04:49:12,133 INFO [IPC Server handler 118 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.765092 2015-07-14 04:49:12,139 INFO [IPC Server handler 50 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_000871_0 2015-07-14 04:49:12,139 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000871_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:12,139 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_000871_0 given a go for committing the task output. 2015-07-14 04:49:12,140 INFO [IPC Server handler 14 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_000871_0 2015-07-14 04:49:12,140 INFO [IPC Server handler 14 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_000871_0:true 2015-07-14 04:49:12,160 INFO [IPC Server handler 83 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.87562823 2015-07-14 04:49:12,179 INFO [IPC Server handler 73 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.1649313 2015-07-14 04:49:12,470 INFO [IPC Server handler 4 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_000871_0 is : 1.0 2015-07-14 04:49:12,471 INFO [IPC Server handler 84 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_000871_0 2015-07-14 04:49:12,471 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000871_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:12,471 INFO [ContainerLauncher #40] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_000988 taskAttempt attempt_1436854849540_0292_m_000871_0 2015-07-14 04:49:12,471 INFO [ContainerLauncher #40] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_000871_0 2015-07-14 04:49:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_000871_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_000871_0 2015-07-14 04:49:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_000871 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:12,477 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3497 2015-07-14 04:49:12,646 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:6 AssignedReds:0 CompletedMaps:3497 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:13,654 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_000988 2015-07-14 04:49:13,654 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:5 AssignedReds:0 CompletedMaps:3497 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:13,654 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_000871_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:13,903 INFO [IPC Server handler 33 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.8669355 2015-07-14 04:49:14,899 INFO [IPC Server handler 33 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_1 is : 0.15518296 2015-07-14 04:49:15,154 INFO [IPC Server handler 83 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.8950157 2015-07-14 04:49:15,182 INFO [IPC Server handler 110 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.9970157 2015-07-14 04:49:15,220 INFO [IPC Server handler 61 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.23281786 2015-07-14 04:49:15,259 INFO [IPC Server handler 76 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 0.9970157 2015-07-14 04:49:15,464 INFO [IPC Server handler 74 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_002882_0 2015-07-14 04:49:15,464 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:15,464 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_002882_0 given a go for committing the task output. 2015-07-14 04:49:15,465 INFO [IPC Server handler 146 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_002882_0 2015-07-14 04:49:15,465 INFO [IPC Server handler 146 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_002882_0:true 2015-07-14 04:49:15,774 INFO [IPC Server handler 13 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_0 is : 1.0 2015-07-14 04:49:15,775 INFO [IPC Server handler 149 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_002882_0 2015-07-14 04:49:15,775 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:15,775 INFO [ContainerLauncher #194] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_002918 taskAttempt attempt_1436854849540_0292_m_002882_0 2015-07-14 04:49:15,775 INFO [ContainerLauncher #194] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002882_0 2015-07-14 04:49:15,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:15,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_002882_0 2015-07-14 04:49:15,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Issuing kill to other attempt attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:15,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_002882 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:15,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3498 2015-07-14 04:49:15,780 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_1 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP 2015-07-14 04:49:15,780 INFO [ContainerLauncher #191] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_003656 taskAttempt attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:15,781 INFO [ContainerLauncher #191] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002882_1 2015-07-14 04:49:15,783 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_1 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 2015-07-14 04:49:15,783 INFO [CommitterEvent Processor #0] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-07-14 04:49:15,801 INFO [IPC Server handler 100 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002882_1 is : 0.15518296 2015-07-14 04:49:16,038 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002882_1 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 2015-07-14 04:49:16,044 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Size of the JobHistory event queue is 0 2015-07-14 04:49:16,660 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:5 AssignedReds:0 CompletedMaps:3498 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:16,903 INFO [IPC Server handler 120 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 0.8669355 2015-07-14 04:49:16,925 INFO [IPC Server handler 23 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 1.0 2015-07-14 04:49:17,088 INFO [IPC Server handler 62 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_002313_0 2015-07-14 04:49:17,088 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002313_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:17,088 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_002313_0 given a go for committing the task output. 2015-07-14 04:49:17,089 INFO [IPC Server handler 32 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_002313_0 2015-07-14 04:49:17,089 INFO [IPC Server handler 32 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_002313_0:true 2015-07-14 04:49:17,391 INFO [IPC Server handler 78 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_0 is : 1.0 2015-07-14 04:49:17,392 INFO [IPC Server handler 41 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_002313_0 2015-07-14 04:49:17,392 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002313_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:17,392 INFO [ContainerLauncher #147] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_002475 taskAttempt attempt_1436854849540_0292_m_002313_0 2015-07-14 04:49:17,392 INFO [ContainerLauncher #147] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002313_0 2015-07-14 04:49:17,397 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002313_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:17,397 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_002313_0 2015-07-14 04:49:17,397 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Issuing kill to other attempt attempt_1436854849540_0292_m_002313_1 2015-07-14 04:49:17,397 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_002313 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:17,397 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3499 2015-07-14 04:49:17,398 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002313_1 TaskAttempt Transitioned from RUNNING to KILL_CONTAINER_CLEANUP 2015-07-14 04:49:17,398 INFO [ContainerLauncher #196] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_003655 taskAttempt attempt_1436854849540_0292_m_002313_1 2015-07-14 04:49:17,399 INFO [ContainerLauncher #196] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_002313_1 2015-07-14 04:49:17,403 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002313_1 TaskAttempt Transitioned from KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP 2015-07-14 04:49:17,403 INFO [CommitterEvent Processor #1] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: TASK_ABORT 2015-07-14 04:49:17,421 INFO [IPC Server handler 21 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_002313_1 is : 0.23281786 2015-07-14 04:49:17,449 INFO [IPC Server handler 24 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 0.8950157 2015-07-14 04:49:17,547 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_002313_1 TaskAttempt Transitioned from KILL_TASK_CLEANUP to KILLED 2015-07-14 04:49:17,571 INFO [IPC Server handler 66 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update from attempt_1436854849540_0292_m_001102_0 2015-07-14 04:49:17,571 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001102_0 TaskAttempt Transitioned from RUNNING to COMMIT_PENDING 2015-07-14 04:49:17,571 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: attempt_1436854849540_0292_m_001102_0 given a go for committing the task output. 2015-07-14 04:49:17,572 INFO [IPC Server handler 94 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request from attempt_1436854849540_0292_m_001102_0 2015-07-14 04:49:17,572 INFO [IPC Server handler 94 on 52733] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for attempt_1436854849540_0292_m_001102_0:true 2015-07-14 04:49:17,662 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:5 AssignedReds:0 CompletedMaps:3499 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:17,664 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_003656 2015-07-14 04:49:17,664 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_002918 2015-07-14 04:49:17,664 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:3499 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:17,664 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002882_1: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:17,664 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002882_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:17,858 INFO [IPC Server handler 108 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt attempt_1436854849540_0292_m_001102_0 is : 1.0 2015-07-14 04:49:17,859 INFO [IPC Server handler 148 on 52733] org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from attempt_1436854849540_0292_m_001102_0 2015-07-14 04:49:17,859 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001102_0 TaskAttempt Transitioned from COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP 2015-07-14 04:49:17,859 INFO [ContainerLauncher #56] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing the event EventType: CONTAINER_REMOTE_CLEANUP for container container_1436854849540_0292_01_001174 taskAttempt attempt_1436854849540_0292_m_001102_0 2015-07-14 04:49:17,859 INFO [ContainerLauncher #56] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING attempt_1436854849540_0292_m_001102_0 2015-07-14 04:49:17,864 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1436854849540_0292_m_001102_0 TaskAttempt Transitioned from SUCCESS_CONTAINER_CLEANUP to SUCCEEDED 2015-07-14 04:49:17,865 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with attempt attempt_1436854849540_0292_m_001102_0 2015-07-14 04:49:17,865 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1436854849540_0292_m_001102 Task Transitioned from RUNNING to SUCCEEDED 2015-07-14 04:49:17,865 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 3500 2015-07-14 04:49:17,865 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1436854849540_0292Job Transitioned from RUNNING to COMMITTING 2015-07-14 04:49:17,865 INFO [CommitterEvent Processor #2] org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_COMMIT 2015-07-14 04:49:18,664 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:3 AssignedReds:0 CompletedMaps:3500 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:18,668 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_002475 2015-07-14 04:49:18,668 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_003655 2015-07-14 04:49:18,668 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1 AssignedReds:0 CompletedMaps:3500 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:18,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002313_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:18,668 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_002313_1: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 04:49:19,671 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received completed container container_1436854849540_0292_01_001174 2015-07-14 04:49:19,671 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:3500 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 04:49:19,671 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics report from attempt_1436854849540_0292_m_001102_0: Container killed by the ApplicationMaster. Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 2015-07-14 05:03:20,343 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Calling handler for JobFinishedEvent 2015-07-14 05:03:20,344 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1436854849540_0292Job Transitioned from COMMITTING to SUCCEEDED 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: We are finishing cleanly so this is the last retry 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify RMCommunicator isAMLastRetry: true 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: RMCommunicator notified that shouldUnregistered is: true 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Notify JHEH isAMLastRetry: true 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: JobHistoryEventHandler notified that forceJobCompletion is true 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Calling stop for all the services 2015-07-14 05:03:20,345 INFO [Thread-1699] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0 2015-07-14 05:03:21,304 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://Matrix-Aggr:8020/tmp/hadoop-yarn/staging/hdfs/.staging/job_1436854849540_0292/job_1436854849540_0292_1.jhist to hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292-1436874310195-hdfs-%5BBED1644E4EC647F98A827FCDEF5AB4FD%2F2DA3E00A49FD4557-1436875400308-3500-0-SUCCEEDED-default-1436874315104.jhist_tmp 2015-07-14 05:03:22,208 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292-1436874310195-hdfs-%5BBED1644E4EC647F98A827FCDEF5AB4FD%2F2DA3E00A49FD4557-1436875400308-3500-0-SUCCEEDED-default-1436874315104.jhist_tmp 2015-07-14 05:03:22,608 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copying hdfs://Matrix-Aggr:8020/tmp/hadoop-yarn/staging/hdfs/.staging/job_1436854849540_0292/job_1436854849540_0292_1_conf.xml to hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292_conf.xml_tmp 2015-07-14 05:03:23,191 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Copied to done location: hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292_conf.xml_tmp 2015-07-14 05:03:23,525 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292.summary_tmp to hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292.summary 2015-07-14 05:03:23,883 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292_conf.xml_tmp to hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292_conf.xml 2015-07-14 05:03:24,050 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Moved tmp to done: hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292-1436874310195-hdfs-%5BBED1644E4EC647F98A827FCDEF5AB4FD%2F2DA3E00A49FD4557-1436875400308-3500-0-SUCCEEDED-default-1436874315104.jhist_tmp to hdfs://Matrix-Aggr:8020/mr-history/tmp/hdfs/job_1436854849540_0292-1436874310195-hdfs-%5BBED1644E4EC647F98A827FCDEF5AB4FD%2F2DA3E00A49FD4557-1436875400308-3500-0-SUCCEEDED-default-1436874315104.jhist 2015-07-14 05:03:24,051 INFO [Thread-1699] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopped JobHistoryEventHandler. super.stop() 2015-07-14 05:03:24,054 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Setting job diagnostics to 2015-07-14 05:03:24,061 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: History url is http://nyc2aggr180.nyc7.pubmatic.com:19888/jobhistory/job/job_1436854849540_0292 2015-07-14 05:03:24,067 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Waiting for application to be successfully unregistered. 2015-07-14 05:03:25,068 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Final Stats: PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:3500 CompletedReds:0 ContAlloc:3655 ContRel:149 HostLocal:3418 RackLocal:88 2015-07-14 05:03:25,069 INFO [Thread-1699] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Deleting staging directory hdfs://Matrix-Aggr /tmp/hadoop-yarn/staging/hdfs/.staging/job_1436854849540_0292 2015-07-14 05:03:25,210 INFO [Thread-1699] org.apache.hadoop.ipc.Server: Stopping server on 52733 2015-07-14 05:03:25,211 INFO [IPC Server listener on 52733] org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 52733 2015-07-14 05:03:25,213 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2015-07-14 05:03:25,213 INFO [TaskHeartbeatHandler PingChecker] org.apache.hadoop.mapreduce.v2.app.TaskHeartbeatHandler: TaskHeartbeatHandler thread interrupted </pre> </td> </tr> </tbody> </table> </html>
