[
https://issues.apache.org/jira/browse/MAPREDUCE-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14561498#comment-14561498
]
Rahul Jain commented on MAPREDUCE-5255:
---------------------------------------
Bob,
I have moved to a different organization in the last year and unfortunately do
not have the test environment where this problem was seen; using hadoop 2.2+
versions in production.
If no one else is encountering this issue, we can mark it duplicate of
MAPREDUCE-5009 and resolve it.
> Reduce task preemption results in task never completing , incomplete fix to
> MAPREDUCE-3858 ?
> ---------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-5255
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5255
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 2.0.3-alpha
> Reporter: Rahul Jain
>
> The problem was seen with symptoms very similar to MAPREDUCE-3858: the job is
> hung with continuous reduce task attempts, each attempt getting killed around
> commit phase.
> After a while the single reduce task was the only one remaining in the job,
> with 50K 'kills' done for the task.
> Relevant logs from application master:
> (the problem task is: attempt_1368653326922_0080_r_001278_0
> {code}
> 2013-05-16 19:27:19,891 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Going to preempt 3
> 2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,892 INFO [IPC Server handler 22 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001266_0 is : 0.7212161
> 2013-05-16 19:27:19,893 INFO [IPC Server handler 13 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001266_0 TaskAttempt Transitioned from
> COMMIT_PENDING to SUCCESS_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #19]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001296 taskAttempt
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #19]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Preempting
> attempt_1368653326922_0080_r_001277_0
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001279_0 TaskAttempt Transitioned from RUNNING
> to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:3 ScheduledReds:0 AssignedMaps:0 AssignedReds:63
> CompletedMaps:16 CompletedReds:1233 ContAlloc:1324 ContRel:25 HostLocal:2
> RackLocal:17
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_0 TaskAttempt Transitioned from
> COMMIT_PENDING to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001277_0 TaskAttempt Transitioned from RUNNING
> to KILL_CONTAINER_CLEANUP
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #10]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001311 taskAttempt
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #10]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001279_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001310 taskAttempt
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #2]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001309 taskAttempt
> attempt_1368653326922_0080_r_001277_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #4]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001278_0
> 2013-05-16 19:27:19,893 INFO [ContainerLauncher #2]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001277_0
> 2013-05-16 19:27:19,894 INFO [IPC Server handler 29 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001225_0
> 2013-05-16 19:27:19,894 INFO [IPC Server handler 29 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001225_0 is : 0.7158718
> 2013-05-16 19:27:19,894 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001266_0 TaskAttempt Transitioned from
> SUCCESS_CONTAINER_CLEANUP to SUCCEEDED
> 2013-05-16 19:27:19,895 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Task succeeded with
> attempt attempt_1368653326922_0080_r_001266_0
> 2013-05-16 19:27:19,895 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> task_1368653326922_0080_r_001266 Task Transitioned from RUNNING to SUCCEEDED
> 2013-05-16 19:27:19,895 INFO [IPC Server handler 26 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001220_0
> 2013-05-16 19:27:19,895 INFO [IPC Server handler 24 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
> attempt_1368653326922_0080_r_001225_0
> 2013-05-16 19:27:19,895 INFO [IPC Server handler 26 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001220_0 is : 0.7038309
> 2013-05-16 19:27:19,896 INFO [IPC Server handler 18 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Done acknowledgement from
> attempt_1368653326922_0080_r_001220_0
> 2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001279_0 TaskAttempt Transitioned from
> KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
> 2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Num completed Tasks: 1250
> 2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001277_0 TaskAttempt Transitioned from
> KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
> 2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_0 TaskAttempt Transitioned from
> KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
> 2013-05-16 19:27:19,897 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001225_0 TaskAtte
> {code}
> Afterwards, we see tasks repeatedly scheduled and killed with the following
> message:
> {code}
> e.v2.app.launcher.ContainerLauncherImpl: Launching
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:20:54,587 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1368653326922_0080_r_001278_311
> : 8080
> 2013-05-16 20:20:54,587 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1368653326922_0080_r_001278_311] using containerId:
> [container_1368653326922_0080_01_001683 on NM:
> [sjc1-eng-perf-g1-grid01.carrieriq.com:48462]
> 2013-05-16 20:20:54,587 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2013-05-16 20:20:56,320 INFO [IPC Server handler 8 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1368653326922_0080_r_001683 asked for a task
> 2013-05-16 20:20:56,321 INFO [IPC Server handler 8 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1368653326922_0080_r_001683 given task:
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:20:57,334 INFO [IPC Server handler 9 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request
> from attempt_1368653326922_0080_r_001278_311. startIndex 0 maxEvents 2343
> 2013-05-16 20:20:57,549 INFO [IPC Server handler 21 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:20:57,549 INFO [IPC Server handler 21 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001278_311 is : 0.0
> 2013-05-16 20:20:57,886 INFO [IPC Server handler 15 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:20:57,886 INFO [IPC Server handler 15 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001278_311 is : 0.0
> 2013-05-16 20:21:00,189 INFO [IPC Server handler 0 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:21:01,766 INFO [IPC Server handler 28 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update
> from attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:21:01,767 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from RUNNING
> to COMMIT_PENDING
> 2013-05-16 20:21:01,767 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> attempt_1368653326922_0080_r_001278_0 already given a go for committing the
> task output, so killing attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:21:01,767 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from
> COMMIT_PENDING to KILL_CONTAINER_CLEANUP
> 2013-05-16 20:21:01,767 INFO [ContainerLauncher #14]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001683 taskAttempt
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:21:01,767 INFO [ContainerLauncher #14]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:21:01,767 INFO [IPC Server handler 20 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request
> from attempt_1368653326922_0080_r_001278_311
> 2013-05-16 20:21:01,767 INFO [IPC Server handler 20 on 40095]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for
> attempt_1368653326922_0080_r_001278_311:false
> 2013-05-16 20:21:01,769 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from
> KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
> 2013-05-16 20:21:01,769 INFO [TaskCleaner #3]
> org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl: Processing the
> event EventType: TASK_CLEAN
> 2013-05-16 20:21:01,780 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_311 TaskAttempt Transitioned from
> KILL_TASK_CLEANUP to KILLED
> 2013-05-16 20:21:01,780 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_312 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2013-05-16 20:21:02,607 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1682 ContRel:67
> HostLocal:2 RackLocal:17
> 2013-05-16 20:21:02,608 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=1544192
> 2013-05-16 20:21:02,608 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps
> assigned. Ramping up all remaining reduces:1
> 2013-05-16 20:21:02,608 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1
> CompletedMaps:19 CompletedReds:1279 ContAlloc:1682 ContRel:67 HostLocal:2
> RackLocal:17
> 2013-05-16 20:21:03,611 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1368653326922_0080: ask=1 release= 0 newContainers=0
> finishedContainers=1 resourcelimit=memory: 1544192 knownNMs=20
> 2013-05-16 20:21:03,611 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1368653326922_0080_01_001683
> 2013-05-16 20:21:03,611 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0
> CompletedMaps:19 CompletedReds:1279 ContAlloc:1682 ContRel:67 HostLocal:2
> RackLocal:17
> 2013-05-16 20:21:03,611 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1368653326922_0080_r_001278_311: Container killed by the
> ApplicationMaster.
> {code}
> another one...
> {code}
> skAttemptImpl: attempt_1368653326922_0080_r_001278_377 TaskAttempt
> Transitioned from UNASSIGNED to ASSIGNED
> 2013-05-16 20:32:07,857 INFO [ContainerLauncher #17]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1368653326922_0080_01_001753 taskAttempt
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:07,857 INFO [ContainerLauncher #17]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:07,859 INFO [ContainerLauncher #17]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1368653326922_0080_r_001278_377
> : 8080
> 2013-05-16 20:32:07,860 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1368653326922_0080_r_001278_377] using containerId:
> [container_1368653326922_0080_01_001753 on NM:
> [sjc1-eng-perf-g1-grid03.carrieriq.com:58209]
> 2013-05-16 20:32:07,860 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> 2013-05-16 20:32:09,596 INFO [IPC Server handler 18 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1368653326922_0080_r_001753 asked for a task
> 2013-05-16 20:32:09,596 INFO [IPC Server handler 18 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1368653326922_0080_r_001753 given task:
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:10,634 INFO [IPC Server handler 7 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: MapCompletionEvents request
> from attempt_1368653326922_0080_r_001278_377. startIndex 0 maxEvents 2343
> 2013-05-16 20:32:10,871 INFO [IPC Server handler 2 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:10,871 INFO [IPC Server handler 2 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001278_377 is : 0.0
> 2013-05-16 20:32:11,241 INFO [IPC Server handler 6 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Status update from
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:11,241 INFO [IPC Server handler 6 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1368653326922_0080_r_001278_377 is : 0.0
> 2013-05-16 20:32:13,488 INFO [IPC Server handler 12 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Ping from
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:15,085 INFO [IPC Server handler 17 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit-pending state update
> from attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:15,085 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from RUNNING
> to COMMIT_PENDING
> 2013-05-16 20:32:15,085 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl:
> attempt_1368653326922_0080_r_001278_0 already given a go for committing the
> task output, so killing attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:15,085 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from
> COMMIT_PENDING to KILL_CONTAINER_CLEANUP
> 2013-05-16 20:32:15,085 INFO [ContainerLauncher #13]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_CLEANUP for container
> container_1368653326922_0080_01_001753 taskAttempt
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:15,086 INFO [ContainerLauncher #13]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: KILLING
> attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:15,086 INFO [IPC Server handler 19 on 40095]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Commit go/no-go request
> from attempt_1368653326922_0080_r_001278_377
> 2013-05-16 20:32:15,086 INFO [IPC Server handler 19 on 40095]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: Result of canCommit for
> attempt_1368653326922_0080_r_001278_377:false
> 2013-05-16 20:32:15,088 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from
> KILL_CONTAINER_CLEANUP to KILL_TASK_CLEANUP
> 2013-05-16 20:32:15,088 INFO [TaskCleaner #4]
> org.apache.hadoop.mapreduce.v2.app.taskclean.TaskCleanerImpl: Processing the
> event EventType: TASK_CLEAN
> 2013-05-16 20:32:15,098 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_377 TaskAttempt Transitioned from
> KILL_TASK_CLEANUP to KILLED
> 2013-05-16 20:32:15,098 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_378 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2013-05-16 20:32:15,878 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:1 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0
> AssignedReds:1 CompletedMaps:19 CompletedReds:1279 ContAlloc:1752 ContRel:71
> HostLocal:2 RackLocal:17
> 2013-05-16 20:32:15,879 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=1544192
> 2013-05-16 20:32:15,892 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: All maps
> assigned. Ramping up all remaining reduces:1
> 2013-05-16 20:32:15,892 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:1
> CompletedMaps:19 CompletedReds:1279 ContAlloc:1752 ContRel:71 HostLocal:2
> RackLocal:17
> 2013-05-16 20:32:16,895 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1368653326922_0080: ask=1 release= 0 newContainers=0
> finishedContainers=1 resourcelimit=memory: 1544192 knownNMs=20
> 2013-05-16 20:32:16,895 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1368653326922_0080_01_001753
> 2013-05-16 20:32:16,895 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:1 AssignedMaps:0 AssignedReds:0
> CompletedMaps:19 CompletedReds:1279 ContAlloc:1752 ContRel:71 HostLocal:2
> RackLocal:17
> 2013-05-16 20:32:16,895 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1368653326922_0080_r_001278_377: Container killed by the
> ApplicationMaster.
> 2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned to reduce
> 2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1368653326922_0080_01_001754 to
> attempt_1368653326922_0080_r_001278_378
> 2013-05-16 20:32:17,898 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After Scheduling:
> PendingReds:0 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:0 AssignedReds:1
> CompletedMaps:19 CompletedReds:1279 ContAlloc:1753 ContRel:71 HostLocal:2
> RackLocal:17
> 2013-05-16 20:32:17,898 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.yarn.util.RackResolver: Resolved
> sjc1-eng-perf-g1-grid18.carrieriq.com to /default-rack
> 2013-05-16 20:32:17,898 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_378 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2013-05-16 20:32:17,899 INFO [ContainerLauncher #12]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Processing
> the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1368653326922_0080_01_001754 taskAttempt
> attempt_1368653326922_0080_r_001278_378
> 2013-05-16 20:32:17,899 INFO [ContainerLauncher #12]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Launching
> attempt_1368653326922_0080_r_001278_378
> 2013-05-16 20:32:17,901 INFO [ContainerLauncher #12]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1368653326922_0080_r_001278_378
> : 8080
> 2013-05-16 20:32:17,901 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1368653326922_0080_r_001278_378] using containerId:
> [container_1368653326922_0080_01_001754 on NM:
> [sjc1-eng-perf-g1-grid18.carrieriq.com:37647]
> 2013-05-16 20:32:17,901 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1368653326922_0080_r_001278_378 TaskAttempt Transitioned from
> ASSIGNED to RUNNING
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
