[
https://issues.apache.org/jira/browse/MAPREDUCE-4488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445401#comment-13445401
]
Karthik Kambatla commented on MAPREDUCE-4488:
---------------------------------------------
It looks like the change, in particular, the implementation of
{{JobInProgress#setupComplete()}} seems to have introduced a race leading to
the following deadlock as noticed in our clusters:
{noformat}
Thread 42 (IPC Server handler 1 on 8021):
State: BLOCKED
Blocked count: 203661
Waited count: 563040
Blocked on org.apache.hadoop.mapred.JobInProgress@6ab8d396
Blocked by 243 (pool-7-thread-1)
Stack:
org.apache.hadoop.mapred.JobInProgress.runningMaps(JobInProgress.java:884)
org.apache.hadoop.mapred.JobSchedulable.getRunningTasks(JobSchedulable.java:110)
org.apache.hadoop.mapred.PoolSchedulable.getRunningTasks(PoolSchedulable.java:132)
org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:351)
org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2935)
sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.ipc.WritableRpcEngine$Server$WritableRpcInvoker.call(WritableRpcEngine.java:474)
org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898)
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693)
org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:396)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332)
org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687)
Thread 243 (pool-7-thread-1):
State: BLOCKED
Blocked count: 435
Waited count: 569
Blocked on org.apache.hadoop.mapred.JobTracker@3cfa54fe
Blocked by 42 (IPC Server handler 1 on 8021)
Stack:
org.apache.hadoop.mapred.JobTracker.getClusterStatus(JobTracker.java:3616)
org.apache.hadoop.mapred.JobInProgress.jobComplete(JobInProgress.java:2713)
org.apache.hadoop.mapred.JobInProgress.setupComplete(JobInProgress.java:837)
org.apache.hadoop.mapred.JobInProgress.initTasks(JobInProgress.java:790)
org.apache.hadoop.mapred.JobTracker.initJob(JobTracker.java:3750)
org.apache.hadoop.mapred.EagerTaskInitializationListener$InitJob.run(EagerTaskInitializationListener.java:79)
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
java.lang.Thread.run(Thread.java:662)
{noformat}
We should probably revert the commit, and fix it.
> Port MAPREDUCE-463 (The job setup and cleanup tasks should be optional) to
> branch-1
> -----------------------------------------------------------------------------------
>
> Key: MAPREDUCE-4488
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4488
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: mrv1, performance
> Affects Versions: 1.0.3
> Reporter: Tom White
> Assignee: Tom White
> Attachments: MAPREDUCE-4488.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
