[ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15888722#comment-15888722 ]
Arun Suresh commented on YARN-6251: ----------------------------------- Posting relevant section of the jstack dump {noformat} Found one Java-level deadlock: ============================= "pool-7-thread-88": waiting for ownable synchronizer 0x0000000083635b98, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync), which is held by "SchedulerEventDispatcher:Event Processor" "SchedulerEventDispatcher:Event Processor": waiting for ownable synchronizer 0x00000000f3f6b808, (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync), which is held by "pool-7-thread-88" Java stack information for the threads listed above: =================================================== "pool-7-thread-88": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x0000000083635b98> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.completedContainer(LeafQueue.java:1520) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.completedContainerInternal(CapacityScheduler.java:1600) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.AbstractYarnScheduler.completedContainer(AbstractYarnScheduler.java:602) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.pullNewlyUpdatedContainers(SchedulerApplicationAttempt.java:852) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt.pullNewlyIncreasedContainers(SchedulerApplicationAttempt.java:789) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.getAllocation(FiCaSchedulerApp.java:693) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocate(CapacityScheduler.java:925) at org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.allocate(SLSCapacityScheduler.java:191) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocateInternal(ApplicationMasterService.java:581) at org.apache.hadoop.yarn.server.resourcemanager.OpportunisticContainerAllocatorAMService.allocateInternal(OpportunisticContainerAllocatorAMService.java:254) at org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService.allocate(ApplicationMasterService.java:446) - locked <0x00000000f3fe59c0> (a org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService$AllocateResponseLock) at org.apache.hadoop.yarn.sls.appmaster.PromotingAMSimulator$1.run(PromotingAMSimulator.java:267) at org.apache.hadoop.yarn.sls.appmaster.PromotingAMSimulator$1.run(PromotingAMSimulator.java:264) at org.apache.hadoop.yarn.sls.appmaster.AMSimulator.middleStep(AMSimulator.java:179) at org.apache.hadoop.yarn.sls.scheduler.TaskRunner$Task.run(TaskRunner.java:96) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) "SchedulerEventDispatcher:Event Processor": at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000f3f6b808> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283) at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.getHeadroom(FiCaSchedulerApp.java:755) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.allocateResource(LeafQueue.java:1578) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.apply(LeafQueue.java:1206) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.fica.FiCaSchedulerApp.apply(FiCaSchedulerApp.java:560) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.tryCommit(CapacityScheduler.java:2378) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.submitResourceCommitRequest(CapacityScheduler.java:2263) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateOrReserveNewContainers(CapacityScheduler.java:1248) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainerOnSingleNode(CapacityScheduler.java:1237) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1326) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.allocateContainersToNode(CapacityScheduler.java:1124) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.nodeUpdate(CapacityScheduler.java:988) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1368) at org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:254) at org.apache.hadoop.yarn.sls.scheduler.SLSCapacityScheduler.handle(SLSCapacityScheduler.java:84) at org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66) at java.lang.Thread.run(Thread.java:745) Found 1 deadlock. {noformat} > Fix Scheduler locking issue introduced by YARN-6216 > --------------------------------------------------- > > Key: YARN-6251 > URL: https://issues.apache.org/jira/browse/YARN-6251 > Project: Hadoop YARN > Issue Type: Bug > Reporter: Arun Suresh > Assignee: Arun Suresh > Fix For: 3.0.0-alpha3 > > > Opening to track a locking issue that was uncovered when running a custom SLS > AMSimulator. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org