[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501420#comment-14501420 ] Hudson commented on YARN-3487: -- SUCCESS: Integrated in Hadoop-Mapreduce-trunk #2117 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2117/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501405#comment-14501405 ] Hudson commented on YARN-3487: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #168 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/168/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/CHANGES.txt > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501299#comment-14501299 ] Hudson commented on YARN-3487: -- FAILURE: Integrated in Hadoop-Yarn-trunk #901 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/901/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/CHANGES.txt > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501291#comment-14501291 ] Hudson commented on YARN-3487: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #158 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/158/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/CHANGES.txt > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501285#comment-14501285 ] Hudson commented on YARN-3487: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2099 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2099/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/CHANGES.txt > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14501242#comment-14501242 ] Hudson commented on YARN-3487: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #167 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/167/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/CHANGES.txt > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500954#comment-14500954 ] Hudson commented on YARN-3487: -- FAILURE: Integrated in Hadoop-trunk-Commit #7610 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/7610/]) YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) (wangda: rev f47a5763acd55cb0b3f16152c7f8df06ec0e09a9) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/CHANGES.txt > CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue > - > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Fix For: 2.7.1 > > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[
https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500891#comment-14500891
]
Hadoop QA commented on YARN-3487:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726214/YARN-3487.003.patch
against trunk revision d573f09.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7387//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7387//console
This message is automatically generated.
> CapacityScheduler scheduler lock obtained unnecessarily
> ---
>
> Key: YARN-3487
> URL: https://issues.apache.org/jira/browse/YARN-3487
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-3487.001.patch, YARN-3487.002.patch,
> YARN-3487.003.patch
>
>
> Recently saw a significant slowdown of applications on a large cluster, and
> we noticed there were a large number of blocked threads on the RM. Most of
> the blocked threads were waiting for the CapacityScheduler lock while calling
> getQueueInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500767#comment-14500767 ] Wangda Tan commented on YARN-3487: -- Re-triggerred Jenkins > CapacityScheduler scheduler lock obtained unnecessarily > --- > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[
https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500484#comment-14500484
]
Hadoop QA commented on YARN-3487:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12726214/YARN-3487.003.patch
against trunk revision c6b5203.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1207 javac
compiler warnings (more than the trunk's current 1181 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerQueueACLs
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesFairScheduler
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStoreZKClientConnections
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestLeveldbRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStorePerf
org.apache.hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerUtils
org.apache.hadoop.yarn.server.resourcemanager.security.TestClientToAMTokens
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRTestTests
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestaTesTests
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorizatTests
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesNodeLabels
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7380//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-YARN-Build/7380//artifact/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7380//console
This message is automatically generated.
> CapacityScheduler scheduler lock obtained unnecessarily
> ---
>
> Key: YARN-3487
> URL: https://issues.apache.org/jira/browse/YARN-3487
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacityscheduler
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-3487.001.patch, YARN-3487.002.patch,
> YARN-3487.003.patch
>
>
> Recently saw a significant slowdown of applications on a large cluster, and
> we noticed there were a large number of blocked threads on the RM. Most of
> the blocked threads were waiting for the CapacityScheduler lock while calling
> getQueueInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14500342#comment-14500342 ] Wangda Tan commented on YARN-3487: -- Thanks for feedback from [~sunilg], [~jlowe]. Make this as a sub JIRA of YARN-3091, and w/r lock for CS is tracked by YARN-3139. The latest patch LGTM, will commit when Jenkins get back. > CapacityScheduler scheduler lock obtained unnecessarily > --- > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-3487.001.patch, YARN-3487.002.patch, > YARN-3487.003.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499393#comment-14499393 ] Sunil G commented on YARN-3487: --- Hi [~leftnoteasy] I am sorry for providing lesser content earlier. After seeing your comment again, i could see that my comment also was going on same line. Runtime updates can add or change some CLIs for a Queue. So if synchronized keyword s removed, checkAccess is open and some checks may pass/fail as per the partial information available for CLI of Queue. So we may run into partial errors which is a race case condition. > CapacityScheduler scheduler lock obtained unnecessarily > --- > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-3487.001.patch, YARN-3487.002.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14499038#comment-14499038 ] Wangda Tan commented on YARN-3487: -- [~sunilg], Sorry I didn't get what you meant, could you elaborate it? > CapacityScheduler scheduler lock obtained unnecessarily > --- > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-3487.001.patch, YARN-3487.002.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[ https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14498324#comment-14498324 ] Sunil G commented on YARN-3487: --- Hi [~leftnoteasy] and [~jlowe] During reloading CS config, if a queue s removed, eventhough 'queues' is a concurrent one, operation for delete may be in progress. And here we may try for checkAccess and pass it with success. > CapacityScheduler scheduler lock obtained unnecessarily > --- > > Key: YARN-3487 > URL: https://issues.apache.org/jira/browse/YARN-3487 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.6.0 >Reporter: Jason Lowe >Assignee: Jason Lowe >Priority: Critical > Attachments: YARN-3487.001.patch, YARN-3487.002.patch > > > Recently saw a significant slowdown of applications on a large cluster, and > we noticed there were a large number of blocked threads on the RM. Most of > the blocked threads were waiting for the CapacityScheduler lock while calling > getQueueInfo. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[
https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495195#comment-14495195
]
Hadoop QA commented on YARN-3487:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725412/YARN-3487.002.patch
against trunk revision fddd552.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterLauncher
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationMasterService
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7338//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7338//console
This message is automatically generated.
> CapacityScheduler scheduler lock obtained unnecessarily
> ---
>
> Key: YARN-3487
> URL: https://issues.apache.org/jira/browse/YARN-3487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-3487.001.patch, YARN-3487.002.patch
>
>
> Recently saw a significant slowdown of applications on a large cluster, and
> we noticed there were a large number of blocked threads on the RM. Most of
> the blocked threads were waiting for the CapacityScheduler lock while calling
> getQueueInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[
https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495137#comment-14495137
]
Wangda Tan commented on YARN-3487:
--
[~jlowe], removing lock of checkAccess may cause consumer get partial queue's
ACL when reinitialize happens. See {{YarnAuthorizationProvider}},
{{CapacityScheduler#setQueueAcls}} will set ACL one by one, so checkAccess's
invoker can pull incomplete result.
To avoid this, one possible solute might be: add a reintialize_lock to CS, it
will be locked when doing reinitialize, and checkAccess will try to acquire it.
> CapacityScheduler scheduler lock obtained unnecessarily
> ---
>
> Key: YARN-3487
> URL: https://issues.apache.org/jira/browse/YARN-3487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-3487.001.patch, YARN-3487.002.patch
>
>
> Recently saw a significant slowdown of applications on a large cluster, and
> we noticed there were a large number of blocked threads on the RM. Most of
> the blocked threads were waiting for the CapacityScheduler lock while calling
> getQueueInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3487) CapacityScheduler scheduler lock obtained unnecessarily
[
https://issues.apache.org/jira/browse/YARN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495095#comment-14495095
]
Hadoop QA commented on YARN-3487:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12725395/YARN-3487.001.patch
against trunk revision 05007b4.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/7334//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/7334//console
This message is automatically generated.
> CapacityScheduler scheduler lock obtained unnecessarily
> ---
>
> Key: YARN-3487
> URL: https://issues.apache.org/jira/browse/YARN-3487
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Affects Versions: 2.6.0
>Reporter: Jason Lowe
>Assignee: Jason Lowe
>Priority: Critical
> Attachments: YARN-3487.001.patch, YARN-3487.002.patch
>
>
> Recently saw a significant slowdown of applications on a large cluster, and
> we noticed there were a large number of blocked threads on the RM. Most of
> the blocked threads were waiting for the CapacityScheduler lock while calling
> getQueueInfo.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
