[jira] [Comment Edited] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263961#comment-17263961 ] zhuqi edited comment on YARN-10506 at 1/13/21, 7:50 AM: [~wangda] [~gandras] Thanks [~wangda] for review. I update a patch for: According the comment by [~wangda] above: 1. CapacityScheduler 1.1 Fix the not return lq in Capacity. 1.2 Add 008 patch remaining include to fix testautocreateleaf: As above in 008 patch, Fix autoCreateLeafQueue to use path , the logic change , the 009 not include this: {code:java} @VisibleForTesting protected LeafQueue autoCreateLeafQueue( ApplicationPlacementContext placementContext) throws IOException, YarnException { String leafQueueName = placementContext.getQueue(); String parentQueueName = placementContext.getParentQueue(); if (!StringUtils.isEmpty(parentQueueName)) { CSQueue parentQueue = getQueue(parentQueueName); if (parentQueue == null) { throw new SchedulerDynamicEditException( "Could not auto-create leaf queue for " + leafQueueName + ". Queue mapping specifies an invalid parent queue " + "which does not exist " + parentQueueName); } if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) { // Case 1: Handle ManagedParentQueue AutoCreatedLeafQueue autoCreatedLeafQueue = null; ManagedParentQueue autoCreateEnabledParentQueue = (ManagedParentQueue) parentQueue; autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName, autoCreateEnabledParentQueue); addQueue(autoCreatedLeafQueue); return autoCreatedLeafQueue; } ... } {code} 2. CapacitySchedulerAutoQueueHandler 2.1 Back create flag in ApplicationPlacementContext, and use it in autoCreateLeafQueue placement check : {code:java} private LeafQueue autoCreateLeafQueue( ApplicationPlacementContext placementContext) throws IOException, YarnException { ... if (!StringUtils.isEmpty(parentQueueName)) { ... if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) { ... } else { ApplicationPlacementContext apc = placementContext.clone(); // Set default true now // TODO get by queue placement policy apc.setCreateParentQueue(true); apc.setCreateLeafQueue(true); if (placementContext.isCreateLeafQueue() || placementContext.isCreateParentQueue()) { } else { ... } } } {code} 2.2 -2.3 Remove autoCreateParentHierarchy, the logic change to autoCreateQueuePath. And the autoCreateQueuePath to autoCreateQueue, and make it atomic, and fix related test. {code:java} public LeafQueue autoCreateQueue(ApplicationPlacementContext queue) throws YarnException { ... // The existingParentQueue is already ParentQueue LeafQueue leafQueue = existingParentQueue.addDynamicLeafQueue( queue.getFullQueuePath()); queueManager.addQueue(leafQueue.getQueuePath(), leafQueue); return leafQueue; } {code} 3. CapacitySchedulerConfiguration change: Remaining todo. 4. ParentQueue#addDynamicChildQueue, change the logic handle static queue with empty childs: {code:java} private QueueCapacityType getCapacityConfigurationTypeForQueues( Collection queues) throws IOException { ... else { if (isDynamicQueue()) { // If it is an empty dynamic queue, consider weight mode instead return QueueCapacityType.WEIGHT; } else { // Double check queues should be empty // When static queue with queues empty, return parent ConfigurationType if (queues.isEmpty()) { return getCapacityConfigurationTypeForQueues(ImmutableList.of(this)); } else { throw new IOException("Unknown case when child queues not empty! " + "But no child config type!"); } } } {code} 5. Change the submit test to trigger the placement: {code:java} @Test public void testAutoQueueCreationOnAppSubmission() throws Exception { startScheduler(); createBasicQueueStructureAndValidate(); submitApp(cs, USER0, USER0, "root.e-auto"); AbstractCSQueue e = (AbstractCSQueue) cs.getQueue("root.e-auto"); Assert.assertNotNull(e); Assert.assertTrue(e.isDynamicQueue()); AbstractCSQueue user0 = (AbstractCSQueue) cs.getQueue("root.e-auto." + USER0); Assert.assertNotNull(user0); Assert.assertTrue(user0.isDynamicQueue()); } {code} 6. Fix clone finding bugs. Remaining to do minor things: # AutoCreateLeafQueue should be moved to autoQueueHandler. # CSQueueUtils#extractQueuePath should be moved to CSAutoQueueHandler. (It won't be used by other classes). And rename extractQueuePath to extractApplicationPlacementContext (we don't just extract path). # Add test case that static queues with empty childs. Major t
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10506: - Attachment: YARN-10506-010.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, > YARN-10506.001.patch, YARN-10506.002.patch, YARN-10506.003.patch, > YARN-10506.004.patch, YARN-10506.005.patch, YARN-10506.006-combined.patch, > YARN-10506.006.patch, YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263961#comment-17263961 ] zhuqi commented on YARN-10506: -- [~wangda] [~gandras] Thanks [~wangda] for review. I update a patch for: According the comment by [~wangda] above: 1. CapacityScheduler 1.1 Fix the not return lq in Capacity. 1.2 Add 008 patch remaining include to fix testautocreateleaf: As above in 008 patch, Fix autoCreateLeafQueue to use path , the logic change , the 009 not include this: {code:java} @VisibleForTesting protected LeafQueue autoCreateLeafQueue( ApplicationPlacementContext placementContext) throws IOException, YarnException { String leafQueueName = placementContext.getQueue(); String parentQueueName = placementContext.getParentQueue(); if (!StringUtils.isEmpty(parentQueueName)) { CSQueue parentQueue = getQueue(parentQueueName); if (parentQueue == null) { throw new SchedulerDynamicEditException( "Could not auto-create leaf queue for " + leafQueueName + ". Queue mapping specifies an invalid parent queue " + "which does not exist " + parentQueueName); } if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) { // Case 1: Handle ManagedParentQueue AutoCreatedLeafQueue autoCreatedLeafQueue = null; ManagedParentQueue autoCreateEnabledParentQueue = (ManagedParentQueue) parentQueue; autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName, autoCreateEnabledParentQueue); addQueue(autoCreatedLeafQueue); return autoCreatedLeafQueue; } ... } {code} 2. CapacitySchedulerAutoQueueHandler 2.1 Back create flag in ApplicationPlacementContext, and use it in autoCreateLeafQueue placement check : {code:java} private LeafQueue autoCreateLeafQueue( ApplicationPlacementContext placementContext) throws IOException, YarnException { ... if (!StringUtils.isEmpty(parentQueueName)) { ... if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) { ... } else { ApplicationPlacementContext apc = placementContext.clone(); // Set default true now // TODO get by queue placement policy apc.setCreateParentQueue(true); apc.setCreateLeafQueue(true); if (placementContext.isCreateLeafQueue() || placementContext.isCreateParentQueue()) { } else { ... } } } {code} 2.2 -2.3 Remove autoCreateParentHierarchy, the logic change to autoCreateQueuePath. And the autoCreateQueuePath to autoCreateQueue, and make it atomic, and fix related test. {code:java} public LeafQueue autoCreateQueue(ApplicationPlacementContext queue) throws YarnException { ... // The existingParentQueue is already ParentQueue LeafQueue leafQueue = existingParentQueue.addDynamicLeafQueue( queue.getFullQueuePath()); queueManager.addQueue(leafQueue.getQueuePath(), leafQueue); return leafQueue; } {code} 3. CapacitySchedulerConfiguration change: Remaining todo. 4. ParentQueue#addDynamicChildQueue, change the logic handle static queue with empty childs: {code:java} private QueueCapacityType getCapacityConfigurationTypeForQueues( Collection queues) throws IOException { ... else { if (isDynamicQueue()) { // If it is an empty dynamic queue, consider weight mode instead return QueueCapacityType.WEIGHT; } else { // Double check queues should be empty // When static queue with queues empty, return parent ConfigurationType if (queues.isEmpty()) { return getCapacityConfigurationTypeForQueues(ImmutableList.of(this)); } else { throw new IOException("Unknown case when child queues not empty! " + "But no child config type!"); } } } {code} 5. Change the submit test to trigger the placement: {code:java} @Test public void testAutoQueueCreationOnAppSubmission() throws Exception { startScheduler(); createBasicQueueStructureAndValidate(); submitApp(cs, USER0, USER0, "root.e-auto"); AbstractCSQueue e = (AbstractCSQueue) cs.getQueue("root.e-auto"); Assert.assertNotNull(e); Assert.assertTrue(e.isDynamicQueue()); AbstractCSQueue user0 = (AbstractCSQueue) cs.getQueue("root.e-auto." + USER0); Assert.assertNotNull(user0); Assert.assertTrue(user0.isDynamicQueue()); } {code} 6. Fix clone finding bugs. Remaining to do minor things: # AutoCreateLeafQueue should be moved to autoQueueHandler. # CSQueueUtils#extractQueuePath should be moved to CSAutoQueueHandler. (It won't be used by other classes). And rename extractQueuePath to extractApplicationPlacementContext (we don't just extract path). # Add test case that static queues with empty childs. Major things: CapacitySchedulerConfiguration change. T
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[ https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263848#comment-17263848 ] zhuqi commented on YARN-10564: -- [~wangda] Ok, i will look into YARN-10506 first.:) > Support Auto Queue Creation template configurations > --- > > Key: YARN-10564 > URL: https://issues.apache.org/jira/browse/YARN-10564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: zhuqi >Priority: Major > Attachments: YARN-10564.poc.001.patch > > > Similar to how the template configuration works for ManagedParents, we need > to support templates for the new auto queue creation logic. Proposition is to > allow wildcards in template configs such as: > {noformat} > yarn.scheduler.capacity.root.*.*.weight 10{noformat} > which would mean, that set weight to 10 of every leaf of every parent under > root. > We should possibly take an approach, that could support arbitrary depth of > template configuration, because we might need to lift the limitation of auto > queue nesting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4589) Diagnostics for localization timeouts is lacking
[ https://issues.apache.org/jira/browse/YARN-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263761#comment-17263761 ] Hadoop QA commented on YARN-4589: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 37s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 13s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 31s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 20s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 24s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 22s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 8s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 8s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 47s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {co
[jira] [Commented] (YARN-10562) Follow up changes for YARN-9833
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263748#comment-17263748 ] Hadoop QA commented on YARN-10562: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m 7s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 25s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 19s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 37s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 36s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 11s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 11s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 5s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 47s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
[jira] [Commented] (YARN-4589) Diagnostics for localization timeouts is lacking
[ https://issues.apache.org/jira/browse/YARN-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263731#comment-17263731 ] Eric Payne commented on YARN-4589: -- I verified that the unit tests are either not failing or failing in the same way with and without the patch. The ASF license error was probably due to the stray file in patch 004. > Diagnostics for localization timeouts is lacking > > > Key: YARN-4589 > URL: https://issues.apache.org/jira/browse/YARN-4589 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Chang Li >Assignee: Chang Li >Priority: Major > Attachments: YARN-4589.004.patch, YARN-4589.005.patch, > YARN-4589.2.patch, YARN-4589.3.patch, YARN-4589.patch > > > When a container takes too long to localize it manifests as a timeout, and > there's no indication that localization was the issue. We need diagnostics > for timeouts to indicate the container was still localizing when the timeout > occurred. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10562) Follow up changes for YARN-9833
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263710#comment-17263710 ] Peter Bacsko commented on YARN-10562: - LGTM +1 from me. > Follow up changes for YARN-9833 > --- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch, YARN-10562.004.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10562) Follow up changes for YARN-9833
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263705#comment-17263705 ] Eric Badger commented on YARN-10562: +1 on the patch. As mentioned above, there is still the race in the code based on the fact that the caller doesn't have to get all Dirs at the same time. But the only issue that this will cause is the dirs being out of date for that iteration. The next time they get a copy, it will be updated. And the list will always be well-constructed. It just has the possibility of being out of sync when compared with the other lists. Will wait for the precommit to come back and commit if there are no errors and no objections > Follow up changes for YARN-9833 > --- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch, YARN-10562.004.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10525) Add weight mode conversion to fs2cs
[ https://issues.apache.org/jira/browse/YARN-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263704#comment-17263704 ] Hadoop QA commented on YARN-10525: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 20s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 48s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 36s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 19s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 30s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/472/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 6 unchanged - 2 fixed = 7 total (was 8) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 24s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color}
[jira] [Updated] (YARN-4589) Diagnostics for localization timeouts is lacking
[ https://issues.apache.org/jira/browse/YARN-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan updated YARN-4589: -- Attachment: YARN-4589.005.patch > Diagnostics for localization timeouts is lacking > > > Key: YARN-4589 > URL: https://issues.apache.org/jira/browse/YARN-4589 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Chang Li >Assignee: Chang Li >Priority: Major > Attachments: YARN-4589.004.patch, YARN-4589.005.patch, > YARN-4589.2.patch, YARN-4589.3.patch, YARN-4589.patch > > > When a container takes too long to localize it manifests as a timeout, and > there's no indication that localization was the issue. We need diagnostics > for timeouts to indicate the container was still localizing when the timeout > occurred. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4589) Diagnostics for localization timeouts is lacking
[ https://issues.apache.org/jira/browse/YARN-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263700#comment-17263700 ] Jim Brennan commented on YARN-4589: --- patch 005 removes the extra file. > Diagnostics for localization timeouts is lacking > > > Key: YARN-4589 > URL: https://issues.apache.org/jira/browse/YARN-4589 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Chang Li >Assignee: Chang Li >Priority: Major > Attachments: YARN-4589.004.patch, YARN-4589.005.patch, > YARN-4589.2.patch, YARN-4589.3.patch, YARN-4589.patch > > > When a container takes too long to localize it manifests as a timeout, and > there's no indication that localization was the issue. We need diagnostics > for timeouts to indicate the container was still localizing when the timeout > occurred. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10562) Follow up changes for YARN-9833
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan updated YARN-10562: --- Summary: Follow up changes for YARN-9833 (was: Alternate fix for DirectoryCollection.checkDirs() race) > Follow up changes for YARN-9833 > --- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch, YARN-10562.004.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10562) Alternate fix for DirectoryCollection.checkDirs() race
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263696#comment-17263696 ] Jim Brennan commented on YARN-10562: Patch 004 replaces the {{CopyOnWriteArrayLists}} with {{ArrayLists}}. It also fixes {{getErroredDirs()}} to use {{ImmutableList.copyOf()}} instead of {{Collections.unmodifiableList()}}. One additional change I made was to change {{getFailedDirs()}} to use {{Collections.unmodifiableList()}} instead of {{ImmutableList.copyOf()}}. There is no need to make another copy in this case, because {{DirectoryCollection.concat()}} was already constructing a new list. > Alternate fix for DirectoryCollection.checkDirs() race > -- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch, YARN-10562.004.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10562) Alternate fix for DirectoryCollection.checkDirs() race
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jim Brennan updated YARN-10562: --- Attachment: YARN-10562.004.patch > Alternate fix for DirectoryCollection.checkDirs() race > -- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch, YARN-10562.004.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4589) Diagnostics for localization timeouts is lacking
[ https://issues.apache.org/jira/browse/YARN-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263691#comment-17263691 ] Jim Brennan commented on YARN-4589: --- I don't think I need to add a unit test for this, as it is only adding a log message. I believe the other unit tests are unrelated, but I will double-check. It looks like I included a stray file - I will remove it and put up another patch. > Diagnostics for localization timeouts is lacking > > > Key: YARN-4589 > URL: https://issues.apache.org/jira/browse/YARN-4589 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Chang Li >Assignee: Chang Li >Priority: Major > Attachments: YARN-4589.004.patch, YARN-4589.2.patch, > YARN-4589.3.patch, YARN-4589.patch > > > When a container takes too long to localize it manifests as a timeout, and > there's no indication that localization was the issue. We need diagnostics > for timeouts to indicate the container was still localizing when the timeout > occurred. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263672#comment-17263672 ] Wangda Tan commented on YARN-10506: --- Thanks for all updates, here's my comments: 1) CapacityScheduler: Minor: autoCreateLeafQueue should be moved to autoQueueHandler. We can do this in a follow up patch, because now we have two places to handle the auto queue creation, and ideally "autoQueueHandler" should be responsible for that. The same follow up patch should also clean up addQueue() method of ResourceScheduler. It is only used by CapacitySchedulerPlanFollow, we don't need to add it to the abstract class. {code} LeafQueue lq = autoQueueHandler.autoCreateQueuePath(placementContext); {code} Return value is not used. 2) CapacitySchedulerAutoQueueHandler: Major: 2.1) I think the existing logic is to create auto queue based on ApplicationPlacementContext, but when we do queue mapping, we need to indicate if auto creation is allowed or not. In the mapping rule, we have "create" flag, I think CSAutoQueueHandler should consider the "create" flag. Previously, in my patch, I added createLeafQueue and createParentQueue flag, the latest patch removed the logic. Can you share more thoughts on this? Any please let me know if I missed anything. Even though this logic can be done in a separate patch, but I still suggest to handle it within this one for completeness. (A side note, I noticed there's a method: ParentQueue#isEligibleForAutoQueueCreation, it only handles "if a parent queue can allow auto creation underneath or not", but cannot handle "if a creation itself is allowed by placement policy or not"). 2.2) autoCreateQueuePath doesn't look like atomic, it could create parent first, but later found LeafQueue cannot be created: {code} if (parent instanceof ParentQueue) { ... } else { throw new SchedulerDynamicEditException( "Could not auto-create leaf queue for " + queue.getQueue() + ". Queue mapping specifies an invalid parent queue " + "which does not exist" + queue.getParentQueue()); } {code} Can we make it atomic? 2.3) I think autoCreateParentHierarchy itself should be able to handle LeafQueue creation, because itself can handle multiple leaves, we don't have to maintain a separate LeafQueue creation logic: {code} ParentQueue parentQueue = (ParentQueue) parent; LeafQueue leafQueue = parentQueue.addDynamicLeafQueue( queue.getFullQueuePath()); queueManager.addQueue(leafQueue.getQueuePath(), leafQueue); return leafQueue; {code} Minor: - Rename autoCreateQueuePath to autoCreateQueue (we will never create a "queue path") - CSQueueUtils#extractQueuePath should be moved to CSAutoQueueHandler. (It won't be used by other classes). And rename extractQueuePath to extractApplicationPlacementContext (we don't just extract path) 3) CapacitySchedulerConfiguration change: Major: - Now we added queue-path.auto-queue-creation.enabled flag, which is in parallel of auto-create-child-queue.enabled flag. First of all, it is very confusing because there're two parameters looks similar. Second, We still need to check the weight mode is configured or not. I'm actually thinking to get rid of the flag, and completely relying on ParentQueue#addDynamicChildQueue to do the weights check: If a parent queue has children use weight, we can proceed with queue creation. We can improve the flag later, thoughts? 4) ParentQueue#addDynamicChildQueue: Major: - The logic ParentQueue#getCapacityConfigurationTypeForQueues returns PERCENT when there's no children under the parent. For that case, I think we should specially handle it inside getCapacityConfigurationTypeForQueues: {code} Which should return WEIGHT when Collection queues isEmpty {code} And we should add an unit test to add queue under a static parent queue which doesn't have children, because it will be a common case when user use the feature. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) ---
[jira] [Commented] (YARN-10562) Alternate fix for DirectoryCollection.checkDirs() race
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263601#comment-17263601 ] Eric Badger commented on YARN-10562: Yea the original problem (before YARN-9833) was that we were getting a view of the list instead of a copy. And those views could iterate the list at any time. The issue there was that checkDirs was going out and clearing those lists in a separate thread. So when the client iterated through the lists, it would periodically see an empty list if it iterated at just the right time. After YARN-9833 there is still a race, but it is a smaller and less nefarious one. The issue there is that we have 3 lists (localDirs, fullDirs, errorDirs). Those can really be thought of as a single list with different attributes for each dir. Because the sum of all of those lists should give you all disks on the node. YARN-9833 added code to return a copy of the list instead of a view. So we'll never have a list that is incomplete. But the race becomes the fact that you could potentially call getGoodDirs(), then have checkDirs run, then call getErrorDirs. If a dir transitioned from good -> error just after getGoodDirs was called, it would show up in both lists. But like you said, [~pbacsko], I think it makes sense to remove complexity of the code if it requires this type of discussion to understand exactly why the code works (or doesn't work). It makes the code harder to maintain and even harder to modify. > Alternate fix for DirectoryCollection.checkDirs() race > -- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10525) Add weight mode conversion to fs2cs
[ https://issues.apache.org/jira/browse/YARN-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263583#comment-17263583 ] Peter Bacsko commented on YARN-10525: - I retriggered it, let's wait for the results. > Add weight mode conversion to fs2cs > --- > > Key: YARN-10525 > URL: https://issues.apache.org/jira/browse/YARN-10525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10525-001.patch, YARN-10525-002.patch, > YARN-10525-003.patch, YARN-10525-004.patch, YARN-10525-005.patch > > > Weight mode will be added to Capacity Scheduler. > Currently, we convert FS weights to percentages, however, it will be more > useful to keep those values and use them in CS as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[ https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263575#comment-17263575 ] Wangda Tan commented on YARN-10564: --- Thanks [~zhuqi], let's get YARN-10506 done shortly (in a day or two), and we can then move on to other patches including this one, auto-delete queue, etc. > Support Auto Queue Creation template configurations > --- > > Key: YARN-10564 > URL: https://issues.apache.org/jira/browse/YARN-10564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: zhuqi >Priority: Major > Attachments: YARN-10564.poc.001.patch > > > Similar to how the template configuration works for ManagedParents, we need > to support templates for the new auto queue creation logic. Proposition is to > allow wildcards in template configs such as: > {noformat} > yarn.scheduler.capacity.root.*.*.weight 10{noformat} > which would mean, that set weight to 10 of every leaf of every parent under > root. > We should possibly take an approach, that could support arbitrary depth of > template configuration, because we might need to lift the limitation of auto > queue nesting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10564) Support Auto Queue Creation template configurations
[ https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned YARN-10564: - Assignee: zhuqi (was: Andras Gyori) > Support Auto Queue Creation template configurations > --- > > Key: YARN-10564 > URL: https://issues.apache.org/jira/browse/YARN-10564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: zhuqi >Priority: Major > Attachments: YARN-10564.poc.001.patch > > > Similar to how the template configuration works for ManagedParents, we need > to support templates for the new auto queue creation logic. Proposition is to > allow wildcards in template configs such as: > {noformat} > yarn.scheduler.capacity.root.*.*.weight 10{noformat} > which would mean, that set weight to 10 of every leaf of every parent under > root. > We should possibly take an approach, that could support arbitrary depth of > template configuration, because we might need to lift the limitation of auto > queue nesting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10525) Add weight mode conversion to fs2cs
[ https://issues.apache.org/jira/browse/YARN-10525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263566#comment-17263566 ] Benjamin Teke commented on YARN-10525: -- Thanks [~pbacsko] for working on this. The patch LGTM (non-binding), and the tests are passing locally. [~snemeth] can you please retrigger the CI? > Add weight mode conversion to fs2cs > --- > > Key: YARN-10525 > URL: https://issues.apache.org/jira/browse/YARN-10525 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: zhuqi >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10525-001.patch, YARN-10525-002.patch, > YARN-10525-003.patch, YARN-10525-004.patch, YARN-10525-005.patch > > > Weight mode will be added to Capacity Scheduler. > Currently, we convert FS weights to percentages, however, it will be more > useful to keep those values and use them in CS as well. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263550#comment-17263550 ] Hadoop QA commented on YARN-10506: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 51s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 22s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 15s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 48s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/471/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 34 new + 737 unchanged - 1 fixed = 771 total (was 738) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 39s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {c
[jira] [Commented] (YARN-10562) Alternate fix for DirectoryCollection.checkDirs() race
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263493#comment-17263493 ] Peter Bacsko commented on YARN-10562: - I was looking at the patch. I *think* I probably understand [~ebadger]'s reasoning about the code being racy. So, we have a view, but even if we update the lists inside a lock, the client which checks the view might miss using a lock, plus elements are added one-by-one. A possible solution could be collecting the new elements in a new local list then call {{CopyOnWriteArrayList.addAll()}} but that also involves a full copy by using {{System.arrayCopy()}} and seems like we go back to square one (not only that, COWAL also uses locks interally). Is my understanding correct? Anyway, usually if it's hard to reason about the correctness of a code in a multi-threaded environment (which, unfortunately, happens more often than not), then let's play safe, copy stuff in a critical section and forget about mutability. I doubt that it affects the performance noticeably, especially since a container launch means starting a new process, which is a far more heavyweight operation from the perspective of the OS. > Alternate fix for DirectoryCollection.checkDirs() race > -- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10562) Alternate fix for DirectoryCollection.checkDirs() race
[ https://issues.apache.org/jira/browse/YARN-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263462#comment-17263462 ] Jim Brennan commented on YARN-10562: Thanks for the discussion and comment [~ebadger]! I agree that probably the best approach is to remove the use of CopyOnWriteArrayList and stick with simple ArrayLists. We can preserve the changes made in [YARN-9833] to return copies of the lists and fix that issue with GetErrorDirs. I don't think the cost of the copies, even for every launch, is really much of a concern in the grand scope of things.My inclination is to make the minimum changes for this - that is, not rewrite checkDirs() as I've done here - it is not nearly as inefficient with ArrayLists as it was with CopyOnWriteArrayLists. I will put up another patch with these changes. We'll probably want to change the Summary as well to indicate this is just a follow-on to [YARN-9833], not an alternate solution. If you'd prefer I file a new Jira instead, let me know. > Alternate fix for DirectoryCollection.checkDirs() race > -- > > Key: YARN-10562 > URL: https://issues.apache.org/jira/browse/YARN-10562 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Jim Brennan >Assignee: Jim Brennan >Priority: Major > Attachments: YARN-10562.001.patch, YARN-10562.002.patch, > YARN-10562.003.patch > > > In YARN-9833, a race condition in DirectoryCollection. {{getGoodDirs()}} and > related methods were returning an unmodifiable view of the lists. These > accesses were protected by read/write locks, but because the lists are > CopyOnWriteArrayLists, subsequent changes to the list, even when done under > the writelock, were exposed when a caller started iterating the list view. > CopyOnWriteArrayLists cache the current underlying list in the iterator, so > it is safe to iterate them even while they are being changed - at least the > view will be consistent. > The problem was that checkDirs() was clearing the lists and rebuilding them > from scratch every time, so if a caller called getGoodDirs() just before > checkDirs cleared it, and then started iterating right after the clear, they > could get an empty list. > The fix in YARN-9833 was to change {{getGoodDirs()}} and related methods to > return a copy of the list, which definitely fixes the race condition. The > disadvantage is that now we create a new copy of these lists every time we > launch a container. The advantage using CopyOnWriteArrayList was that the > lists should rarely ever change, and we can avoid all the copying. > Unfortunately, the way checkDirs() was written, it guaranteed that it would > modify those lists multiple times every time. > So this Jira proposes an alternate solution for YARN-9833, which mainly just > rewrites checkDirs() to minimize the changes to the underlying lists. There > are still some small windows where a disk will have been added to one list, > but not yet removed from another if you hit it just right, but I think these > should be pretty rare and relatively harmless, and in the vast majority of > cases I suspect only one disk will be moving from one list to another at any > time. The question is whether this type of inconsistency (which was always > there before -YARN-9833- is worth reducing all the copying. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10535) Make changes in queue placement policy to use auto-queue-placement API in CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan reassigned YARN-10535: - Assignee: Gergely Pollak > Make changes in queue placement policy to use auto-queue-placement API in > CapacityScheduler > --- > > Key: YARN-10535 > URL: https://issues.apache.org/jira/browse/YARN-10535 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Wangda Tan >Assignee: Gergely Pollak >Priority: Major > > Once YARN-10506 is done, we need to call the API from the queue placement > policy to create queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263457#comment-17263457 ] Wangda Tan commented on YARN-10506: --- [~zhuqi], [~gandras] can you check the findbugs warning? > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[ https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263426#comment-17263426 ] zhuqi edited comment on YARN-10564 at 1/12/21, 3:32 PM: [~leftnoteasy] [~gandras] Add an auto create with template poc patch, combined with YARN-10506-008.patch for test. The poc just handle template related conf, next we should reuse or extend AutoCreatedQueueManagementPolicy and realize the related logic. The test is in : testAutoCreateQueueWithTemplateWeight function. Thanks. was (Author: zhuqi): [~leftnoteasy] [~gandras] Add an auto create with template poc patch, combined with YARN-10506-008.patch for test. The poc just handle template related conf, next we should reuse or extend AutoCreatedQueueManagementPolicy and realize the related logic. > Support Auto Queue Creation template configurations > --- > > Key: YARN-10564 > URL: https://issues.apache.org/jira/browse/YARN-10564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10564.poc.001.patch > > > Similar to how the template configuration works for ManagedParents, we need > to support templates for the new auto queue creation logic. Proposition is to > allow wildcards in template configs such as: > {noformat} > yarn.scheduler.capacity.root.*.*.weight 10{noformat} > which would mean, that set weight to 10 of every leaf of every parent under > root. > We should possibly take an approach, that could support arbitrary depth of > template configuration, because we might need to lift the limitation of auto > queue nesting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations
[ https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263426#comment-17263426 ] zhuqi commented on YARN-10564: -- [~leftnoteasy] [~gandras] Add an auto create with template poc patch, combined with YARN-10506-008.patch for test. The poc just handle template related conf, next we should reuse or extend AutoCreatedQueueManagementPolicy and realize the related logic. > Support Auto Queue Creation template configurations > --- > > Key: YARN-10564 > URL: https://issues.apache.org/jira/browse/YARN-10564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10564.poc.001.patch > > > Similar to how the template configuration works for ManagedParents, we need > to support templates for the new auto queue creation logic. Proposition is to > allow wildcards in template configs such as: > {noformat} > yarn.scheduler.capacity.root.*.*.weight 10{noformat} > which would mean, that set weight to 10 of every leaf of every parent under > root. > We should possibly take an approach, that could support arbitrary depth of > template configuration, because we might need to lift the limitation of auto > queue nesting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10564) Support Auto Queue Creation template configurations
[ https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhuqi updated YARN-10564: - Attachment: YARN-10564.poc.001.patch > Support Auto Queue Creation template configurations > --- > > Key: YARN-10564 > URL: https://issues.apache.org/jira/browse/YARN-10564 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10564.poc.001.patch > > > Similar to how the template configuration works for ManagedParents, we need > to support templates for the new auto queue creation logic. Proposition is to > allow wildcards in template configs such as: > {noformat} > yarn.scheduler.capacity.root.*.*.weight 10{noformat} > which would mean, that set weight to 10 of every leaf of every parent under > root. > We should possibly take an approach, that could support arbitrary depth of > template configuration, because we might need to lift the limitation of auto > queue nesting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263399#comment-17263399 ] zhuqi commented on YARN-10506: -- [~gandras] Thanks for including the fix. LGTM +1 Waiting for the CI. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263393#comment-17263393 ] Andras Gyori commented on YARN-10506: - I have uploaded a new patch, which is based on trunk with the freshly committed YARN-10504. Added an additional test case for queue creation with application submission. Thank you [~zhuqi] for the fixes, I have included them in the patch. At this point, nothing comes to my mind as to what else is needed, therefore this could be reviewed now. > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori updated YARN-10506: Attachment: YARN-10506.009.patch > Update queue creation logic to use weight mode and allow the flexible > static/dynamic creation > - > > Key: YARN-10506 > URL: https://issues.apache.org/jira/browse/YARN-10506 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10506-006-10504-010.patch, > YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch, > YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, > YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, > YARN-10506.007.patch, YARN-10506.009.patch > > > The queue creation logic should be updated to use weight mode and support the > flexible creation. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10563) Fix dependency exclusion problem in poms
[ https://issues.apache.org/jira/browse/YARN-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10563: -- Fix Version/s: 3.4.0 > Fix dependency exclusion problem in poms > > > Key: YARN-10563 > URL: https://issues.apache.org/jira/browse/YARN-10563 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Fix For: 3.4.0 > > Attachments: YARN-10563-001.patch > > > Currently, {{hadoop-project/pom.xml}} contains a bunch of exclusions for > {{jsonschema2pojo-core}}. It turned out that when a project imports either > hadoop-minicluster or hadoop-yarn-server-resourcemanager, excluded artifacts > can actually show up as active dependencies. This is probably has to do with > Maven as it does not handle exclusions properly when listed inside a > {{dependencyManagement}} section. I confirmed it with some local builds (eg. > Oozie uses hadoop-minicluster and the dependency tree showed a bunch of items > that were not supposed to be there). > So all excludeable artifacts should be moved to > {{hadoop-yarn-server-resourcemanager/pom.xml}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10563) Fix dependency exclusion problem in poms
[ https://issues.apache.org/jira/browse/YARN-10563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263366#comment-17263366 ] Szilard Nemeth commented on YARN-10563: --- Thanks [~pbacsko], Good catch! Committed to trunk. Resolving jira. > Fix dependency exclusion problem in poms > > > Key: YARN-10563 > URL: https://issues.apache.org/jira/browse/YARN-10563 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Critical > Attachments: YARN-10563-001.patch > > > Currently, {{hadoop-project/pom.xml}} contains a bunch of exclusions for > {{jsonschema2pojo-core}}. It turned out that when a project imports either > hadoop-minicluster or hadoop-yarn-server-resourcemanager, excluded artifacts > can actually show up as active dependencies. This is probably has to do with > Maven as it does not handle exclusions properly when listed inside a > {{dependencyManagement}} section. I confirmed it with some local builds (eg. > Oozie uses hadoop-minicluster and the dependency tree showed a bunch of items > that were not supposed to be there). > So all excludeable artifacts should be moved to > {{hadoop-yarn-server-resourcemanager/pom.xml}}. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10570) Remove "experimental" warning message from fs2cs
[ https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10570: -- Fix Version/s: 3.4.0 > Remove "experimental" warning message from fs2cs > > > Key: YARN-10570 > URL: https://issues.apache.org/jira/browse/YARN-10570 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Labels: fs2cs > Fix For: 3.4.0 > > Attachments: YARN-10570-001.patch > > > Although {{fs2cs}} tool has been in constant development, it's been used and > tested by a group of people, so let's remove the following message: > {{WARNING: This feature is experimental and not intended for production use!}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10570) Remove "experimental" warning message from fs2cs
[ https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263363#comment-17263363 ] Szilard Nemeth commented on YARN-10570: --- Thanks [~pbacsko] for this patch. Straightforward change, LGTM, commited to trunk. Resolving jira. > Remove "experimental" warning message from fs2cs > > > Key: YARN-10570 > URL: https://issues.apache.org/jira/browse/YARN-10570 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Labels: fs2cs > Attachments: YARN-10570-001.patch > > > Although {{fs2cs}} tool has been in constant development, it's been used and > tested by a group of people, so let's remove the following message: > {{WARNING: This feature is experimental and not intended for production use!}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7200) SLS generates a realtimetrack.json file but that file is missing the closing ']'
[ https://issues.apache.org/jira/browse/YARN-7200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263356#comment-17263356 ] Szilard Nemeth commented on YARN-7200: -- Hi [~akshink], Thanks for the explanation, makes sense. I also checked the code and I'm with the same opinion, I don't see a way in the SLS framework that would allow delayed executions of certain AMs so the scenario I described with [my comment here|https://issues.apache.org/jira/browse/YARN-7200?focusedCommentId=17252048&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17252048] is not possible in reality. Some thoughts: 1. SLSCapacityScheduler / SLSFairScheduler: The code block that is added is the same for both: {code} if (SLSRunner.getRemainingApps() == 0) { try { getSchedulerMetrics().tearDown(); SLSRunner.exitSLSRunner(); } catch (Exception e) { e.printStackTrace(); } } {code} I can see that there's no common parent for these classes, so don't spend time with code deduplication here as there's another jira for that: YARN-10552. However, invoking printStackTrace is not the best, as we want to leverage the underlying logging system to print the exception details into the configured outputs (files, console, anything). Printstacktrace only prints it to the standard error, so this is a limitation. Please use a LOG.error statement, add a message and pass the exception as an argument. 2. org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics#tearDown It's okay that metricsLogBW is set to null. I can see 3 write calls of this field in org.apache.hadoop.yarn.sls.scheduler.SchedulerMetrics.MetricsLogRunnable. How it is guaranteed that these won't be invoked after teardown has been executed? Please fix these 2 issues and we're good to go. > SLS generates a realtimetrack.json file but that file is missing the closing > ']' > > > Key: YARN-7200 > URL: https://issues.apache.org/jira/browse/YARN-7200 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler-load-simulator >Reporter: Grant Sohn >Assignee: Agshin Kazimli >Priority: Minor > Labels: newbie, newbie++ > Attachments: YARN-7200-branch-trunk.patch, YARN-7200.002.patch, > YARN-7200.003.patch, YARN-7200.004.patch, snemeth-testing-20201113.zip > > > File > hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SchedulerMetrics.java > shows: > {noformat} > void tearDown() throws Exception { > if (metricsLogBW != null) { > metricsLogBW.write("]"); > metricsLogBW.close(); > } > > {noformat} > So the exit logic is flawed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10570) Remove "experimental" warning message from fs2cs
[ https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263325#comment-17263325 ] Hadoop QA commented on YARN-10570: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 29s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 4s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 24s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 52s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 29s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 48s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 47s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04
[jira] [Commented] (YARN-4589) Diagnostics for localization timeouts is lacking
[ https://issues.apache.org/jira/browse/YARN-4589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263298#comment-17263298 ] Hadoop QA commented on YARN-4589: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 11s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 42s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 10s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 22m 0s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 23s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 21s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 7m 29s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 34m 28s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 35m 41s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 17s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 21m 10s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 21m 10s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m 15s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 18m 15s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 17s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 20m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} There were no new shellcheck issues. {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:gree
[jira] [Updated] (YARN-7713) Add parallel copying of directories into FSDownload
[ https://issues.apache.org/jira/browse/YARN-7713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-7713: - Labels: newbie pull-request-available (was: newbie) > Add parallel copying of directories into FSDownload > --- > > Key: YARN-7713 > URL: https://issues.apache.org/jira/browse/YARN-7713 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Miklos Szegedi >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > YARN currently copies directories sequentially when localizing. This could be > improved to do in parallel, since the source blocks are normally on different > nodes. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263252#comment-17263252 ] VADAGA ANANYO RAO edited comment on YARN-10559 at 1/12/21, 11:09 AM: - Just FYI, the UT failure is not related to the code changes in this JIRA. was (Author: ananyo_rao): Just FYI, the UT failure is not related to the code changes. > Fair sharing intra-queue preemption support in Capacity Scheduler > - > > Key: YARN-10559 > URL: https://issues.apache.org/jira/browse/YARN-10559 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.1.4 >Reporter: VADAGA ANANYO RAO >Assignee: VADAGA ANANYO RAO >Priority: Major > Attachments: FairOP_preemption-design_doc_v1.pdf, > FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch, > YARN-10559.0002.patch, YARN-10559.0003.patch, YARN-10559.0004.patch, > YARN-10559.0005.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Usecase: > Due to the way Capacity Scheduler preemption works, If a single user submits > a large application to a queue (using 100% of resources), that job will not > be preempted by future applications from the same user within the same queue. > This implies that the later applications will be forced to wait for > completion of the long running application. This prevents multiple long > running, large, applications from running concurrently. > Support fair sharing among apps while preempting applications from same queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263252#comment-17263252 ] VADAGA ANANYO RAO commented on YARN-10559: -- Just FYI, the UT failure is not related to the code changes. > Fair sharing intra-queue preemption support in Capacity Scheduler > - > > Key: YARN-10559 > URL: https://issues.apache.org/jira/browse/YARN-10559 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacityscheduler >Affects Versions: 3.1.4 >Reporter: VADAGA ANANYO RAO >Assignee: VADAGA ANANYO RAO >Priority: Major > Attachments: FairOP_preemption-design_doc_v1.pdf, > FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch, > YARN-10559.0002.patch, YARN-10559.0003.patch, YARN-10559.0004.patch, > YARN-10559.0005.patch > > Original Estimate: 168h > Remaining Estimate: 168h > > Usecase: > Due to the way Capacity Scheduler preemption works, If a single user submits > a large application to a queue (using 100% of resources), that job will not > be preempted by future applications from the same user within the same queue. > This implies that the later applications will be forced to wait for > completion of the long running application. This prevents multiple long > running, large, applications from running concurrently. > Support fair sharing among apps while preempting applications from same queue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263233#comment-17263233 ] Hadoop QA commented on YARN-10559: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 5s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 8s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 58s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 55s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 31s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 10s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green}{color} | {col
[jira] [Updated] (YARN-10570) Remove "experimental" warning message from fs2cs
[ https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10570: Labels: fs2cs (was: ) > Remove "experimental" warning message from fs2cs > > > Key: YARN-10570 > URL: https://issues.apache.org/jira/browse/YARN-10570 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Labels: fs2cs > Attachments: YARN-10570-001.patch > > > Although {{fs2cs}} tool has been in constant development, it's been used and > tested by a group of people, so let's remove the following message: > {{WARNING: This feature is experimental and not intended for production use!}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10570) Remove "experimental" warning message from fs2cs
[ https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated YARN-10570: Attachment: YARN-10570-001.patch > Remove "experimental" warning message from fs2cs > > > Key: YARN-10570 > URL: https://issues.apache.org/jira/browse/YARN-10570 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Attachments: YARN-10570-001.patch > > > Although {{fs2cs}} tool has been in constant development, it's been used and > tested by a group of people, so let's remove the following message: > {{WARNING: This feature is experimental and not intended for production use!}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263220#comment-17263220 ] Hadoop QA commented on YARN-10559: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 44s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 54s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 47s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 33s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/468/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 7 new + 26 unchanged - 0 fixed = 33 total (was 26) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 42s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {col
[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation
[ https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263211#comment-17263211 ] Hadoop QA commented on YARN-10506: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 54s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 58s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 53s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 51s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 59s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 42s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/467/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 34 new + 738 unchanged - 1 fixed = 772 total (was 739) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 45s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {c
[jira] [Created] (YARN-10570) Remove "experimental" warning message from fs2cs
Peter Bacsko created YARN-10570: --- Summary: Remove "experimental" warning message from fs2cs Key: YARN-10570 URL: https://issues.apache.org/jira/browse/YARN-10570 Project: Hadoop YARN Issue Type: Sub-task Reporter: Peter Bacsko Assignee: Peter Bacsko Although {{fs2cs}} tool has been in constant development, it's been used and tested by a group of people, so let's remove the following message: {{WARNING: This feature is experimental and not intended for production use!}} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org