[jira] [Created] (YARN-4860) Created Node label disappear after restart Resoure Manager
Yi Zhou created YARN-4860: - Summary: Created Node label disappear after restart Resoure Manager Key: YARN-4860 URL: https://issues.apache.org/jira/browse/YARN-4860 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Reporter: Yi Zhou In 2.6, if restart RM, it cause created node label to disappear and rm failed to start up Error starting ResourceManager org.apache.hadoop.service.ServiceStateException: java.io.IOException: NodeLabelManager doesn't include label = y, please check. at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:569) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1000) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1221) Caused by: java.io.IOException: NodeLabelManager doesn't include label = y, please check. at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkIfLabelInClusterNodeLabels(SchedulerUtils.java:287) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:106) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:120) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:569) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:464) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:296) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 7 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208499#comment-15208499 ] Yi Zhou commented on YARN-4847: --- I have simulated the negative case successfully. Thanks [~Naganarasimha] for your patience :) ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208093#comment-15208093 ] Yi Zhou commented on YARN-4847: --- Thanks! I will double check this. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208092#comment-15208092 ] Yi Zhou commented on YARN-4847: --- Thanks! I will double check this. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15208021#comment-15208021 ] Yi Zhou commented on YARN-4847: --- Hi [~Naganarasimha] Thank you for your great help! OK. If it is relative to doc, i will input here. i will post my issues in mailing list.. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15207938#comment-15207938 ] Yi Zhou commented on YARN-4847: --- Hi [~Naganarasimha], Could you please have a chance to check the issues i came across ? Thanks in advance ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15206002#comment-15206002 ] Yi Zhou commented on YARN-4847: --- Thanks again [~Naganarasimha]! Now i have below CS configuration. I issued a admin operation like below and my question is why the 'm' queue will run successfully instead of job failure ? The m queue should not have permission to access the node label y. {code} sudo -u yarn yarn jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -jar /usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -shell_command sleep -shell_args 30 -num_containers 20 -container_memory 16384 -queue m -node_label_expression y {code} {code} yarn.scheduler.capacity.root.queues e,m,default yarn.scheduler.capacity.root.e.capacity 50 yarn.scheduler.capacity.root.m.capacity 25 yarn.scheduler.capacity.root.default.capacity 25 yarn.scheduler.capacity.root.e.accessible-node-labels y yarn.scheduler.capacity.root.accessible-node-labels.y.capacity 100 yarn.scheduler.capacity.root.e.accessible-node-labels.y.capacity 100 yarn.scheduler.capacity.root.e.default-node-label-expression y {code} > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205953#comment-15205953 ] Yi Zhou commented on YARN-4847: --- Hi [~Naganarasimha] Could you please help to check below that the create node label is a shareable node label instead of exclusive one in 2.6 ? since i have some of confused test result for the node label behavior in 2.6 Thanks in advance ! {code} $ sudo -u yarn yarn rmadmin -addToClusterNodeLabels "y" {code} > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205722#comment-15205722 ] Yi Zhou commented on YARN-4847: --- Hi, The created node labels disappear after restart resource manager(i have a node label 'y'). It is a bug or limitation for node label feature ? {code} 2016-03-22 11:31:28,936 FATAL org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager org.apache.hadoop.service.ServiceStateException: java.io.IOException: NodeLabelManager doesn't include label = y, please check. at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:569) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1000) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1221) Caused by: java.io.IOException: NodeLabelManager doesn't include label = y, please check. at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkIfLabelInClusterNodeLabels(SchedulerUtils.java:287) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:106) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:120) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:569) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:464) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:296) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 7 more {code} Thanks > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205677#comment-15205677 ] Yi Zhou commented on YARN-4847: --- Hi, I found if don't configure a 'default' queue with capacity in capacity-scheduler.xml it will get failure when refresh queues(Restart resource manager). I am not sure there is a limitation in CS of 2.6 ? {code} yarn.scheduler.capacity.root.queues e,m,default yarn.scheduler.capacity.root.default.capacity 25 {code} > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205567#comment-15205567 ] Yi Zhou commented on YARN-4847: --- Add 5) There is no 'Node Labels' in left navigation on 'ResourceManager' UI even after enabling the node label feature in Hadoop 2.6 > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15205498#comment-15205498 ] Yi Zhou commented on YARN-4847: --- Hi [~Naganarasimha], [~sunilg] I summarized the below issues i came across in 2.6 so far..And i will make progress to see others. Thank you for your patience ! 1) Node label exclusivity is not supported like below command sudo -u yarn yarn rmadmin -addToClusterNodeLabels "c(exclusive=false)" addToClusterNodeLabels: java.io.IOException: label name should only contains {0-9, a-z, A-Z, -, _} and should not started with {-,_}, now it is=c(exclusive=false) 2) Node labels are supported only in Capacity Scheduler 3) Listing node labels are not supported like below command sudo -u yarn yarn cluster --list-node-labels Error: Could not find or load main class cluster 4) Assign Node Labels to Cluster Node Not supported format - sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node3=y" Supported format - sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node3,y" > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204195#comment-15204195 ] Yi Zhou commented on YARN-4847: --- [~Naganarasimha] Thanks a lot for this work ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204191#comment-15204191 ] Yi Zhou commented on YARN-796: -- BTW, i am confused that there is no 'Node Labels' in left navigation on 'ResourceManager' UI even after enabling the node label feature..It is not yet supported in 2.6 ? > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204184#comment-15204184 ] Yi Zhou commented on YARN-796: -- I configured below in capacity-scheduler.xml and try to restart RM but it always got failure with "Illegal capacity of 1.0 for children of queue root for label=y". I think the below setting for CS seem be OK and not very sure why it happened. Could you please help me to have a look for this ? Thanks ! yarn.scheduler.capacity.root.queues e,m,s yarn.scheduler.capacity.root.e.capacity 33 yarn.scheduler.capacity.root.m.capacity 34 yarn.scheduler.capacity.root.s.capacity 33 yarn.scheduler.capacity.root.e.accessible-node-labels y yarn.scheduler.capacity.root.m.accessible-node-labels y yarn.scheduler.capacity.root.e.accessible-node-labels.y.capacity 50 yarn.scheduler.capacity.root.m.accessible-node-labels.y.capacity 50 yarn.scheduler.capacity.root.e.default-node-label-expression y Caused by: java.io.IOException: Failed to re-init queues at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:358) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:363) ... 10 more Caused by: java.lang.IllegalArgumentException: Illegal capacity of 1.0 for children of queue root for label=y at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setChildQueues(ParentQueue.java:184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:594) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:477) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:355) ... 11 more > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204117#comment-15204117 ] Yi Zhou commented on YARN-796: -- Hi, Successfully assigned the node label as per your instruction in 2.6 ! I will continue to make progress.. sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node4,y" 16/03/21 20:38:03 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8033 [root@bb-node1 ~]# sudo -u yarn yarn node -status bb-node4:8041 16/03/21 20:39:09 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8032 Node Report : Node-Id : bb-node4:8041 Rack : /default Node-State : RUNNING Node-Http-Address : bb-node4:8042 Last-Health-Update : Mon 21/Mar/16 08:39:06:548CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 65536MB CPU-Used : 0 vcores CPU-Capacity : 40 vcores Node-Labels : y > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204100#comment-15204100 ] Yi Zhou commented on YARN-796: -- OK i will also try it as per your suggestion in my cluster. Thanks again guys ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15204071#comment-15204071 ] Yi Zhou commented on YARN-796: -- Hi, I found there can't take effect after assigning a created node label to a node in 2.6.0. You can find that there is EMPTY in ''Node-Labels :" row by 'yarn node -status XX'. Please kindly see below steps for details. If there is a workaround for this issue ? Thanks advance ! step: 1) Create a Label Directory in HDFS sudo -u hdfs hadoop fs -mkdir -p /yarn/node-labels sudo -u hdfs hadoop fs -chown -R yarn:yarn /yarn sudo -u hdfs hadoop fs -chmod -R 700 /yarn sudo -u hdfs hadoop fs -ls /yarn 2) Configure YARN for Node Labels yarn.node-labels.enabled true yarn.node-labels.fs-store.root-dir hdfs://bb-node1:8020/yarn/node-labels/ 3) Restart RM service 4) Add Node Labels sudo -u yarn yarn rmadmin -addToClusterNodeLabels "y" 5) Assign Node Labels to Cluster Node sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node3=y" 6) Check the Node label sudo -u yarn yarn node -status bb-node3:8041 16/03/21 19:37:12 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8032 Node Report : Node-Id : bb-node3:8041 Rack : /default Node-State : RUNNING Node-Http-Address : bb-node3:8042 Last-Health-Update : Mon 21/Mar/16 07:36:42:522CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 65536MB CPU-Used : 0 vcores CPU-Capacity : 40 vcores Node-Labels : > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203759#comment-15203759 ] Yi Zhou commented on YARN-796: -- Hi [~Naganarasimha], If you finished the jira for 2.6 doc, please kindly posted the ID number for me to track and reference. Thanks a lot ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203757#comment-15203757 ] Yi Zhou commented on YARN-796: -- Appreciate[~Naganarasimha] [~wangda] for you great help! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15203684#comment-15203684 ] Yi Zhou commented on YARN-796: -- Hi [~Naganarasimha] It seems the below commands are still not supported in 2.6.0 ? sudo -u yarn yarn cluster --list-node-labels Error: Could not find or load main class cluster > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197300#comment-15197300 ] Yi Zhou commented on YARN-796: -- Thanks [~Naganarasimha] > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197169#comment-15197169 ] Yi Zhou commented on YARN-796: -- Thank you [~Naganarasimha Garla] for you response. I used CDH 5.5-Hadoop 2.6.0. I believed it is based on Hadoop 2.6.0. Have you some of hints or suggestion for this issue i run across ? Thanks in advance ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15197015#comment-15197015 ] Yi Zhou commented on YARN-796: -- Hi I created node label name as per https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html but came across below issue. Could you please have a look this issue? $ sudo -u yarn yarn rmadmin -addToClusterNodeLabels "c(exclusive=false)" 16/03/16 16:25:47 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8033 addToClusterNodeLabels: java.io.IOException: label name should only contains {0-9, a-z, A-Z, -, _} and should not started with {-,_}, now it is=c(exclusive=false) at org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.addToClusterNodeLabels(AdminService.java:664) at org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceManagerAdministrationProtocolPBServiceImpl.addToClusterNodeLabels(ResourceManagerAdministrationProtocolPBServiceImpl.java:229) at org.apache.hadoop.yarn.proto.ResourceManagerAdministrationProtocol$ResourceManagerAdministrationProtocolService$2.callBlockingMethod(ResourceManagerAdministrationProtocol.java:239) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080) Caused by: java.io.IOException: label name should only contains {0-9, a-z, A-Z, -, _} and should not started with {-,_}, now it is=c(exclusive=false) at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.checkAndThrowLabelName(CommonNodeLabelsManager.java:662) at org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.addToCluserNodeLabels(CommonNodeLabelsManager.java:263) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.addToClusterNodeLabels(AdminService.java:656) ... 10 more > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)