[jira] [Created] (YARN-4860) Created Node label disappear after restart Resoure Manager
Yi Zhou created YARN-4860: - Summary: Created Node label disappear after restart Resoure Manager Key: YARN-4860 URL: https://issues.apache.org/jira/browse/YARN-4860 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 2.6.0 Reporter: Yi Zhou In 2.6, if restart RM, it cause created node label to disappear and rm failed to start up Error starting ResourceManager org.apache.hadoop.service.ServiceStateException: java.io.IOException: NodeLabelManager doesn't include label = y, please check. at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:172) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:569) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1000) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1221) Caused by: java.io.IOException: NodeLabelManager doesn't include label = y, please check. at org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkIfLabelInClusterNodeLabels(SchedulerUtils.java:287) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:106) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:120) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:569) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:464) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:296) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) ... 7 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208499#comment-15208499 ] Yi Zhou commented on YARN-4847: --- I have simulated the negative case successfully. Thanks [~Naganarasimha] for your patience :) ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208093#comment-15208093 ] Yi Zhou commented on YARN-4847: --- Thanks! I will double check this. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208092#comment-15208092 ] Yi Zhou commented on YARN-4847: --- Thanks! I will double check this. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15208021#comment-15208021 ] Yi Zhou commented on YARN-4847: --- Hi [~Naganarasimha] Thank you for your great help! OK. If it is relative to doc, i will input here. i will post my issues in mailing list.. > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15207938#comment-15207938 ] Yi Zhou commented on YARN-4847: --- Hi [~Naganarasimha], Could you please have a chance to check the issues i came across ? Thanks in advance ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15206002#comment-15206002
]
Yi Zhou commented on YARN-4847:
---
Thanks again [~Naganarasimha]!
Now i have below CS configuration. I issued a admin operation like below and my
question is why the 'm' queue will run successfully instead of job failure ?
The m queue should not have permission to access the node label y.
{code}
sudo -u yarn yarn jar
/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar -jar
/usr/lib/hadoop-yarn/hadoop-yarn-applications-distributedshell.jar
-shell_command sleep -shell_args 30 -num_containers 20 -container_memory 16384
-queue m -node_label_expression y
{code}
{code}
yarn.scheduler.capacity.root.queues
e,m,default
yarn.scheduler.capacity.root.e.capacity
50
yarn.scheduler.capacity.root.m.capacity
25
yarn.scheduler.capacity.root.default.capacity
25
yarn.scheduler.capacity.root.e.accessible-node-labels
y
yarn.scheduler.capacity.root.accessible-node-labels.y.capacity
100
yarn.scheduler.capacity.root.e.accessible-node-labels.y.capacity
100
yarn.scheduler.capacity.root.e.default-node-label-expression
y
{code}
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205953#comment-15205953
]
Yi Zhou commented on YARN-4847:
---
Hi [~Naganarasimha]
Could you please help to check below that the create node label is a shareable
node label instead of exclusive one in 2.6 ? since i have some of confused test
result for the node label behavior in 2.6 Thanks in advance !
{code}
$ sudo -u yarn yarn rmadmin -addToClusterNodeLabels "y"
{code}
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205722#comment-15205722
]
Yi Zhou commented on YARN-4847:
---
Hi,
The created node labels disappear after restart resource manager(i have a node
label 'y'). It is a bug or limitation for node label feature ?
{code}
2016-03-22 11:31:28,936 FATAL
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting
ResourceManager
org.apache.hadoop.service.ServiceStateException: java.io.IOException:
NodeLabelManager doesn't include label = y, please check.
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:59)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:172)
at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:569)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1000)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:262)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at
org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1221)
Caused by: java.io.IOException: NodeLabelManager doesn't include label = y,
please check.
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerUtils.checkIfLabelInClusterNodeLabels(SchedulerUtils.java:287)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:106)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:120)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:569)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:589)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initializeQueues(CapacityScheduler.java:464)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.initScheduler(CapacityScheduler.java:296)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.serviceInit(CapacityScheduler.java:326)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
... 7 more
{code}
Thanks
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205677#comment-15205677
]
Yi Zhou commented on YARN-4847:
---
Hi,
I found if don't configure a 'default' queue with capacity in
capacity-scheduler.xml it will get failure when refresh queues(Restart resource
manager). I am not sure there is a limitation in CS of 2.6 ?
{code}
yarn.scheduler.capacity.root.queues
e,m,default
yarn.scheduler.capacity.root.default.capacity
25
{code}
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205567#comment-15205567 ] Yi Zhou commented on YARN-4847: --- Add 5) There is no 'Node Labels' in left navigation on 'ResourceManager' UI even after enabling the node label feature in Hadoop 2.6 > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[
https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205498#comment-15205498
]
Yi Zhou commented on YARN-4847:
---
Hi [~Naganarasimha], [~sunilg]
I summarized the below issues i came across in 2.6 so far..And i will make
progress to see others. Thank you for your patience !
1) Node label exclusivity is not supported like below command
sudo -u yarn yarn rmadmin -addToClusterNodeLabels "c(exclusive=false)"
addToClusterNodeLabels: java.io.IOException: label name should only
contains
{0-9, a-z, A-Z, -, _} and should not started with {-,_}, now it
is=c(exclusive=false)
2) Node labels are supported only in Capacity Scheduler
3) Listing node labels are not supported like below command
sudo -u yarn yarn cluster --list-node-labels
Error: Could not find or load main class cluster
4) Assign Node Labels to Cluster Node
Not supported format - sudo -u yarn yarn rmadmin -replaceLabelsOnNode
"bb-node3=y"
Supported format - sudo -u yarn yarn rmadmin -replaceLabelsOnNode
"bb-node3,y"
> Add documentation for the Node Label features supported in 2.6
> ---
>
> Key: YARN-4847
> URL: https://issues.apache.org/jira/browse/YARN-4847
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: api, client, resourcemanager
>Affects Versions: 2.6.4
>Reporter: Naganarasimha G R
>Assignee: Naganarasimha G R
>
> We constantly face issue with what are the node label supported features in
> 2.6 and general commands to use it. So it would be better to have
> documentation capturing what all is supported as part of 2.6
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4847) Add documentation for the Node Label features supported in 2.6
[ https://issues.apache.org/jira/browse/YARN-4847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204195#comment-15204195 ] Yi Zhou commented on YARN-4847: --- [~Naganarasimha] Thanks a lot for this work ! > Add documentation for the Node Label features supported in 2.6 > --- > > Key: YARN-4847 > URL: https://issues.apache.org/jira/browse/YARN-4847 > Project: Hadoop YARN > Issue Type: Sub-task > Components: api, client, resourcemanager >Affects Versions: 2.6.4 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > > We constantly face issue with what are the node label supported features in > 2.6 and general commands to use it. So it would be better to have > documentation capturing what all is supported as part of 2.6 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204191#comment-15204191 ] Yi Zhou commented on YARN-796: -- BTW, i am confused that there is no 'Node Labels' in left navigation on 'ResourceManager' UI even after enabling the node label feature..It is not yet supported in 2.6 ? > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204184#comment-15204184 ] Yi Zhou commented on YARN-796: -- I configured below in capacity-scheduler.xml and try to restart RM but it always got failure with "Illegal capacity of 1.0 for children of queue root for label=y". I think the below setting for CS seem be OK and not very sure why it happened. Could you please help me to have a look for this ? Thanks ! yarn.scheduler.capacity.root.queues e,m,s yarn.scheduler.capacity.root.e.capacity 33 yarn.scheduler.capacity.root.m.capacity 34 yarn.scheduler.capacity.root.s.capacity 33 yarn.scheduler.capacity.root.e.accessible-node-labels y yarn.scheduler.capacity.root.m.accessible-node-labels y yarn.scheduler.capacity.root.e.accessible-node-labels.y.capacity 50 yarn.scheduler.capacity.root.m.accessible-node-labels.y.capacity 50 yarn.scheduler.capacity.root.e.default-node-label-expression y Caused by: java.io.IOException: Failed to re-init queues at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:358) at org.apache.hadoop.yarn.server.resourcemanager.AdminService.refreshQueues(AdminService.java:363) ... 10 more Caused by: java.lang.IllegalArgumentException: Illegal capacity of 1.0 for children of queue root for label=y at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.setChildQueues(ParentQueue.java:184) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.parseQueue(CapacityScheduler.java:594) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitializeQueues(CapacityScheduler.java:477) at org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.reinitialize(CapacityScheduler.java:355) ... 11 more > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204117#comment-15204117 ] Yi Zhou commented on YARN-796: -- Hi, Successfully assigned the node label as per your instruction in 2.6 ! I will continue to make progress.. sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node4,y" 16/03/21 20:38:03 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8033 [root@bb-node1 ~]# sudo -u yarn yarn node -status bb-node4:8041 16/03/21 20:39:09 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8032 Node Report : Node-Id : bb-node4:8041 Rack : /default Node-State : RUNNING Node-Http-Address : bb-node4:8042 Last-Health-Update : Mon 21/Mar/16 08:39:06:548CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 65536MB CPU-Used : 0 vcores CPU-Capacity : 40 vcores Node-Labels : y > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204100#comment-15204100 ] Yi Zhou commented on YARN-796: -- OK i will also try it as per your suggestion in my cluster. Thanks again guys ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15204071#comment-15204071 ] Yi Zhou commented on YARN-796: -- Hi, I found there can't take effect after assigning a created node label to a node in 2.6.0. You can find that there is EMPTY in ''Node-Labels :" row by 'yarn node -status XX'. Please kindly see below steps for details. If there is a workaround for this issue ? Thanks advance ! step: 1) Create a Label Directory in HDFS sudo -u hdfs hadoop fs -mkdir -p /yarn/node-labels sudo -u hdfs hadoop fs -chown -R yarn:yarn /yarn sudo -u hdfs hadoop fs -chmod -R 700 /yarn sudo -u hdfs hadoop fs -ls /yarn 2) Configure YARN for Node Labels yarn.node-labels.enabled true yarn.node-labels.fs-store.root-dir hdfs://bb-node1:8020/yarn/node-labels/ 3) Restart RM service 4) Add Node Labels sudo -u yarn yarn rmadmin -addToClusterNodeLabels "y" 5) Assign Node Labels to Cluster Node sudo -u yarn yarn rmadmin -replaceLabelsOnNode "bb-node3=y" 6) Check the Node label sudo -u yarn yarn node -status bb-node3:8041 16/03/21 19:37:12 INFO client.RMProxy: Connecting to ResourceManager at bb-node1/192.168.30.1:8032 Node Report : Node-Id : bb-node3:8041 Rack : /default Node-State : RUNNING Node-Http-Address : bb-node3:8042 Last-Health-Update : Mon 21/Mar/16 07:36:42:522CST Health-Report : Containers : 0 Memory-Used : 0MB Memory-Capacity : 65536MB CPU-Used : 0 vcores CPU-Capacity : 40 vcores Node-Labels : > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203759#comment-15203759 ] Yi Zhou commented on YARN-796: -- Hi [~Naganarasimha], If you finished the jira for 2.6 doc, please kindly posted the ID number for me to track and reference. Thanks a lot ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203757#comment-15203757 ] Yi Zhou commented on YARN-796: -- Appreciate[~Naganarasimha] [~wangda] for you great help! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15203684#comment-15203684 ] Yi Zhou commented on YARN-796: -- Hi [~Naganarasimha] It seems the below commands are still not supported in 2.6.0 ? sudo -u yarn yarn cluster --list-node-labels Error: Could not find or load main class cluster > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197300#comment-15197300 ] Yi Zhou commented on YARN-796: -- Thanks [~Naganarasimha] > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[ https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197169#comment-15197169 ] Yi Zhou commented on YARN-796: -- Thank you [~Naganarasimha Garla] for you response. I used CDH 5.5-Hadoop 2.6.0. I believed it is based on Hadoop 2.6.0. Have you some of hints or suggestion for this issue i run across ? Thanks in advance ! > Allow for (admin) labels on nodes and resource-requests > --- > > Key: YARN-796 > URL: https://issues.apache.org/jira/browse/YARN-796 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.4.1 >Reporter: Arun C Murthy >Assignee: Wangda Tan > Attachments: LabelBasedScheduling.pdf, > Node-labels-Requirements-Design-doc-V1.pdf, > Node-labels-Requirements-Design-doc-V2.pdf, > Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf, > YARN-796.node-label.consolidate.1.patch, > YARN-796.node-label.consolidate.10.patch, > YARN-796.node-label.consolidate.11.patch, > YARN-796.node-label.consolidate.12.patch, > YARN-796.node-label.consolidate.13.patch, > YARN-796.node-label.consolidate.14.patch, > YARN-796.node-label.consolidate.2.patch, > YARN-796.node-label.consolidate.3.patch, > YARN-796.node-label.consolidate.4.patch, > YARN-796.node-label.consolidate.5.patch, > YARN-796.node-label.consolidate.6.patch, > YARN-796.node-label.consolidate.7.patch, > YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1, > YARN-796.patch, YARN-796.patch4 > > > It will be useful for admins to specify labels for nodes. Examples of labels > are OS, processor architecture etc. > We should expose these labels and allow applications to specify labels on > resource-requests. > Obviously we need to support admin operations on adding/removing node labels. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-796) Allow for (admin) labels on nodes and resource-requests
[
https://issues.apache.org/jira/browse/YARN-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15197015#comment-15197015
]
Yi Zhou commented on YARN-796:
--
Hi
I created node label name as per
https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html
but came across below issue. Could you please have a look this issue?
$ sudo -u yarn yarn rmadmin -addToClusterNodeLabels "c(exclusive=false)"
16/03/16 16:25:47 INFO client.RMProxy: Connecting to ResourceManager at
bb-node1/192.168.30.1:8033
addToClusterNodeLabels: java.io.IOException: label name should only contains
{0-9, a-z, A-Z, -, _} and should not started with {-,_}, now it
is=c(exclusive=false)
at
org.apache.hadoop.yarn.ipc.RPCUtil.getRemoteException(RPCUtil.java:38)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.addToClusterNodeLabels(AdminService.java:664)
at
org.apache.hadoop.yarn.server.api.impl.pb.service.ResourceManagerAdministrationProtocolPBServiceImpl.addToClusterNodeLabels(ResourceManagerAdministrationProtocolPBServiceImpl.java:229)
at
org.apache.hadoop.yarn.proto.ResourceManagerAdministrationProtocol$ResourceManagerAdministrationProtocolService$2.callBlockingMethod(ResourceManagerAdministrationProtocol.java:239)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:617)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2086)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2082)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2080)
Caused by: java.io.IOException: label name should only contains {0-9, a-z, A-Z,
-, _} and should not started with {-,_}, now it is=c(exclusive=false)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.checkAndThrowLabelName(CommonNodeLabelsManager.java:662)
at
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.addToCluserNodeLabels(CommonNodeLabelsManager.java:263)
at
org.apache.hadoop.yarn.server.resourcemanager.AdminService.addToClusterNodeLabels(AdminService.java:656)
... 10 more
> Allow for (admin) labels on nodes and resource-requests
> ---
>
> Key: YARN-796
> URL: https://issues.apache.org/jira/browse/YARN-796
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 2.4.1
>Reporter: Arun C Murthy
>Assignee: Wangda Tan
> Attachments: LabelBasedScheduling.pdf,
> Node-labels-Requirements-Design-doc-V1.pdf,
> Node-labels-Requirements-Design-doc-V2.pdf,
> Non-exclusive-Node-Partition-Design.pdf, YARN-796-Diagram.pdf,
> YARN-796.node-label.consolidate.1.patch,
> YARN-796.node-label.consolidate.10.patch,
> YARN-796.node-label.consolidate.11.patch,
> YARN-796.node-label.consolidate.12.patch,
> YARN-796.node-label.consolidate.13.patch,
> YARN-796.node-label.consolidate.14.patch,
> YARN-796.node-label.consolidate.2.patch,
> YARN-796.node-label.consolidate.3.patch,
> YARN-796.node-label.consolidate.4.patch,
> YARN-796.node-label.consolidate.5.patch,
> YARN-796.node-label.consolidate.6.patch,
> YARN-796.node-label.consolidate.7.patch,
> YARN-796.node-label.consolidate.8.patch, YARN-796.node-label.demo.patch.1,
> YARN-796.patch, YARN-796.patch4
>
>
> It will be useful for admins to specify labels for nodes. Examples of labels
> are OS, processor architecture etc.
> We should expose these labels and allow applications to specify labels on
> resource-requests.
> Obviously we need to support admin operations on adding/removing node labels.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
