[
https://issues.apache.org/jira/browse/YARN-7575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16269175#comment-16269175
]
Eric Payne commented on YARN-7575:
----------------------------------
[~sunilg], the fix for the UI NPE looks good, but the other problem I'm having
is that when I increase a node size, the queue doesn't grow.
My configs are as follows:
- 4 node managers, 5120GB and 10 Vcores each for a total of [20480GB, 40 VCores]
- {{yarn.scheduler.capacity.root.default.capacity}}: [memory=10240,vcores=20]
- {{yarn.scheduler.capacity.root.eng.capacity}}: [memory=10240,vcores=20]
- Note that I do not set root.capacity, nor do I set any maximum-capacity.
My use case is as follows:
- I start a job requesting 22.5GB and 45 vcores (container size=0.5GB)
- the job consumes 20GB and 40 vcores
- I add 2.5GB and 5 vcores to one of the nodes:
{{yarn rmadmin -updateNodeResource host:port 7680 15}}
- One more container is assigned to the job, but that only brings the job to
20.5GB and 41 vcores.
> When using absolute capacity configuration with no max capacity, scheduler UI
> NPEs and can't grow queue
> -------------------------------------------------------------------------------------------------------
>
> Key: YARN-7575
> URL: https://issues.apache.org/jira/browse/YARN-7575
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
> Reporter: Eric Payne
> Attachments: YARN-7575-YARN-5881.001.patch
>
>
> I encountered the following while reviewing and testing branch YARN-5881.
> The design document from YARN-5881 says that for max-capacity:
> {quote}
> 3) For each queue, we require:
> a) if max-resource not set, it automatically set to parent.max-resource
> {quote}
> When I try leaving blank {{yarn.scheduler.capacity.<
> queue-path>.maximum-capacity}}, the RMUI scheduler page refuses to render. It
> looks like it's in {{CapacitySchedulerPage$ LeafQueueInfoBlock}}:
> {noformat}
> 2017-11-28 11:29:16,974 [qtp43473566-220] ERROR webapp.Dispatcher: error
> handling URI: /cluster/scheduler
> java.lang.reflect.InvocationTargetException
> ...
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderQueueCapacityInfo(CapacitySchedulerPage.java:164)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.CapacitySchedulerPage$LeafQueueInfoBlock.renderLeafQueueInfoWithoutParition(CapacitySchedulerPage.java:129)
> {noformat}
> Also... A job will run in the leaf queue with no max capacity set and it will
> grow to the max capacity of the cluster, but if I add resources to the node,
> the job won't grow any more even though it has pending resources.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
