[
https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626131#comment-16626131
]
Haibo Chen commented on YARN-8808:
----------------------------------
Hi [~asuresh]. Not sure if I follow you correctly. Sounds like you are
referring to aggregateUtilization as the aggregate resource *ALLOCATION* of all
containers, right?
In the case of Fair Scheduler + oversubscription, aggregateUtilization is the
aggregate resource *UTILIZATION* of all containers on a node (rather than
aggregate allocation).
The issue here came up in our testing configuration where only a fraction of
the node's hardware resources is allowed to run containers (Say the node has
10GB memory and 10 vcores, but through configuration we allow the RM to only
see 8GB and 8vcores). Therefore, I think the scheduler side should just see
numbers based on the configured node capacity (8GB, 8vcores). nodeUtlization by
default, is detected from OS and is therefore based on the node's actually
capacity (10GB, 10vcores).
aggregateUtilization/nodeUtilization would tell us how much percentage of the
node utilization is attributed to running containers, no?
> Use aggregate container utilization instead of node utilization to determine
> resources available for oversubscription
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8808
> URL: https://issues.apache.org/jira/browse/YARN-8808
> Project: Hadoop YARN
> Issue Type: Sub-task
> Affects Versions: YARN-1011
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Major
> Attachments: YARN-8088-YARN-1011.01.patch,
> YARN-8808-YARN-1011.00.patch
>
>
> Resource oversubscription should be bound to the amount of the resources that
> can be allocated to containers, hence the allocation threshold should be with
> respect to aggregate container utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
