[
https://issues.apache.org/jira/browse/YARN-8808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16626864#comment-16626864
]
Arun Suresh commented on YARN-8808:
-----------------------------------
{quote}Not sure if I follow you correctly. Sounds like you are referring to
aggregateUtilization as the aggregate resource ALLOCATION of all containers,
right?
{quote}
Hmmm.. nope, I meant {{aggregateContainerUtilization}}. Ah.. Apologize if my
comment was not clear: I did not mean {{aggregateUtilization _DIVIDED BY_
nodeUtilization}}, I meant {{aggregateUtilization _OR_ nodeUtilization}}.
I was just saying.. we need an additional check to see if either one of them
(you are proposing to use the former in this JIRA) is {{0}}.. in which case, we
should return {{Resources.none()}} since we do not want to over-allocate on a
node which hasn't allocated anything yet right ?
I hope, now this makes more sense:
bq. Also, I am thinking a combination of containerUtilization + nodeUtilization
should be used though. Consider the situation where the container utilization
is high but the node utilization is low - Node has capacity for 4 1GB
containers, but is currently running 2 containers each using more than 1.9GB -
in this case, overallocation should be allowed.
Again, in the below comment - I was just saying if either one is null, there is
a case where we get an NPE, since {{ResourceUtilization.newInstance(..)}} cant
handle null arguments.
bq. Also, it looks like you need to null check getNodeUtilization() /
getAggregatedContainersUtilization() - there seems to be case where you can get
an NPE if you don't
> Use aggregate container utilization instead of node utilization to determine
> resources available for oversubscription
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8808
> URL: https://issues.apache.org/jira/browse/YARN-8808
> Project: Hadoop YARN
> Issue Type: Sub-task
> Affects Versions: YARN-1011
> Reporter: Haibo Chen
> Assignee: Haibo Chen
> Priority: Major
> Attachments: YARN-8088-YARN-1011.01.patch,
> YARN-8808-YARN-1011.00.patch
>
>
> Resource oversubscription should be bound to the amount of the resources that
> can be allocated to containers, hence the allocation threshold should be with
> respect to aggregate container utilization.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
