[
https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321091#comment-15321091
]
Carlo Curino commented on YARN-5215:
------------------------------------
This sounds a generally good idea (patch obviously need work). It would be able
for example to take into account of HDFS resource consumption,
or more generally if other services are run on the same box and have
non-constant resource utilization we would not need to pessimistically bound
the
resources given to YARN.
Questions:
# How do we ensure that there are no weird feedback loops, e.g., a task is
schedule and consume lots of resources, and as a consequence the scheduler
lower the load on the node, and this task graphs even more resources? For
CPU/Mem we might rely on enforcement, but what about adding non-enforced
resources?
# Would you also trigger preemption based on this? Or only avoid scheduling
more load if the node is busy?
# What is the interplay between this and the work on Overcommit?
# Patch looks very simple/small for this feature, is that all there is needed
here? More dependencies?
# How do we test this till we are convinced works? (are you using it anywhere?)
[~kasha], [~kkaranasos], [~asuresh] can you guys comment on this?
> Scheduling containers based on load in the servers
> --------------------------------------------------
>
> Key: YARN-5215
> URL: https://issues.apache.org/jira/browse/YARN-5215
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Inigo Goiri
> Attachments: YARN-5215.000.patch
>
>
> Currently YARN runs containers in the servers assuming that they own all the
> resources. The proposal is to use the utilization information in the node and
> the containers to estimate how much is actually available in the NMs.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
