[ https://issues.apache.org/jira/browse/YARN-5215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15321696#comment-15321696 ]
Carlo Curino commented on YARN-5215: ------------------------------------ [~jlowe] thanks for the comment, very useful for context, and you bring up good points on how users "perceive" the cluster. [~elgoiri], correct me if I am wrong, but this feature seems ideal to "scavenge" a YARN cluster out of otherwise utilized machines. In these settings, users should be aware that the cluster is not constant, i.e., the effects of the fluctuations are non-trivial and expected. However, I agree with you that surfacing them in the UI somehow is important. All in all, I see a strong connection with over-commit, but this should be represented not just as a heavily overcommitted cluster. I agree with [~elgoiri] that it is useful to build this feature in a way that more explicitly acknowledges that YARN is not the only thing running on the cluster. At the same time, we should try to have a set of configurable that makes over/under-commit appear unified and coherent to the admins, and UIs that surface them properly to users. [~elgoiri] since you were involved in YARN-1011, can you propose a way to do that? > Scheduling containers based on external load in the servers > ----------------------------------------------------------- > > Key: YARN-5215 > URL: https://issues.apache.org/jira/browse/YARN-5215 > Project: Hadoop YARN > Issue Type: Improvement > Reporter: Inigo Goiri > Attachments: YARN-5215.000.patch > > > Currently YARN runs containers in the servers assuming that they own all the > resources. The proposal is to use the utilization information in the node and > the containers to estimate how much is consumed by external processes and > schedule based on this estimation. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org