[ https://issues.apache.org/jira/browse/YARN-1011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15083957#comment-15083957 ]
Bikas Saha commented on YARN-1011: ---------------------------------- I agree with natural container churn in favor of preemption to avoid lost work though the issue of clearly defining scheduler policy still remains. bq. If we were oversubscribing 10X then I'd probably want it for sure, but if it's at most 2X capacity then worst case is a container only gets 50% of the resource it had requested. Obviously for something like memory this has to be closely controlled because going over the physical capabilities of the machine has very significant consequences. But for CPU, I'd definitely be inclined to live with the occasional 50% worst case for all containers, in order to avoid the 1/1024th worst case for OPPORTUNISTIC containers on a busy node. I did not understand this. Does this mean, its ok for normal containers to run 50% slower in the presence of opportunistic containers? If yes, then there are scenarios where this may not be a valid choice. E.g. when a cluster is running a mix of SLA and non-SLA jobs. Non-SLA jobs are ok if there containers got slowed down to increase cluster utilization by running opportunistic containers because we are getting higher overall throughput. But SLA jobs are not ok with missing deadlines because there tasks ran 50% slower. IMO, the litmus test for a feature like this would be to take an existing cluster (with low utilization because tasks are asking for more resources than what they need 100% of the time). Then turn this feature on and get better cluster utilization and throughput without affecting the existing workload. Whatever be the internal implementation details. Agree? bq. 50% of maximum-under-utilized resource of past 30 min for each NM can be used to allocate opportunistic containers. These are heuristics and may all be valid under different circumstances. What we should step back and see is what is the source of this optimization. Observation : Cluster is under-utilized despite being fully allocated Possible reasons : 1) Tasks are incorrectly over-allocated. Will never use the resources they ask for and hence we can safely run additional opportunistic containers. So this feature is used to compensate for poorly configured applications. Probably a valid scenario but is it common? 2) Tasks are correctly allocated but dont use their capacity to the limit all the time. E.g. Terasort will use high cpu only during the sorting but not during the entire length of the job. But its containers will ask for enough CPU to run the sort in the desired time. This is a typical application behavior where resource usage varies over time. So this feature is used to soak up the fallow resources in the cluster while tasks are not using their quoted capacity. The arguments and assumptions we make need to be considered in the light of which of 1 or 2 is the common case and where this feature will be useful. While its useful to have configuration knobs, for a complex dynamic feature like this that is basically reacting to runtime observations, it may be quite hard to be able to configure this statically using manual configuration. While some limits about max over-allocation limit etc. are easy and probably required to configure, we should look at making this feature work by itself instead of relying exclusively on configuration (hell :P) for users to make this feature usable. > [Umbrella] Schedule containers based on utilization of currently allocated > containers > ------------------------------------------------------------------------------------- > > Key: YARN-1011 > URL: https://issues.apache.org/jira/browse/YARN-1011 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Arun C Murthy > Attachments: yarn-1011-design-v0.pdf, yarn-1011-design-v1.pdf > > > Currently RM allocates containers and assumes resources allocated are > utilized. > RM can, and should, get to a point where it measures utilization of allocated > containers and, if appropriate, allocate more (speculative?) containers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)