[
https://issues.apache.org/jira/browse/YARN-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741191#comment-13741191
]
Sandy Ryza commented on YARN-1024:
----------------------------------
bq. To paraphrase your proposal...
Thanks for summarizing. That captures it perfectly.
bq, The one issue that we need to think about is that we'll need to enhance the
schedulers to track how much YCUs are available on which core on any given
node... you could have 5 YCUs in a node but split 3-2-1 across 3 cores. Any
good ideas on how to get to this?
Correct me if I'm wrong, but you're talking about availability based on usage,
not heterogeneous cores within a node, right? If my assumptions about how we
can view CPUs are sufficient, I'm thinking we shouldn't need this, at least for
a start. I.e. if we have two threads to run and two cores to run them on, we
can be agnostic to whether the OS scheduler is running each on its own core or
splitting both across two. The CGroups properties discussed in YARN-810 allow
you to limit the total processing power that a process gets without pinning its
threads to cores. Assigning tasks to cores might matter for things like cache
performance, so I agree it's a useful thing to work on eventually. But I think
any solution will either end up with a decent amount of fragmentation or
require doing some NP-hard combinatorial optimization repeatedly.
> Define a virtual core unambigiously
> -----------------------------------
>
> Key: YARN-1024
> URL: https://issues.apache.org/jira/browse/YARN-1024
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
>
> We need to clearly define the meaning of a virtual core unambiguously so that
> it's easy to migrate applications between clusters.
> For e.g. here is Amazon EC2 definition of ECU:
> http://aws.amazon.com/ec2/faqs/#What_is_an_EC2_Compute_Unit_and_why_did_you_introduce_it
> Essentially we need to clearly define a YARN Virtual Core (YVC).
> Equivalently, we can use ECU itself: *One EC2 Compute Unit provides the
> equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor.*
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
