[
https://issues.apache.org/jira/browse/YARN-1089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13777071#comment-13777071
]
Sandy Ryza commented on YARN-1089:
----------------------------------
As was requested, I posted a summary of the proposal on YARN-1024.
In case it's not clear on the summary, here's the problem we're trying to solve:
We want jobs to be portable between clusters. CPU is not a fluid resource in
the way memory is. The number of cores on a machine is just as important its
total processing power when scheduling tasks.
Imagine a cluster where every node has powerful CPUs with many cores. One type
of task that will be run on the cluster saturates a full CPU, but another type
of task that will be run on the cluster contains two threads, each which can
saturate only half a full CPU. If we have a single dimension for CPU requests,
these tasks will request an equal number of those. What happens if we then
move those tasks to a cluster with CPUs whose cores are half as fast? The
first task will run half as fast, and the second task will run in the same
amount of time. It's in the first task's interest to only request half as many
CPU resources on that cluster.
I'm also afraid of things getting complicated, but I can't think of anything
better that doesn't require having the meaning of a virtual core vary widely
from cluster to cluster.
> Add YARN compute units alongside virtual cores
> ----------------------------------------------
>
> Key: YARN-1089
> URL: https://issues.apache.org/jira/browse/YARN-1089
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
> Affects Versions: 2.1.0-beta
> Reporter: Sandy Ryza
> Assignee: Sandy Ryza
> Attachments: YARN-1089-1.patch, YARN-1089.patch
>
>
> Based on discussion in YARN-1024, we will add YARN compute units as a
> resource for requesting and scheduling CPU processing power.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
