[
https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13675988#comment-13675988
]
Tom White commented on YARN-689:
--------------------------------
I had a look at the latest patch. Why is
yarn.scheduler.maximum-allocation-vcores in yarn-default.xml changed from 32 to
4? Apart from that change it looks good to me. +1
Arun, I don't know why you are so resistant to this change. It is a
backwards-compatible change for users. Indeed it doesn't change the behaviour
at all by default. It's not a complicated or risky patch. It adds an extra
configuration knob so that instead of the increment being the minimum value (of
memory or cores) it can be a different value to allow for more fine-grained
control in some cases, such as the one that Sandy
[outlined|https://issues.apache.org/jira/browse/YARN-689?focusedCommentId=13661159&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13661159].
How would you configure YARN to handle this situation?
I don't understand the point about fragmentation either. In the scenario you
[describe
above|https://issues.apache.org/jira/browse/YARN-689?focusedCommentId=13673517&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13673517],
I think you are talking about a situation where AMs are asking for containers
that are greater than 1GB in size, e.g. 1.7 or 1.9GB. If so, then today if you
set the minimum to 128MB (to give finer-grained increments) you open yourself
up to the same possibility of fragmentation since there could be nodes with
<1GB left if AMs ask for over 1GB. The patch in this JIRA will not change that
situation (set minimum to 1GB, increment to 128MB) - there will still be the
same number of nodes with <1GB left and the AMs will not be able to use them.
So I don't see how this change makes the situation worse.
In fact I think this change could improve utilization. If you set the minimum
to 1GB then a request for 1.7 or 1.9GB will be changed to 2GB, but this might
not be what the user wants since the extra memory may not be needed (after all,
the AM didn't ask for it) and is a form of poor utilization since it means
there's certainly no way that memory can be used by other containers. In the
case of a smaller increment (128MB) the extra memory would not be allocated to
containers that are not using it, so it can be used by other containers, e.g. 3
x 1.7GB requests and one 1.9GB request could fit in a node with 7GB, whereas
only three of them could fit if the minimum was 1GB.
> Add multiplier unit to resourcecapabilities
> -------------------------------------------
>
> Key: YARN-689
> URL: https://issues.apache.org/jira/browse/YARN-689
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api, scheduler
> Affects Versions: 2.0.4-alpha
> Reporter: Alejandro Abdelnur
> Assignee: Alejandro Abdelnur
> Attachments: YARN-689.patch, YARN-689.patch, YARN-689.patch,
> YARN-689.patch, YARN-689.patch
>
>
> Currently we overloading the minimum resource value as the actual multiplier
> used by the scheduler.
> Today with a minimum memory set to 1GB, requests for 1.5GB are always
> translated to allocation of 2GB.
> We should decouple the minimum allocation from the multiplier.
> The multiplier should also be exposed to the client via the
> RegisterApplicationMasterResponse
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
