[
https://issues.apache.org/jira/browse/TWILL-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295294#comment-16295294
]
Terence Yim commented on TWILL-252:
-----------------------------------
Do you expect the application to fail to launch due to unable to acquire
container of the desired size? It can be a static check before the application
is submitted to YARN.
> Not providing any feedback when size of the container requested can't be
> allocated
> ----------------------------------------------------------------------------------
>
> Key: TWILL-252
> URL: https://issues.apache.org/jira/browse/TWILL-252
> Project: Apache Twill
> Issue Type: Bug
> Reporter: Yuliya Feldman
>
> Looks like when YARN is configured with max memory per container
> (yarn.scheduler.maximum-allocation-mb) less then amount of memory end user
> allocates for their application and container is allocated with just
> yarn.scheduler.maximum-allocation-mb value there is no way to know about it
> until container is allocated.
> We try to divide memory into heap and off-heap and end up with setting up off
> heap to the value higher then allocated for the container, as application
> assumes it gets what it asked for or container is not allocated at all.
> Need either ability to fail application in this case or not allocate
> container with memory less then asked.
> As currently Twill adjusts memory and cpu with only INFO level messages in
> AppMaster log:
> from Hadoop21YarnAMClient.java
> {code:java}
> protected Resource adjustCapability(Resource resource) {
> int cores = resource.getVirtualCores();
> int updatedCores = Math.min(resource.getVirtualCores(),
> maxCapability.getVirtualCores());
> if (cores != updatedCores) {
> resource.setVirtualCores(updatedCores);
> LOG.info("Adjust virtual cores requirement from {} to {}.", cores,
> updatedCores);
> }
> int updatedMemory = Math.min(resource.getMemory(),
> maxCapability.getMemory());
> if (resource.getMemory() != updatedMemory) {
> resource.setMemory(updatedMemory);
> LOG.info("Adjust memory requirement from {} to {} MB.",
> resource.getMemory(), updatedMemory);
> }
> return resource;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
