[
https://issues.apache.org/jira/browse/TWILL-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yuliya Feldman updated TWILL-252:
---------------------------------
Description:
Looks like when YARN is configured with max memory per container
(yarn.scheduler.maximum-allocation-mb) less then amount of memory end user
allocates for their application and container is allocated with just
yarn.scheduler.maximum-allocation-mb value there is no way to know about it
until container is allocated.
We try to divide memory into heap and off-heap and end up with setting up off
heap to the value higher then allocated for the container, as application
assumes it gets what it asked for or container is not allocated at all.
Need either ability to fail application in this case or not allocate container
with memory less then asked.
As currently Twill adjusts memory and cpu with only INFO level messages in
AppMaster log:
from Hadoop21YarnAMClient.java
{code:java}
protected Resource adjustCapability(Resource resource) {
int cores = resource.getVirtualCores();
int updatedCores = Math.min(resource.getVirtualCores(),
maxCapability.getVirtualCores());
if (cores != updatedCores) {
resource.setVirtualCores(updatedCores);
LOG.info("Adjust virtual cores requirement from {} to {}.", cores,
updatedCores);
}
int updatedMemory = Math.min(resource.getMemory(),
maxCapability.getMemory());
if (resource.getMemory() != updatedMemory) {
resource.setMemory(updatedMemory);
LOG.info("Adjust memory requirement from {} to {} MB.",
resource.getMemory(), updatedMemory);
}
return resource;
}
{code}
was:
Looks like when YARN is configured with max memory per container
(yarn.scheduler.maximum-allocation-mb) less then amount of memory end user
allocates for their application and container is allocated with just
yarn.scheduler.maximum-allocation-mb value there is no way to know about it
until container is allocated.
We try to divide memory into heap and off-heap and end up with setting up off
heap to the value higher then allocated for the container, as application
assumes it gets what it asked for or container is not allocated at all.
Need either ability to fail application in this case or not allocate container
with memory less then asked.
> Not providing any feedback when size of the container requested can't be
> allocated
> ----------------------------------------------------------------------------------
>
> Key: TWILL-252
> URL: https://issues.apache.org/jira/browse/TWILL-252
> Project: Apache Twill
> Issue Type: Bug
> Reporter: Yuliya Feldman
>
> Looks like when YARN is configured with max memory per container
> (yarn.scheduler.maximum-allocation-mb) less then amount of memory end user
> allocates for their application and container is allocated with just
> yarn.scheduler.maximum-allocation-mb value there is no way to know about it
> until container is allocated.
> We try to divide memory into heap and off-heap and end up with setting up off
> heap to the value higher then allocated for the container, as application
> assumes it gets what it asked for or container is not allocated at all.
> Need either ability to fail application in this case or not allocate
> container with memory less then asked.
> As currently Twill adjusts memory and cpu with only INFO level messages in
> AppMaster log:
> from Hadoop21YarnAMClient.java
> {code:java}
> protected Resource adjustCapability(Resource resource) {
> int cores = resource.getVirtualCores();
> int updatedCores = Math.min(resource.getVirtualCores(),
> maxCapability.getVirtualCores());
> if (cores != updatedCores) {
> resource.setVirtualCores(updatedCores);
> LOG.info("Adjust virtual cores requirement from {} to {}.", cores,
> updatedCores);
> }
> int updatedMemory = Math.min(resource.getMemory(),
> maxCapability.getMemory());
> if (resource.getMemory() != updatedMemory) {
> resource.setMemory(updatedMemory);
> LOG.info("Adjust memory requirement from {} to {} MB.",
> resource.getMemory(), updatedMemory);
> }
> return resource;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
