[ https://issues.apache.org/jira/browse/TWILL-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16295356#comment-16295356 ]
Yuliya Feldman commented on TWILL-252: -------------------------------------- In order to do this before YARN application is submitted the app that submits YARN app needs to deal with YARN APIs to get information about max container size allowed, while Twill is doing it already - as it adjusts the size. > Not providing any feedback when size of the container requested can't be > allocated > ---------------------------------------------------------------------------------- > > Key: TWILL-252 > URL: https://issues.apache.org/jira/browse/TWILL-252 > Project: Apache Twill > Issue Type: Bug > Reporter: Yuliya Feldman > > Looks like when YARN is configured with max memory per container > (yarn.scheduler.maximum-allocation-mb) less then amount of memory end user > allocates for their application and container is allocated with just > yarn.scheduler.maximum-allocation-mb value there is no way to know about it > until container is allocated. > We try to divide memory into heap and off-heap and end up with setting up off > heap to the value higher then allocated for the container, as application > assumes it gets what it asked for or container is not allocated at all. > Need either ability to fail application in this case or not allocate > container with memory less then asked. > As currently Twill adjusts memory and cpu with only INFO level messages in > AppMaster log: > from Hadoop21YarnAMClient.java > {code:java} > protected Resource adjustCapability(Resource resource) { > int cores = resource.getVirtualCores(); > int updatedCores = Math.min(resource.getVirtualCores(), > maxCapability.getVirtualCores()); > if (cores != updatedCores) { > resource.setVirtualCores(updatedCores); > LOG.info("Adjust virtual cores requirement from {} to {}.", cores, > updatedCores); > } > int updatedMemory = Math.min(resource.getMemory(), > maxCapability.getMemory()); > if (resource.getMemory() != updatedMemory) { > resource.setMemory(updatedMemory); > LOG.info("Adjust memory requirement from {} to {} MB.", > resource.getMemory(), updatedMemory); > } > return resource; > } > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)