Hi Nico! The RM is configured with a minimum allocation. Take a look at "yarn.scheduler.minimum-allocation-mb" . You can also read through code here: https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java#L1295 . Please note that the code above is for the trunk branch, you should go look in the branch that you are running. Thanks to great work from some folks, this code has been evolving a lot recently.
HTH Ravi On Tue, Aug 30, 2016 at 12:09 PM, Nico Pappagianis < nico.pappagia...@salesforce.com> wrote: > How does YARN decide if remaining memory is enough AM allocation > > Hi all, > > Somewhat of a newb here so hopefully this question isn't too trivial. > > I want to know how YARN decides if the remaining memory after AM resource > allocation is "enough". Here's a breakdown of what's going on in my > particular case. > > We have a YARN queue that has ~1.7GB memory allocated to it. We have a > parent MR job that spawns multiple child jobs. The AM container memory for > the parent job is configured to 1GB. Once the AM container gets allocated > its 1GB there is a remaining ~0.7GB memory available. In my case that 0.7GB > is enough for the child jobs to succeed. > > My question is, what if that remaining amount wasn't 0.7GB but something > much less, like 0.1GB, or 0.0001GB? How does YARN handle those cases - will > it allocate *any* remaining resources to the child jobs? Will there be a > ton of thrashing if there is only a small amount of memory available? > > Thanks for any help, or a pointer in the right direction! >
