I just completed a 3 day test[1] of Fluo on EC2 using the Twill reserved memory setting and YARN did not kill a Fluo process once[2]. I wish I had kept Grafana plots from previous runs where YARN was killing Fluo processes all of the time.
[1]: http://fluo.io/blog/2016/05/17/webindex-long-run-2/ [2]: http://fluo.io/blog/2016/05/17/webindex-long-run-2/#preventing-yarn-from-killing-workers On Fri, Apr 22, 2016 at 5:41 PM, Terence Yim <[email protected]> wrote: > Hi Keith, > > Yes. It can be controlled by the "twill.java.reserved.memory.mb" setting in > the YarnConfiguration passed to the "YarnTwillRunnerService". The actual > "-Xmx" set for the java process is the container resource memory minus the > value set in the "twill.java.reserved.memory.mb". By default the reserved > memory is 200MB. It also has a minimum memory ratio constant (0.7), meaning > the container size to non-heap memory ratio cannot be lower than 0.7. > > E.g. > If container size = 1GB, reserved = 200MB, then -Xmx = 800MB > If container size = 1GB, reserved = 500MB, then -Xmx = 700MB (because of > the min heap ratio). > > Terence > > On Fri, Apr 22, 2016 at 9:32 AM, Keith Turner <[email protected]> wrote: > > > I am running into a problem where YARN is killing my containers started > by > > Twill because its using too much memory. I would like to increase the > gap > > between the java -Xmx setting and the yarn memory limit. For example > make > > the -Xmx setting 75% of the YARN memory limit. Is there a way I can do > > this in Twill? > > > > Thanks, > > > > Keith > > >
