On Fri, Jun 3, 2016 at 7:17 PM, Terence Yim <[email protected]> wrote:
> That's seems like it is the reserved memory was just not enough. Glad that > it is working now. > Yeah it was too low. I am not exactly sure what is using the extra memory. I don't think any native code is in use that would allocate memory off heap. Maybe threads stacks are using the non-heap memory, I am running lots of threads. Or maybe the JVM just goes over,. > > Terence > > Sent from my iPhone > > > On Jun 3, 2016, at 2:13 PM, Keith Turner <[email protected]> wrote: > > > > I just completed a 3 day test[1] of Fluo on EC2 using the Twill reserved > > memory setting and YARN did not kill a Fluo process once[2]. I wish I > had > > kept Grafana plots from previous runs where YARN was killing Fluo > processes > > all of the time. > > > > [1]: http://fluo.io/blog/2016/05/17/webindex-long-run-2/ > > [2]: > > > http://fluo.io/blog/2016/05/17/webindex-long-run-2/#preventing-yarn-from-killing-workers > > > >> On Fri, Apr 22, 2016 at 5:41 PM, Terence Yim <[email protected]> wrote: > >> > >> Hi Keith, > >> > >> Yes. It can be controlled by the "twill.java.reserved.memory.mb" > setting in > >> the YarnConfiguration passed to the "YarnTwillRunnerService". The actual > >> "-Xmx" set for the java process is the container resource memory minus > the > >> value set in the "twill.java.reserved.memory.mb". By default the > reserved > >> memory is 200MB. It also has a minimum memory ratio constant (0.7), > meaning > >> the container size to non-heap memory ratio cannot be lower than 0.7. > >> > >> E.g. > >> If container size = 1GB, reserved = 200MB, then -Xmx = 800MB > >> If container size = 1GB, reserved = 500MB, then -Xmx = 700MB (because of > >> the min heap ratio). > >> > >> Terence > >> > >>> On Fri, Apr 22, 2016 at 9:32 AM, Keith Turner <[email protected]> > wrote: > >>> > >>> I am running into a problem where YARN is killing my containers started > >> by > >>> Twill because its using too much memory. I would like to increase the > >> gap > >>> between the java -Xmx setting and the yarn memory limit. For example > >> make > >>> the -Xmx setting 75% of the YARN memory limit. Is there a way I can do > >>> this in Twill? > >>> > >>> Thanks, > >>> > >>> Keith > >> >
