Hi Julien, Did you try to change yarn.nodemanager.resource.memory-mb to 13 GB for example (the other 3 for OS) ?
Thanks On 1 August 2014 05:41, Julien Naour <julna...@gmail.com> wrote: > Hello, > > I'm currently using HDP 2.0 so it's Hadoop 2.2.0. > My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each. > > Recently we passed from 2 users to 8. We need now a more appropriate > Scheduler. > We begin with Capacity Scheduler. There was some issues with the different > queues particularly when using some spark shell that used some resources > for a long time. > So we decide to try Fair Scheduler which seems to be a good solution. > The problem is that FairScheduler doesn't allow all available resources. > It's capped at 73% of the available memory for one jobs 63% for 2 jobs and > 45% for 3 jobs. The problem could come from shells that take resources for > a long time. > > We tried some configuration like > yarn.scheduler.fair.user-as-default-queue=false > or play with the minimum ressources allocated minResources in > fair-scheduler.xml but it doesn't seems to resolve the issue. > > Any advices or good practices to held a good Fair Scheduler? > > Regards, > > Julien >
