Hello, I'm currently using HDP 2.0 so it's Hadoop 2.2.0. My cluster consist in 4 nodes, 16 coeurs 16 GB RAM 4*3To each.
Recently we passed from 2 users to 8. We need now a more appropriate Scheduler. We begin with Capacity Scheduler. There was some issues with the different queues particularly when using some spark shell that used some resources for a long time. So we decide to try Fair Scheduler which seems to be a good solution. The problem is that FairScheduler doesn't allow all available resources. It's capped at 73% of the available memory for one jobs 63% for 2 jobs and 45% for 3 jobs. The problem could come from shells that take resources for a long time. We tried some configuration like yarn.scheduler.fair.user-as-default-queue=false or play with the minimum ressources allocated minResources in fair-scheduler.xml but it doesn't seems to resolve the issue. Any advices or good practices to held a good Fair Scheduler? Regards, Julien
