Hi Everyone,
I'm using Hadoop-2.2.0 with fair scheduler in my YARN cluster, but something is
wrong with the fair scheduler.
Here is my fair-scheduler.xml looks like:
<allocations>
<queue name="longrun">
<maxResources>15360 mb, 5 vcores</maxResources>
<weight>0.5</weight>
<minMaps>2</minMaps>
<maxMaps>5</maxMaps>
<minReduces>1</minReduces>
</queue>
</allocations>
I create a "longrun" queue to ensure that huge MR application can only use 5
resources. My YARN setup for each resource memory is 3072 MB:
<property>
<name>mapreduce.map.memory.mb</name>
<value>3072</value>
</property>
<property>
<name>mapreduce.reduce.memory.mb</name>
<value>3072</value>
</property>
When the huge application started, it works just fine and scheduler restrict it
to only run 5 maps in parallel.
But after running for some time, the application run 10 maps in parallel.
The scheduler page show that the "longrun" queue used 66%, exceed the fair
share 30%.
Can anyone tell me why the application can get more than it deserved?
Is the problem with my configuration? Or there is a bug?
Best regards,
Henry Hung
________________________________
The privileged confidential information contained in this email is intended for
use only by the addressees as indicated by the original sender of this email.
If you are not the addressee indicated in this email or are not responsible for
delivery of the email to such a person, please kindly reply to the sender
indicating this fact and delete all copies of it from your computer and network
server immediately. Your cooperation is highly appreciated. It is advised that
any unauthorized use of confidential information of Winbond is strictly
prohibited; and any information in this email irrelevant to the official
business of Winbond shall be deemed as neither given nor endorsed by Winbond.
