Hi Harsh, Thanks for your reply. And I am sorry for my unclear description.
As I mentioned previous, I think I configured the fairsheduler correctly in hadoop-0.22.0. But when I commit lots of the jobs: many big jobs (map number and reduce number is bigger than the map/reduce slot) commit first. and many small jobs(just 1-2map/reduce per job) commit later. And I find in the jobtracker http page that it used the JobQueueTaskScheduler, and even the http://jobtracker:port/scheduler page is not found. When the big jobs is running, the small job can't start before big job' complete. So I guess hadoop-0.22 do not support fairscheduler or somewhere I configured wrong. 专注于Mysql,MSSQL,Oracle,Hadoop 2012/9/8 Harsh J <ha...@cloudera.com> > Hey Jameson, > > When calling something inefficient, perhaps also share some details on > how/why/what? How else would we know what you wish to see and what > you're seeing instead? :) > > On Thu, Sep 6, 2012 at 2:47 PM, Jameson Li <hovlj...@gmail.com> wrote: > > I want to test version hadoop-0.22. > > But when configurate the fairescheduler, I have some troublesome. The > > fairscheduler is not efficient. > > And I have configured this items in the mapred-site.xml, and also I hava > > copy the fairscheduler jar file to the $HADOOP_HOME/lib: > > > > <property> > > <name>mapreduce.jobtracker.taskScheduler</name> > > <value>org.apache.hadoop.mapred.FairScheduler</value> > > </property> > > > > <property> > > <name>mapred.fairscheduler.allocation.file</name> > > <value>conf/pools.xml</value> > > </property> > > <property> > > <name>mapred.fairscheduler.preemption</name> > > <value>true</value> > > </property> > > <property> > > <name>mapred.fairscheduler.assignmultiple</name> > > <value>true</value> > > </property> > > <property> > > <name>mapred.fairscheduler.poolnameproperty</name> > > <value>mapred.queue.name</value> > > <description>job.set("mapred.queue.name",pool); // pool is set > to > > either 'high' or 'low' </description> > > </property> > > <property> > > <name>mapred.queue.names</name> > > <value>default,aaa,bbb</value> > > </property> > > > > And the pools.xml in $HADOOP_HOME/conf 's content: > > > > <?xml version="1.0"?> > > <allocations> > > <pool name="putindb"> > > <minMaps>72</minMaps> > > <minReduces>16</minReduces> > > <maxRunningJobs>20</maxRunningJobs> > > <weight>3.0</weight> > > <minSharePreemptionTimeout>60</minSharePreemptionTimeout> > > </pool> > > > > <pool name="machinelearning"> > > <minMaps>9</minMaps> > > <minReduces>2</minReduces> > > <maxRunningJobs>10</maxRunningJobs> > > <weight>2.0</weight> > > <minSharePreemptionTimeout>60</minSharePreemptionTimeout> > > </pool> > > > > <pool name="default"> > > <minMaps>9</minMaps> > > <minReduces>2</minReduces> > > <maxRunningJobs>10</maxRunningJobs> > > <weight>1.0</weight> > > <minSharePreemptionTimeout>60</minSharePreemptionTimeout> > > </pool> > > > > <defaultMinSharePreemptionTimeout>60</defaultMinSharePreemptionTimeout> > > <fairSharePreemptionTimeout>60</fairSharePreemptionTimeout> > > </allocations> > > > > Can someone help me? > > > > > > 专注于Mysql,MSSQL,Oracle,Hadoop > > > > -- > Harsh J >
