2GB for a task tracker? Here are some possible thoughts. Compress map output. Change mapred.reduce.slowstart.completed.maps
By the way I see no swapping. Anything interesting from the task tracker log? System log? Raj >________________________________ >From: john smith <js1987.sm...@gmail.com> >To: common-user@hadoop.apache.org >Sent: Wednesday, September 21, 2011 4:52 PM >Subject: Reducer hanging ( swapping? ) > >Hi Folks, > >I am running hive on a 10 node cluster. Since my hive queries have joins in >them, their reduce phases are a bit heavy. > >I have 2GB RAM on each TT . The problem is that my reducer hangs at 76% for >a large amount of time. I guess this is due to excessive swapping from disk >to memory. My vmstat shows (on one of the TTs) > >procs -----------memory---------- ---swap-- -----io---- -system-- >----cpu---- >r b swpd free buff cache si so bi bo in cs us sy id >wa >1 0 1860 34884 189948 1997644 0 0 2 1 0 1 0 0 100 >0 > >My related config parms are pasted below. (I turned off speculative >execution for both maps and reduces). Can anyone suggest me >some improvements so as to make my reduce a bit faster? >(I've allotted 900MB to task and reduced other params. Even then it is not >showing any improvments.) . Any suggestions? > >======================================== > ><property> ><name>mapred.min.split.size</name> ><value>65536</value> ></property> > > <property> > <name>mapred.reduce.copy.backoff</name> > <value>5</value> > </property> > > > <property> > <name>io.sort.factor</name> > <value>60</value> > </property> > > <property> > <name>mapred.reduce.parallel.copies</name> > <value>25</value> > </property> > > <property> > <name>io.sort.mb</name> > <value>70</value> > </property> > ><property> > <name>io.file.buffer.size</name> > <value>32768</value> > </property> > ><property> > <name>mapred.child.java.opts</name> > <value>-Xmx900m</value> > </property> > >=================================== > > >
