On Sat, Feb 25, 2012 at 7:11 AM, Jie Li <ji...@cs.duke.edu> wrote: > Hello Mohit, > > I am looking at some hadoop tuning parameters like io.sort.mb, >> mapred.child.javaopts etc. > > - My question was where to look at for current setting >> > > The default settings as well as the documentations can be found in Hadoop > directory: > > src/mapred/mapred-default.xml > src/core/core-default.xml > src/hdfs/hdfs-default.xml > > >> - Are these settings configured cluster wide or per job? >> > > Some settings are configured cluster wide, e.g. the number of map/reduce > slots per node, while some settings are configured per job, e.g. > io.sort.mb. It depends on the functionality of that specific parameter. >
For cluster wide setting I am assuming it needs a cluster restart? > > >> - What's the best way to look at reasons of slow performance? >> > > Well, I want to introduce Starfish to you. Starfish is a self-tuning > system built on Hadoop to provide good performance automatically, without > any need for users to understand and manipulate the many tuning knobs in > Hadoop. > > With Starfish, you can analyze the performance of your Hadoop job at fine > grained level, e.g. the time for map processing, spilling, merging, > shuffling, sorting, and reduce processing. So you can understand which > part is the bottleneck of the performance. > > You can also ask "what-if" questions, e.g. "What if I double io.sort.mb > ?", and Starfish will predict the new behaviour of the job, so you can > better understand how these parameters work. In addition, you can simply > delegate Starfish to find the optimal configurations for you to achieve the > best performance. > > Welcome to join our Google Group to discuss more about Starfish and any > feedback will be appreciated. If you meet any problems, please don't > hesitate to let us know. The Group address is > http://groups.google.com/group/hadoop-starfish. > > Thanks, > Jie > --------------------- > Starfish Group, Duke University > Starfish Homepage: www.cs.duke.edu/starfish/ > Starfish Google Group: http://groups.google.com/group/hadoop-starfish >
