Hi, see answers inline below HTH, Jothi
> I would like to know: > > 1. How can block size impact the performance of a mapred job. >From the M/R side, the fileSystem block size of the input files is treated as an upper bound for input splits. . Since each input split translates into one map, this can affect the actual number of maps for the job > 2. Does the performance improve if I setup NameNode and JobTracker on > different machine. At present, > I am running Namenode and JobTracker on the same machine as Master > interconnected to 2 slave machines running Datanode and TaskTracker Intuitively, it should help. Namenode is really memory intensive and the job tracker could also be heavily loaded depending on the number of concurrent jobs running and the number of maps and reducers of these jobs (for scheduling). > 3. What should be the replication factor for a 3 node cluster I think having a higher replication factor might not increase performance for a 3 node cluster, it might degrade if at all because of the extra replication. If replication is only for performance and not for availability/fault tolerance, you could try setting the replication factor to a smaller number (1?). > 4. How does io.sort.mb impact the performance of the cluster Look here http://hadoop.apache.org/core/docs/r0.19.0/mapred_tutorial.html > > Thanks, > Sandeep > > > Brian Bockelman wrote: >> >> Hey Sandeep, >> >> I'd do a couple of things: >> 1) Run your test. Do something which will be similar to your actual >> workflow. >> 2) Save the resulting Ganglia plots. This will give you a hint as to >> where things are bottlenecking (memory, CPU, wait I/O). >> 3) Watch iostat and find out the I/O rates during the test. Compare >> this to the I/O rates of a known I/O benchmark (i.e., Bonnie+). >> 4) Finally, watch the logfiles closely. If you start to overload >> things, you'll usually get a pretty good indication from Hadoop where >> things go wrong. Once something does go wrong, *then* look through >> the parameters to see what can be done. >> >> There's about a hundred things which can go wrong between the kernel, >> the OS, Java, and the application code. It's difficult to make an >> educated guess beforehand without some hint from the data. >> >> Brian >> >> On Dec 31, 2008, at 1:30 AM, Sandeep Dhawan wrote: >> >>> >>> Hi Brian, >>> >>> That's what my issue is i.e. "How do I ascertain the bottleneck" or >>> in other >>> words if the results obtained after doing the performance testing >>> are not >>> upto the mark then How do I find the bottleneck. >>> >>> How can we confidently say that OS and hardware are the culprits. I >>> understand that by using the latest OS and hardware can improve the >>> performance irrespective of the application but my real worry is >>> "What Next >>> ". How can I further increase the performance. What should I look >>> for which >>> can suggest or point the areas which can be potential problems or >>> "hotspot". >>> >>> Thanks for your comments. >>> >>> ~Sandeep~ >>> >>> >>> Brian Bockelman wrote: >>>> >>>> Hey Sandeep, >>>> >>>> I would warn against premature optimization: first, run your test, >>>> then see how far from your target you are. >>>> >>>> Of course, I'd wager you'd find that the hardware you are using is >>>> woefully underpowered and that your OS is 5 years old. >>>> >>>> Brian >>>> >>>> On Dec 30, 2008, at 5:57 AM, Sandeep Dhawan wrote: >>>> >>>>> >>>>> Hi, >>>>> >>>>> I am trying to create a hadoop cluster which can handle 2000 write >>>>> requests >>>>> per second. >>>>> In each write request I would writing a line of size 1KB in a file. >>>>> >>>>> I would be using machine having following configuration: >>>>> Platfom: Red Hat Linux 9.0 >>>>> CPU : 2.07 GHz >>>>> RAM : 1GB >>>>> >>>>> Can anyone help in giving me some pointers/guideline as to how to go >>>>> about >>>>> setting up such a cluster. >>>>> What are the configuration parameters in hadoop with which we can >>>>> tweak to >>>>> ehance the performance of the hadoop cluster. >>>>> >>>>> Thanks, >>>>> Sandeep >>>>> -- >>>>> View this message in context: >>>>> http://www.nabble.com/Performance-testing-tp21216266p21216266.html >>>>> Sent from the Hadoop core-user mailing list archive at Nabble.com. >>>> >>>> >>>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Performance-testing-tp21216266p21228264.html >>> Sent from the Hadoop core-user mailing list archive at Nabble.com. >> >> >>