I would change "mapred.tasktracker.reduce.tasks.maximum" to one. With your setting
On May 2, 2011 8:48 AM, "baran cakici" <barancak...@gmail.com> wrote: > without job; > > CPU Usage = 0% > Memory = 585 MB (2GB Ram) > > Baran > 2011/5/2 baran cakici <barancak...@gmail.com> > >> CPU Usage = 95-100% >> Memory = 650-850 MB (2GB Ram) >> >> Baran >> >> >> 2011/5/2 James Seigel <ja...@tynt.com> >> >>> If you have windows and cygwin you probably don't have a lot if memory >>> left at 2 gig. >>> >>> Pull up system monitor on the data nodes and check for free memory >>> when you have you jobs running. I bet it is quite low. >>> >>> I am not a windows guy so I can't take you much farther. >>> >>> James >>> >>> Sent from my mobile. Please excuse the typos. >>> >>> On 2011-05-02, at 8:32 AM, baran cakici <barancak...@gmail.com> wrote: >>> >>> > yes, I am running under cygwin on my datanodes too. OS of Datanodes are >>> > Windows as well. >>> > >>> > What can I do exactly for a better Performance. I changed >>> > mapred.child.java.opts to default value.How can I solve this "swapping" >>> > problem? >>> > >>> > PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS. >>> > >>> > thanks, both of you >>> > >>> > Regards, >>> > >>> > Baran >>> > 2011/5/2 Richard Nadeau <strout...@gmail.com> >>> > >>> >> Are you running under cygwin on your data nodes as well? That is >>> certain to >>> >> cause performance problems. As James suggested, swapping to disk is >>> going >>> >> to >>> >> be a killer, running on Windows with Celeron processors only compounds >>> the >>> >> problem. The Celeron processor is also sub-optimal for CPU intensive >>> tasks >>> >> >>> >> Rick >>> >> >>> >> On Apr 28, 2011 9:22 AM, "baran cakici" <barancak...@gmail.com> wrote: >>> >>> Hi Everyone, >>> >>> >>> >>> I have a Cluster with one Master(JobTracker and NameNode - Intel >>> Core2Duo >>> >> 2 >>> >>> GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram). >>> My >>> >>> Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line >>> by >>> >>> line. Now, I try to accelerate my System(Benchmark), but I'm not sure, >>> if >>> >> my >>> >>> Configuration is correctly. Can you please just look, if it is ok? >>> >>> >>> >>> -mapred-site.xml >>> >>> >>> >>> <property> >>> >>> <name>mapred.job.tracker</name> >>> >>> <value>apple:9001</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.child.java.opts</name> >>> >>> <value>-Xmx512m -server</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.job.tracker.handler.count</name> >>> >>> <value>2</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.local.dir</name> >>> >>> >>> >> >>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.map.tasks</name> >>> >>> <value>1</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.reduce.tasks</name> >>> >>> <value>4</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.submit.replication</name> >>> >>> <value>2</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.system.dir</name> >>> >>> >>> >> >>> >> >>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.tasktracker.indexcache.mb</name> >>> >>> <value>10</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.tasktracker.map.tasks.maximum</name> >>> >>> <value>1</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.tasktracker.reduce.tasks.maximum</name> >>> >>> <value>4</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.temp.dir</name> >>> >>> >>> >> >>> <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>webinterface.private.actions</name> >>> >>> <value>true</value> >>> >>> </property> >>> >>> >>> >>> <property> >>> >>> <name>mapred.reduce.slowstart.completed.maps</name> >>> >>> <value>0.01</value> >>> >>> </property> >>> >>> >>> >>> -hdfs-site.xml >>> >>> >>> >>> <property> >>> >>> <name>dfs.block.size</name> >>> >>> <value>268435456</value> >>> >>> </property> >>> >>> PS: I extended dfs.block.size, because I won 50% better performance >>> with >>> >>> this change. >>> >>> >>> >>> I am waiting for your comments... >>> >>> >>> >>> Regards, >>> >>> >>> >>> Baran >>> >> >>> >> >>
