yes, I am running under cygwin on my datanodes too. OS of Datanodes are Windows as well.
What can I do exactly for a better Performance. I changed mapred.child.java.opts to default value.How can I solve this "swapping" problem? PS: I dont have a chance to get Slaves(Celeron 2GHz) with Liniux OS. thanks, both of you Regards, Baran 2011/5/2 Richard Nadeau <strout...@gmail.com> > Are you running under cygwin on your data nodes as well? That is certain to > cause performance problems. As James suggested, swapping to disk is going > to > be a killer, running on Windows with Celeron processors only compounds the > problem. The Celeron processor is also sub-optimal for CPU intensive tasks > > Rick > > On Apr 28, 2011 9:22 AM, "baran cakici" <barancak...@gmail.com> wrote: > > Hi Everyone, > > > > I have a Cluster with one Master(JobTracker and NameNode - Intel Core2Duo > 2 > > GB Ram) and four Slaves(Datanode and Tasktracker - Celeron 2 GB Ram). My > > Inputdata are between 2GB-10GB and I read Inputdata in MapReduce line by > > line. Now, I try to accelerate my System(Benchmark), but I'm not sure, if > my > > Configuration is correctly. Can you please just look, if it is ok? > > > > -mapred-site.xml > > > > <property> > > <name>mapred.job.tracker</name> > > <value>apple:9001</value> > > </property> > > > > <property> > > <name>mapred.child.java.opts</name> > > <value>-Xmx512m -server</value> > > </property> > > > > <property> > > <name>mapred.job.tracker.handler.count</name> > > <value>2</value> > > </property> > > > > <property> > > <name>mapred.local.dir</name> > > > <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/local</value> > > </property> > > > > <property> > > <name>mapred.map.tasks</name> > > <value>1</value> > > </property> > > > > <property> > > <name>mapred.reduce.tasks</name> > > <value>4</value> > > </property> > > > > <property> > > <name>mapred.submit.replication</name> > > <value>2</value> > > </property> > > > > <property> > > <name>mapred.system.dir</name> > > > > <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/system</value> > > </property> > > > > <property> > > <name>mapred.tasktracker.indexcache.mb</name> > > <value>10</value> > > </property> > > > > <property> > > <name>mapred.tasktracker.map.tasks.maximum</name> > > <value>1</value> > > </property> > > > > <property> > > <name>mapred.tasktracker.reduce.tasks.maximum</name> > > <value>4</value> > > </property> > > > > <property> > > <name>mapred.temp.dir</name> > > > <value>/cygwin/usr/local/hadoop-datastore/hadoop-Baran/mapred/temp</value> > > </property> > > > > <property> > > <name>webinterface.private.actions</name> > > <value>true</value> > > </property> > > > > <property> > > <name>mapred.reduce.slowstart.completed.maps</name> > > <value>0.01</value> > > </property> > > > > -hdfs-site.xml > > > > <property> > > <name>dfs.block.size</name> > > <value>268435456</value> > > </property> > > PS: I extended dfs.block.size, because I won 50% better performance with > > this change. > > > > I am waiting for your comments... > > > > Regards, > > > > Baran >
