Unable to apply patch to contrib/mumak
Hi guys ! I was trying to use capacity scheduler with mumak simulator. hadoop version : haddop-0.21 I am able to run mumak simulator as per slides of Hong Tong http://www.slideshare.net/hadoopusergroup/mumak As per https://issues.apache.org/jira/browse/MAPREDUCE-1253 i need to apply the patch. i have tried like this : 1 > copied patch file to HADOOP_HOME/mapreduce/src/ contrib/mumak 2 > $cd HADOOP_HOME/mapreduce/src/contrib/mumak 3 > $patch -p0 < patch_file_name but i get ERROR 3 out of 3 HUNK failed Is it the right way to apply patches ? If not what is the general way ? Thanks, Arun
Re: Running example application with capacity scheduler ?
Hi all ! Thanks Thomas ! it's working in terminal. I saw the queues in web UI of JT. when i try to run normally again (default) i get this error : i tried formatting namenode and making safemode off and restart but didn't work. hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ bin/hadoop jar hadoop*examples*.jar wordcount /user/hduser/wcinput /user/hduser/wcoutput6 java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local exception: java.io.IOException: Connection reset by peer The log of JobTracker shows : 2011-09-15 12:46:13,346 INFO org.apache.hadoop.mapred.JobTracker: JobTracker up at: 54311 2011-09-15 12:46:13,347 INFO org.apache.hadoop.mapred.JobTracker: JobTracker webserver: 50030 2011-09-15 12:46:13,634 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory 2011-09-15 12:46:13,646 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: hdfs://localhost:54310/app/hadoop/tmp/mapred/system org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /app/hadoop/tmp/mapred/system. Name node is in safe mode. Thanks, Arun On Wed, Sep 14, 2011 at 7:46 PM, Thomas Graves wrote: > I believe it defaults to submit a job to the default queue if you don't > specify it. You don't have the default queue defined in your list of > mapred.queue.names. So add -Dmapred.job.queue.name=myqueue1 (or another > queue you have defined) to the wordcount command like: > > bin/hadoop jar > > hadoop*examples*.jar wordcount -Dmapred.job.queue.name=myqueue1 > /user/hduser/wcinput /user/hduser/wcoutput5 > > Tom > > > On 9/14/11 5:57 AM, "arun k" wrote: > > > Hi ! > > > > I have set up single-node cluster using > > > http://www.google.co.in/url?sa=t&source=web&cd=1&ved=0CB0QFjAA&url=http%3A%2F% > > 2Fwww.michael-noll.com > %2Ftutorials%2Frunning-hadoop-on-ubuntu-linux-single-nod > > > e-cluster%2F&rct=j&q=michael%20noll%20single%20node&ei=b4ZwTvrCLsOrrAei-N32Bg& > > > usg=AFQjCNGhuvv0tNdvPj4u23bbj-qXJDlixg&sig2=7ij8Dy7aQZUkBwhTnS1rLw&cad=rja > > and could run wordcount example application. > > I was trying to run this application using capacity scheduler. > > As per > http://hadoop.apache.org/common/docs/current/capacity_scheduler.htmli > > have done : > > 1.Copied the hadoop-capacity-scheduler-*.jar from * > > contrib/capacity-scheduler* directory to HADOOP_HOME/lib > > 2.Set mapred.jobtracker.taskScheduler > > 3.Set *mapred.queue.names to myqueue1,myqueue2. > > 4.Set *mapred.capacity-scheduler.queue..capacity to 30 and 70 > > for two queues. > > > > When i run i get the error : > > hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ bin/hadoop jar > > hadoop*examples*.jar wordcount /user/hduser/wcinput > /user/hduser/wcoutput5 > > 11/09/14 16:00:56 INFO input.FileInputFormat: Total input paths to > process : > > 4 > > org.apache.hadoop.ipc.RemoteException: java.io.IOException: Queue > "default" > > does not exist > > at > org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2998) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j > > ava:25) > > at java.lang.reflect.Method.invoke(Method.java:597) > > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) > > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:396) > > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) > > > > at org.apache.hadoop.ipc.Client.call(Client.java:740) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) > > at org.apache.hadoop.mapred.$Proxy0.submitJob(Unknown Source) > > at > > org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800) > > at org.apache.hadoop.mapreduce.Job.submit(Job.java:432) > > at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447) > > at org.apache.hadoop.examples.WordCount.main(WordCount.java:67) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > > at > > > sun.reflect.Delegatin
Remote exception and IO exception when running a job with capacity scheduler
Hi ! I have set up hadoop-0.20.203 in local mode as per http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ I have run the wordcount example Job and have all daemons running. Then to run this job with capacity scheduler i have done the following : 1. Added properties value mapred.jobtracker.taskScheduler org.apache.hadoop.mapred.CapacityTaskScheduler mapred.queue.names myqueue1,myqueue2 mapred.capacity-scheduler.queue.myqueue1.capacity 25 mapred.capacity-scheduler.queue.myqueue1.capacity 75 Already present property mapred.job.trackerlocalhost:54311 2.${HADOOP_HOME}$ bin/stop-all.sh 3.${HADOOP_HOME}$ bin/start-all.sh 4.$jps shows all daemons 5.${HADOOP_HOME}$ bin/hadoop jar hadoop*examples*.jar wordcount - Dmapred.job.queue.name=myqueue1 /user/hduser/wcinput /user/hduser/wcoutput I get the error: java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local exception: java.io.IOException: Connection reset by peer at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) ... When i give $jps 32463 NameNode 32763 SecondaryNameNode 32611 DataNode 931 Jps The jobracker log gives info ~ 2011-09-16 00:21:42,012 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory 2011-09-16 00:21:42,014 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: hdfs://localhost:54310/app203/hadoop203/tmp/mapred/system org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /app203/hadoop203/tmp/mapred/system. Name node is in safe mode. The ratio of reported blocks 1. has reached the threshold 0.9990. Safe mode will be turned off automatically in 6 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1851) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1831) 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: java.io.IOException: Queue 'myqueue1' doesn't have configured capacity! at org.apache.hadoop.mapred.CapacityTaskScheduler.parseQueues(CapacityTaskScheduler.java:905) at org.apache.hadoop.mapred.CapacityTaskScheduler.start(CapacityTaskScheduler.java:822) at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2563) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4957) 2011-09-16 00:21:52,322 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG: ~~~ Even if i submit the job to "myqueue2" i see the same error of "myqueue1" 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: java.io.IOException: Queue 'myqueue1' doesn't have configured capacity! Thanks, Arun
Re: Running example application with capacity scheduler ?
Hi all ! Harsh ! Namenode appears to be out of safe mode : In http://nn-host:50070 i see in time T1>Safe mode is ON. The ratio of reported blocks 0. has not reached the threshold 0.9990. Safe mode will be turned off automatically. 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / 966.69 MB (1%) T2>Safe mode is ON. The ratio of reported blocks 1. has reached the threshold 0.9990. Safe mode will be turned off automatically in 17 seconds. 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / 966.69 MB (1%) T3>9 files and directories, 3 blocks = 12 total. Heap Size is 15.06 MB / 966.69 MB (1%) Added properties : mapred.jobtracker.taskScheduler org.apache.hadoopertiep.mapred.CTS mapred.queue.names myqueue1,myqueue2 mapred.capacity-scheduler.queue.myqueue1.capacity 25 mapred.capacity-scheduler.queue.myqueue1.capacity 75 ${HADOOP_HOME}$ bin/hadoop jar hadoop*examples*.jar wordcount - Dmapred.job.queue.name <http://dmapred.job.queue.name/>= myqueue1 /user/hduser/wcinput /user/hduser/wcoutput I get the error: java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local exception: java.io.IOException: Connection reset by peer at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) at org.apache.hadoop.ipc.Client.call(Client.java:1033) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) ... When i give $jps 32463 NameNode 32763 SecondaryNameNode 32611 DataNode 931 Jps The jobracker log gives info ~ 2011-09-16 00:21:42,012 INFO org.apache.hadoop.mapred.JobTracker: Cleaning up the system directory 2011-09-16 00:21:42,014 INFO org.apache.hadoop.mapred.JobTracker: problem cleaning system directory: hdfs://localhost:54310/app203/hadoop203/tmp/mapred/system org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete /app203/hadoop203/tmp/mapred/system. Name node is in safe mode. The ratio of reported blocks 1. has reached the threshold 0.9990. Safe mode will be turned off automatically in 6 seconds. at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1851) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1831) 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: java.io.IOException: Queue 'myqueue1' doesn't have configured capacity! at org.apache.hadoop.mapred.CapacityTaskScheduler.parseQueues(CapacityTaskScheduler.java:905) at org.apache.hadoop.mapred.CapacityTaskScheduler.start(CapacityTaskScheduler.java:822) at org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2563) at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4957) 2011-09-16 00:21:52,322 INFO org.apache.hadoop.mapred.JobTracker: SHUTDOWN_MSG: ~~~ Even if i submit the job to "myqueue2" i see the same error of "myqueue1" 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: java.io.IOException: Queue 'myqueue1' doesn't have configured capacity! Thanks, Arun On Thu, Sep 15, 2011 at 5:23 PM, Harsh J wrote: > Hello Arun, > > To me it looks like your HDFS isn't setup properly, in this case. Can > you ensure all DNs are properly up? Your NN appears to have gotten > stuck somehow into a safemode. Check out your http://nn-host:50070 > page for more details on why. > > Your JT won't come up until the NN is properly up and out of safemode > (for which it needs the DNs). And once it comes up, I think you should > be good to go, keeping in mind the changes Thomas mentioned earlier. > > On Thu, Sep 15, 2011 at 3:58 PM, arun k wrote: > > Hi all ! > > > > Thanks Thomas ! it's working in terminal. > > I saw the queues in web UI of JT. > > when i try to run normally again (default) i get this error : > > i tried formatting namenode and making safemode off and restart but > didn't > > work. > > > > hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ bin/hadoop jar > > hadoop*examples*.jar wordcount /user/hduser/wcinput > /user/hduser/wcoutput6 > > java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local > > exception: java.io.IOException: Connection reset by peer > > > > The log of JobTracker shows : > > 2011-09-15 12:46:13,346 INFO org.apache.hadoop.mapred.JobTracker: > JobTracker > > up at: 54311 > > 2011-09-15 12:46:13,347 INFO org.apache.hadoop.mapred.JobTracker: > JobTracker > > webserver: 50030 > > 2011-09-15 12:46:13,634 INFO org.apache.hadoop.mapred.JobTracker
[no subject]
Hi ! Thanks Harsh ! The problem was that i have set up queue info in mapred-site.xml instead of capacity-scheduler.xml . Arun On Fri, Sep 16, 2011 at 10:52 AM, Harsh J wrote: > Arun, > > Please do not cross-post to multiple lists. Lets continue this on > mapreduce-user@ alone. > > Your problem isn't the job submission here, but your Capacity > Scheduler configuration. For every queue you configure, you need to > add in capacities: Please see the queue properties documentation at > > http://hadoop.apache.org/common/docs/current/capacity_scheduler.html#Queue+properties > for the vital configs required in additional to mapred.queue.names. > Once done, you should have a fully functional JobTracker! > > On Fri, Sep 16, 2011 at 10:17 AM, arun k wrote: > > Hi all ! > > > > Harsh ! Namenode appears to be out of safe mode : > > In http://nn-host:50070 i see in time > > > > T1>Safe mode is ON. The ratio of reported blocks 0. has not reached > the > > threshold 0.9990. Safe mode will be turned off automatically. > > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / > 966.69 > > MB (1%) > > > > T2>Safe mode is ON. The ratio of reported blocks 1. has reached the > > threshold 0.9990. Safe mode will be turned off automatically in 17 > seconds. > > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / > 966.69 > > MB (1%) > > > > T3>9 files and directories, 3 blocks = 12 total. Heap Size is 15.06 MB / > > 966.69 MB (1%) > > > > Added properties : > > > > mapred.jobtracker.taskScheduler org.apache.hadoopertiep.mapred.CTS > > > > mapred.queue.names myqueue1,myqueue2 > > mapred.capacity-scheduler.queue.myqueue1.capacity 25 > > mapred.capacity-scheduler.queue.myqueue1.capacity 75 > > ${HADOOP_HOME}$ bin/hadoop jar hadoop*examples*.jar wordcount > > -Dmapred.job.queue.name= > > myqueue1 /user/hduser/wcinput /user/hduser/wcoutput > > > > I get the error: > > java.io.IOException: Call to localhost/127.0.0.1:54311 failed on local > > exception: java.io.IOException: Connection reset by peer > > at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) > > at org.apache.hadoop.ipc.Client.call(Client.java:1033) > > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) > > ... > > > > When i give > > $jps > > 32463 NameNode > > 32763 SecondaryNameNode > > 32611 DataNode > > 931 Jps > > > > > > The jobracker log gives info > > ~ > > 2011-09-16 00:21:42,012 INFO org.apache.hadoop.mapred.JobTracker: > Cleaning > > up the system directory > > 2011-09-16 00:21:42,014 INFO org.apache.hadoop.mapred.JobTracker: problem > > cleaning system directory: > > hdfs://localhost:54310/app203/hadoop203/tmp/mapred/system > > org.apache.hadoop.ipc.RemoteException: > > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete > > /app203/hadoop203/tmp/mapred/system. Name node is in safe mode. > > The ratio of reported blocks 1. has reached the threshold 0.9990. > Safe > > mode will be turned off automatically in 6 seconds. > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1851) > > at > > > org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1831) > > 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: > > java.io.IOException: Queue 'myqueue1' doesn't have configured capacity! > > at > > > org.apache.hadoop.mapred.CapacityTaskScheduler.parseQueues(CapacityTaskScheduler.java:905) > > at > > > org.apache.hadoop.mapred.CapacityTaskScheduler.start(CapacityTaskScheduler.java:822) > > at > > org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2563) > > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4957) > > > > 2011-09-16 00:21:52,322 INFO org.apache.hadoop.mapred.JobTracker: > > SHUTDOWN_MSG: > > ~~~ > > > > Even if i submit the job to "myqueue2" i see the same error of "myqueue1" > > 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: > > java.io.IOException: Queue 'myqueue1' doesn't have configured capacity! > > > > Thanks, > > Arun > > > > > > > > On Thu, Sep 15, 2011 a
Re: Running Jobs with capacity scheduler on hadoop in eclipse
Hi ! I have setup hadoop0.20.2 on eclipse Helios and able to run the Example wordcount using ExampleDriver class as mentioned by Faraz in http://lucene.472066.n3.nabble.com/HELP-configuring-hadoop-on-ECLIPSE-td1086829.html#a2241534 Two questions ? 1. I am unable to see the jobtracker & others in browser at http addr mentioned in mapred- default.xml. i have not edited any site.xml files. I have tried to edit site.xml files as per Michael noll site but that didn't help. 2.Capacity Scheduler :I see the capacity -*- jar in lib folder. I have modified mapred-site.xml and capacity-scheduler.xml as required. How do i run some application jobs by submitting a job to a queue in this case ? I have tried to run: Program & Args as : wordcount -Dmapred.job.queue.name=myqueue1 input_file_loc output_file_loc But i get error : Exception in thread "main" java.lang.Error: Unresolved compilation problems: ProgramDriver cannot be resolved to a type ProgramDriver cannot be resolved to a type DistributedPentomino cannot be resolved to a type . Thanks, Arun On Fri, Sep 16, 2011 at 12:46 PM, Harsh J wrote: > Arun, > > Good to know. Happy Hadoopin'! > > On Fri, Sep 16, 2011 at 12:34 PM, arun k wrote: > > Hi ! > > Thanks Harsh ! > > The problem was that i have set up queue info in mapred-site.xml instead > of > > capacity-scheduler.xml . > > Arun > > > > On Fri, Sep 16, 2011 at 10:52 AM, Harsh J wrote: > >> > >> Arun, > >> > >> Please do not cross-post to multiple lists. Lets continue this on > >> mapreduce-user@ alone. > >> > >> Your problem isn't the job submission here, but your Capacity > >> Scheduler configuration. For every queue you configure, you need to > >> add in capacities: Please see the queue properties documentation at > >> > >> > http://hadoop.apache.org/common/docs/current/capacity_scheduler.html#Queue+properties > >> for the vital configs required in additional to mapred.queue.names. > >> Once done, you should have a fully functional JobTracker! > >> > >> On Fri, Sep 16, 2011 at 10:17 AM, arun k wrote: > >> > Hi all ! > >> > > >> > Harsh ! Namenode appears to be out of safe mode : > >> > In http://nn-host:50070 i see in time > >> > > >> > T1>Safe mode is ON. The ratio of reported blocks 0. has not > reached > >> > the > >> > threshold 0.9990. Safe mode will be turned off automatically. > >> > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / > >> > 966.69 > >> > MB (1%) > >> > > >> > T2>Safe mode is ON. The ratio of reported blocks 1. has reached > the > >> > threshold 0.9990. Safe mode will be turned off automatically in 17 > >> > seconds. > >> > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / > >> > 966.69 > >> > MB (1%) > >> > > >> > T3>9 files and directories, 3 blocks = 12 total. Heap Size is 15.06 MB > / > >> > 966.69 MB (1%) > >> > > >> > Added properties : > >> > > >> > mapred.jobtracker.taskScheduler org.apache.hadoopertiep.mapred.CTS > >> > > >> > mapred.queue.names myqueue1,myqueue2 > >> > mapred.capacity-scheduler.queue.myqueue1.capacity 25 > >> > mapred.capacity-scheduler.queue.myqueue1.capacity 75 > >> > ${HADOOP_HOME}$ bin/hadoop jar hadoop*examples*.jar wordcount > >> > -Dmapred.job.queue.name= > >> > myqueue1 /user/hduser/wcinput /user/hduser/wcoutput > >> > > >> > I get the error: > >> > java.io.IOException: Call to localhost/127.0.0.1:54311 failed on > local > >> > exception: java.io.IOException: Connection reset by peer > >> > at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) > >> > at org.apache.hadoop.ipc.Client.call(Client.java:1033) > >> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) > >> > ... > >> > > >> > When i give > >> > $jps > >> > 32463 NameNode > >> > 32763 SecondaryNameNode > >> > 32611 DataNode > >> > 931 Jps > >> > > >> > > >> > The jobracker log gives info > >> > ~ > >> > 2011-09-16 00:21:42,012 INFO org.apache.
Re: Running Jobs with capacity scheduler on hadoop in eclipse
Hi ! I am using hadoop on eclipse. I am able to run jobs. My question was how can i run some application with capacity scheduler ? Also I am unable to see Jobtracker in Web GUI at default Jobtracker address. I have not edited any *-site.xmls. Thanks, Arun On Fri, Sep 16, 2011 at 3:56 PM, Swathi V wrote: > It might be in safemode. turn off the safemode by using > *bin/hadoop dfsadmin safemode -leave* > to see the task tracker. > > > On Fri, Sep 16, 2011 at 2:09 PM, arun k wrote: > >> Hi ! >> >> I have setup hadoop0.20.2 on eclipse Helios and able to run the Example >> wordcount using ExampleDriver class as mentioned by Faraz in >> http://lucene.472066.n3.nabble.com/HELP-configuring-hadoop-on-ECLIPSE-td1086829.html#a2241534 >> >> Two questions ? >> 1. I am unable to see the jobtracker & others in browser at http addr >> mentioned in mapred- default.xml. i have not edited any site.xml files. >> I have tried to edit site.xml files as per Michael noll site but that >> didn't help. >> >> 2.Capacity Scheduler :I see the capacity -*- jar in lib folder. I have >> modified mapred-site.xml and capacity-scheduler.xml as required. How do i >> run some application jobs by submitting a job to a queue in this case ? >> I have tried to run: >> Program & Args as : wordcount -Dmapred.job.queue.name=myqueue1 >> input_file_loc output_file_loc >> But i get error : >> Exception in thread "main" java.lang.Error: Unresolved compilation >> problems: >> ProgramDriver cannot be resolved to a type >> ProgramDriver cannot be resolved to a type >> DistributedPentomino cannot be resolved to a type >> ..... >> >> Thanks, >> Arun >> >> >> On Fri, Sep 16, 2011 at 12:46 PM, Harsh J wrote: >> >>> Arun, >>> >>> Good to know. Happy Hadoopin'! >>> >>> On Fri, Sep 16, 2011 at 12:34 PM, arun k wrote: >>> > Hi ! >>> > Thanks Harsh ! >>> > The problem was that i have set up queue info in mapred-site.xml >>> instead of >>> > capacity-scheduler.xml . >>> > Arun >>> > >>> > On Fri, Sep 16, 2011 at 10:52 AM, Harsh J wrote: >>> >> >>> >> Arun, >>> >> >>> >> Please do not cross-post to multiple lists. Lets continue this on >>> >> mapreduce-user@ alone. >>> >> >>> >> Your problem isn't the job submission here, but your Capacity >>> >> Scheduler configuration. For every queue you configure, you need to >>> >> add in capacities: Please see the queue properties documentation at >>> >> >>> >> >>> http://hadoop.apache.org/common/docs/current/capacity_scheduler.html#Queue+properties >>> >> for the vital configs required in additional to mapred.queue.names. >>> >> Once done, you should have a fully functional JobTracker! >>> >> >>> >> On Fri, Sep 16, 2011 at 10:17 AM, arun k wrote: >>> >> > Hi all ! >>> >> > >>> >> > Harsh ! Namenode appears to be out of safe mode : >>> >> > In http://nn-host:50070 i see in time >>> >> > >>> >> > T1>Safe mode is ON. The ratio of reported blocks 0. has not >>> reached >>> >> > the >>> >> > threshold 0.9990. Safe mode will be turned off automatically. >>> >> > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / >>> >> > 966.69 >>> >> > MB (1%) >>> >> > >>> >> > T2>Safe mode is ON. The ratio of reported blocks 1. has reached >>> the >>> >> > threshold 0.9990. Safe mode will be turned off automatically in 17 >>> >> > seconds. >>> >> > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / >>> >> > 966.69 >>> >> > MB (1%) >>> >> > >>> >> > T3>9 files and directories, 3 blocks = 12 total. Heap Size is 15.06 >>> MB / >>> >> > 966.69 MB (1%) >>> >> > >>> >> > Added properties : >>> >> > >>> >> > mapred.jobtracker.taskScheduler org.apache.hadoopertiep.mapred.CTS >>> >> > >>> >> > mapred.queue.names myqueue1,myqueue2 >>> >> > mapred.capacity-scheduler.queue.my
Building modified capacity scheduler and seeing console ouputs.
Hi ! I have set up hadoop as per http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ I am able to run jobs with capacity scheduler. I am interested in extending the capacity scheduler. So first i wanted to see whether i can build it after making changes. i have tried this : >modified mapre-site.xml and capcity-sche.xml accordingly. > Put some println statements in CapacityTaskScheduler.java >In src/contrib/capacity_scheduler$ ant BUILD was successful. >Copied the hadoop-capacity-*-jar in build/contrib to lib/ >ran some job with capacity scheduler by submitting to a queue But i don't see any console output and i couldn't find print statements in any logs too. How do i go about it ? Thanks, Arun
Re: RE: Making Mumak work with capacity scheduler
Hi ! I have changed he permissions for hadoop extract and /jobstory and /history/done dir recursively: $chmod -R 777 branch-0.22 $chmod -R logs $chmod -R jobracker but still i get the same problem. The permissions are like this <http://pastebin.com/sw3UPM8t> The log is here <http://pastebin.com/CztUPywB>. I am able to run as sudo. Arun On Thu, Sep 22, 2011 at 7:19 PM, Uma Maheswara Rao G 72686 < mahesw...@huawei.com> wrote: > Yes Devaraj, > From the logs, looks it failed to create /jobtracker/jobsInfo > > > > code snippet: > > if (!fs.exists(path)) { >if (!fs.mkdirs(path, new > FsPermission(JOB_STATUS_STORE_DIR_PERMISSION))) { > throw new IOException( > "CompletedJobStatusStore mkdirs failed to create " > + path.toString()); >} > > @ Arun, Can you check, you have correct permission as Devaraj said? > > > 2011-09-22 15:53:57.598::INFO: Started > SelectChannelConnector@0.0.0.0:50030 > 11/09/22 15:53:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with > processName=JobTracker, sessionId= > 11/09/22 15:53:57 WARN conf.Configuration: mapred.task.cache.levels is > deprecated. Instead, use mapreduce.jobtracker.taskcache.levels > 11/09/22 15:53:57 WARN mapred.SimulatorJobTracker: Error starting tracker: > java.io.IOException: CompletedJobStatusStore mkdirs failed to create > /jobtracker/jobsInfo >at > org.apache.hadoop.mapred.CompletedJobStatusStore.(CompletedJobStatusStore.java:83) >at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:4684) >at > org.apache.hadoop.mapred.SimulatorJobTracker.(SimulatorJobTracker.java:81) >at > org.apache.hadoop.mapred.SimulatorJobTracker.startTracker(SimulatorJobTracker.java:100) >at > org.apache.hadoop.mapred.SimulatorEngine.init(SimulatorEngine.java:210) >at > org.apache.hadoop.mapred.SimulatorEngine.init(SimulatorEngine.java:184) >at > org.apache.hadoop.mapred.SimulatorEngine.run(SimulatorEngine.java:292) >at > org.apache.hadoop.mapred.SimulatorEngine.run(SimulatorEngine.java:323) > > I cc'ed to Mapreduce user mailing list as well. > > Regards, > Uma > > - Original Message - > From: Devaraj K > Date: Thursday, September 22, 2011 6:01 pm > Subject: RE: Making Mumak work with capacity scheduler > To: common-u...@hadoop.apache.org > > > Hi Arun, > > > >I have gone through the logs. Mumak simulator is trying to > > start the job > > tracker and job tracking is failing to start because it is not able to > > create "/jobtracker/jobsinfo" directory. > > > > I think the directory doesn't have enough permissions. Please check > > thepermissions or any other reason why it is failing to create the > > dir. > > > > > > Devaraj K > > > > > > -Original Message- > > From: arun k [mailto:arunk...@gmail.com] > > Sent: Thursday, September 22, 2011 3:57 PM > > To: common-u...@hadoop.apache.org > > Subject: Re: Making Mumak work with capacity scheduler > > > > Hi Uma ! > > > > u got me right ! > > >Actually without any patch when i modified appropriate mapred- > > site.xml and > > capacity-scheduler.xml and copied capaciy jar accordingly. > > I am able to see see queues in Jobracker GUI but both the queues > > show same > > set of job's execution. > > I ran with trace and topology files from test/data : > > $bin/mumak.sh trace_file topology_file > > Is it because i am not submitting jobs to a particular queue ? > > If so how can i do it ? > > > > >Got hadoop-0.22 from > > http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.22/ > > builded all three components but when i give > > arun@arun-Presario-C500-RU914PA-ACJ:~/hadoop22/branch- > > 0.22/mapreduce/src/contrib/mumak$ > > bin/mumak.sh src/test/data/19-jobs.trace.json.gz > > src/test/data/19-jobs.topology.json.gz > > it gets stuck at some point. Log is here > > <http://pastebin.com/9SNUHLFy> > > Thanks, > > Arun > > > > > > > > > > > > On Wed, Sep 21, 2011 at 2:03 PM, Uma Maheswara Rao G 72686 < > > mahesw...@huawei.com> wrote: > > > > > > > > Hello Arun, > > > If you want to apply MAPREDUCE-1253 on 21 version, > > > applying patch directly using commands may not work because of > > codebase> changes. > > > > > > So, you take the patch and apply the lines in your code base > > manually. I > > > am not sure any otherway for this. > > > > > > D
Fwd: RE: Making Mumak work with capacity scheduler
Hi guys ! I have run mumak as sudo. It works fine. i am trying to run jobtrace in test/data with capacity scheduler. I have done : 1> Build contrib/capacity-scheduler 2>Copied hadoop-*-capacity-jar from build/contrib/capacity_scheduler to lib/ 3>added mapred.jobtracker.taskScheduler and mapred.queue.names in mapred-site.xml 4>In conf/capacity-scheduler set the propoery value for 2 queues mapred.capacity-scheduler.queue.default.capacity 20 mapred.capacity-scheduler.queue.myqueue2.capacity 80 When i run mumak.sh i see in console 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: No capacity specified for queue default 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: Created a jobQueue default and added it as a child to 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: No capacity specified for queue myqueue2 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: Created a jobQueue myqueue2 and added it as a child to 11/09/23 11:51:19 INFO mapred.AbstractQueue: Total capacity to be distributed among the others are 100.0 11/09/23 11:51:19 INFO mapred.AbstractQueue: Capacity share for un configured queue default is 50.0 11/09/23 11:51:19 INFO mapred.AbstractQueue: Capacity share for un configured queue myqueue2 is 50.0 11/09/23 11:51:19 INFO mapred.CapacityTaskScheduler: Capacity scheduler started successfully 2 Q's : 1> In web GUI of Jobtracker i see both he queues but "CAPACITIES ARE REFLECTED" 2> All the jobs by defaul are submitted to "default" queue. How can i submit jobs to various queues in mumak ? Regards, Arun On Fri, Sep 23, 2011 at 10:12 AM, arun k wrote: > Hi ! > > I have changed he permissions for hadoop extract and /jobstory and > /history/done dir recursively: > $chmod -R 777 branch-0.22 > $chmod -R logs > $chmod -R jobracker > but still i get the same problem. > The permissions are like this <http://pastebin.com/sw3UPM8t> > The log is here <http://pastebin.com/CztUPywB>. > I am able to run as sudo. > > Arun > > On Thu, Sep 22, 2011 at 7:19 PM, Uma Maheswara Rao G 72686 < > mahesw...@huawei.com> wrote: > >> Yes Devaraj, >> From the logs, looks it failed to create /jobtracker/jobsInfo >> >> >> >> code snippet: >> >> if (!fs.exists(path)) { >>if (!fs.mkdirs(path, new >> FsPermission(JOB_STATUS_STORE_DIR_PERMISSION))) { >> throw new IOException( >> "CompletedJobStatusStore mkdirs failed to create " >> + path.toString()); >>} >> >> @ Arun, Can you check, you have correct permission as Devaraj said? >> >> >> 2011-09-22 15:53:57.598::INFO: Started >> SelectChannelConnector@0.0.0.0:50030 >> 11/09/22 15:53:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with >> processName=JobTracker, sessionId= >> 11/09/22 15:53:57 WARN conf.Configuration: mapred.task.cache.levels is >> deprecated. Instead, use mapreduce.jobtracker.taskcache.levels >> 11/09/22 15:53:57 WARN mapred.SimulatorJobTracker: Error starting tracker: >> java.io.IOException: CompletedJobStatusStore mkdirs failed to create >> /jobtracker/jobsInfo >>at >> org.apache.hadoop.mapred.CompletedJobStatusStore.(CompletedJobStatusStore.java:83) >>at org.apache.hadoop.mapred.JobTracker.(JobTracker.java:4684) >>at >> org.apache.hadoop.mapred.SimulatorJobTracker.(SimulatorJobTracker.java:81) >>at >> org.apache.hadoop.mapred.SimulatorJobTracker.startTracker(SimulatorJobTracker.java:100) >>at >> org.apache.hadoop.mapred.SimulatorEngine.init(SimulatorEngine.java:210) >>at >> org.apache.hadoop.mapred.SimulatorEngine.init(SimulatorEngine.java:184) >>at >> org.apache.hadoop.mapred.SimulatorEngine.run(SimulatorEngine.java:292) >>at >> org.apache.hadoop.mapred.SimulatorEngine.run(SimulatorEngine.java:323) >> >> I cc'ed to Mapreduce user mailing list as well. >> >> Regards, >> Uma >> >> - Original Message - >> From: Devaraj K >> Date: Thursday, September 22, 2011 6:01 pm >> Subject: RE: Making Mumak work with capacity scheduler >> To: common-u...@hadoop.apache.org >> >> > Hi Arun, >> > >> >I have gone through the logs. Mumak simulator is trying to >> > start the job >> > tracker and job tracking is failing to start because it is not able to >> > create "/jobtracker/jobsinfo" directory. >> > >> > I think the directory doesn't have enough permissions. Please check >> > thepermissions or any other reason why it is failing to create the >> > dir. >> > >> &g
Re: RE: Making Mumak work with capacity scheduler
Sorry , 1Q: In web GUI of Jobtracker i see both he queues but "CAPACITIES ARE NOT REFLECTED" 2Q:All the jobs by defaul are submitted to "default" queue. How can i submit jobs to various queues in mumak ? regards, Arun On Fri, Sep 23, 2011 at 11:57 AM, arun k wrote: > Hi guys ! > > I have run mumak as sudo. It works fine. > i am trying to run jobtrace in test/data with capacity scheduler. > I have done : > 1> Build contrib/capacity-scheduler > 2>Copied hadoop-*-capacity-jar from build/contrib/capacity_scheduler to > lib/ > 3>added mapred.jobtracker.taskScheduler and mapred.queue.names in > mapred-site.xml > 4>In conf/capacity-scheduler > set the propoery value for 2 queues > mapred.capacity-scheduler.queue.default.capacity 20 > mapred.capacity-scheduler.queue.myqueue2.capacity 80 > > When i run mumak.sh > i see in console > 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: No capacity specified > for queue default > 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: Created a jobQueue > default and added it as a child to > 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: No capacity specified > for queue myqueue2 > 11/09/23 11:51:19 INFO mapred.QueueHierarchyBuilder: Created a jobQueue > myqueue2 and added it as a child to > 11/09/23 11:51:19 INFO mapred.AbstractQueue: Total capacity to be > distributed among the others are 100.0 > 11/09/23 11:51:19 INFO mapred.AbstractQueue: Capacity share for un > configured queue default is 50.0 > 11/09/23 11:51:19 INFO mapred.AbstractQueue: Capacity share for un > configured queue myqueue2 is 50.0 > 11/09/23 11:51:19 INFO mapred.CapacityTaskScheduler: Capacity scheduler > started successfully > > 2 Q's : > > 1> In web GUI of Jobtracker i see both he queues but "CAPACITIES ARE > REFLECTED" > 2> All the jobs by defaul are submitted to "default" queue. How can i > submit jobs to various queues in mumak ? > > Regards, > Arun > > > On Fri, Sep 23, 2011 at 10:12 AM, arun k wrote: > >> Hi ! >> >> I have changed he permissions for hadoop extract and /jobstory and >> /history/done dir recursively: >> $chmod -R 777 branch-0.22 >> $chmod -R logs >> $chmod -R jobracker >> but still i get the same problem. >> The permissions are like this <http://pastebin.com/sw3UPM8t> >> The log is here <http://pastebin.com/CztUPywB>. >> I am able to run as sudo. >> >> Arun >> >> On Thu, Sep 22, 2011 at 7:19 PM, Uma Maheswara Rao G 72686 < >> mahesw...@huawei.com> wrote: >> >>> Yes Devaraj, >>> From the logs, looks it failed to create /jobtracker/jobsInfo >>> >>> >>> >>> code snippet: >>> >>> if (!fs.exists(path)) { >>>if (!fs.mkdirs(path, new >>> FsPermission(JOB_STATUS_STORE_DIR_PERMISSION))) { >>> throw new IOException( >>> "CompletedJobStatusStore mkdirs failed to create " >>> + path.toString()); >>>} >>> >>> @ Arun, Can you check, you have correct permission as Devaraj said? >>> >>> >>> 2011-09-22 15:53:57.598::INFO: Started >>> SelectChannelConnector@0.0.0.0:50030 >>> 11/09/22 15:53:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with >>> processName=JobTracker, sessionId= >>> 11/09/22 15:53:57 WARN conf.Configuration: mapred.task.cache.levels is >>> deprecated. Instead, use mapreduce.jobtracker.taskcache.levels >>> 11/09/22 15:53:57 WARN mapred.SimulatorJobTracker: Error starting >>> tracker: java.io.IOException: CompletedJobStatusStore mkdirs failed to >>> create /jobtracker/jobsInfo >>>at >>> org.apache.hadoop.mapred.CompletedJobStatusStore.(CompletedJobStatusStore.java:83) >>>at >>> org.apache.hadoop.mapred.JobTracker.(JobTracker.java:4684) >>>at >>> org.apache.hadoop.mapred.SimulatorJobTracker.(SimulatorJobTracker.java:81) >>>at >>> org.apache.hadoop.mapred.SimulatorJobTracker.startTracker(SimulatorJobTracker.java:100) >>>at >>> org.apache.hadoop.mapred.SimulatorEngine.init(SimulatorEngine.java:210) >>>at >>> org.apache.hadoop.mapred.SimulatorEngine.init(SimulatorEngine.java:184) >>>at >>> org.apache.hadoop.mapred.SimulatorEngine.run(SimulatorEngine.java:292) >>>at >>> org.apache.hadoop.mapred.SimulatorEngine.run(SimulatorEngine.java:323) >>> >>> I cc'ed to Mapreduce user mailing list as well. >>> &
Modifyiing Scheduler code in Mumak
Hi guys ! I have gone thru Mumak code. I ran mumak.sh with given Job and Topology trace files .In my understanding i see that when a job is fetched from JobStoryProducer an event is asscoiated with it and the listener / node where it is assigned is fixed when these events are created. I have not understood how *mumak is simulating scheduling exactly*.Can anyone help me in this regard ? I am interested in coding a simple scheduler in Mumak. 1> Which files do i need to modify if i want to change the way the events are allocated to a particular node/Taskracker ? 2> How do i compare throughput with existing schedulers ? 3> Do i need to have my own Job trace file ? Thanks, Arun
How is data of each job assigned to nodes in Mumak ?
Hi guys ! Q> How can i assign data of each job in mumak nodes and what else i need to do ? In general how can i use the pluggable block-placement for HDFS in Mumak ? Meaning in my context i am using 19-jobs-trace json file and modified topology json file consisting of say 4 nodes. Since the number of tasks(map & reduce) are fixed for these jobs i want to assign the input data( with their replication) of each job to some particular nodes, so that i can use this info in my scheduler . Does the code for "ObtainLocalMapTask" / "ObtainNonlocalMaptask" in Scheduler need to be changed if i have to use this data placement info ? Q>If i have to add some fields to jobs in job trace file , how do i add it to job trace and access in my scheduler code for scheduling in Mumak. Which classes do i need to modify for this? Thanks, Arun
How does splitting and choosing hosts for splits is done in Mumak ?
Hi ! I see that InputFormat, FileSplit and FileInputFormat files are involved in creating the splits of the input data for jobs. I am interested in knowing how this splitting and choosing hosts where splits are stored is managed in Mumak. I see that SimulatorJobInProgress is involved in this but the metainfo like split location has empty value. where does mumak assign split location's or does it do it at all ? Can anyone help me in this regard ? Thanks, Arun
Creating a custom trace file using Rumen
Hi guys ! I was trying to use Rumen to generate trace files. I have few queries : Q1> Is there any way to create a new trace file from job history logs with custom set of split locations ? Q2> Can we create new trace files from existing trace files with new values for the attributes like preferred locations ? Q3> How can i add new attributes / fields (which are not in job history logs) to the job or the tasks in the trace ? (or) Is there any way to generate trace with extra fields ? Can anyone help me in this regard ? Thanks, Arun
Capturing Map and Reduce I/O time
Hi guys ! I see that hadoop doesn't capture the Map task I/O time and Reduce task I/O time and captures only map runtime and reduce runtime. Am i right ? By I/O time for map task i meant time taken by the map task to read the input chunk allocated to it for processing and the time for it to write the O/P data to the local disk. By I/O time for Reduce task i meant time for reduce task to transfer map O/Ps to reduce task(shuffle phase) and writing reduce O/Ps to DFS. > What does that runtime of Map and reduce tasks mean ? Does it mean time taken to execute the Map method and reduce method respectively ? (or) Does it mean time taken from the start of the Map/Reduce task to the completion of the Map/Reduce task(i.e including time to read,sort ,compute map or reduce ,merge,etc.) ? > Which files do i need to look at and modify in Hadoop if i want to capture the map and reduce I/O time's ? > If i want to capture these values for few jobs of applications like wordcount,sort,etc. what is the best way to do ? Can anyone guide me in this regard ? Thanks, Arun
Availability of Job traces
Hi guys ! Apart from generating the job traces from RUMEN , can i get logs or job traces of varied sizes from some organizations. How can i make sure that the rumen generates only say 25 jobs,50 jobs or so ? Thanks, Arun
Re: Capturing Map/reduce task run times and bytes read
Harsh, Sorry for creating confusion. The question is if i have a single node setup and i give Sysout statements in maptask.java and reducetask.java. {HADOOP_HOME}$ant build {HADOOP_HOME}$start all daemons {HADOOP_HOME}$ run wordcount example Yes i am able to see o/p in *.out files of tasktrackers. Q>Does the map/reduce task run time displayed in web GUI is decent/accurate enough ? Q>If i want to do find the IO rate of a task, will the task run time divided by total number of FIle bytes and HDFS bytes read/written give it approximately ? Q>Does the FILE Bytes read for the reduce task include the map output record bytes read non-locally over network or the bytes read locally from the map output records after they are copied locally ? Thanks, Arun
Re: Capturing Map/reduce task run times and bytes read
Harsh, I wanted to conform about it b'coz in case if it doesn't i want to write code to capture it. Does it make sense to classify a map/reduce task as I/O bound or cpu bound based on its I/O rate ? Arun On Sat, Dec 3, 2011 at 2:43 PM, Harsh J wrote: > Arun, > > Inline again. > > On 03-Dec-2011, at 12:39 PM, arun k wrote: > > > Q>Does the map/reduce task run time displayed in web GUI is > decent/accurate enough ? > > > Don't see why not. We only display what's been genuinely collected. What > you get out of an API on the CLI is absolutely the same thing. Or perhaps I > do not understand your question completely here - what's led you to ask > this? > > Q>If i want to do find the IO rate of a task, will the task run time > divided by total number of FIle bytes and HDFS bytes read/written give it > approximately ? > > > Yes, that should give you a stop-watch measure. Task start -> Task end, > and the counters the task puts up for itself. > > Q>Does the FILE Bytes read for the reduce task include the map output > record bytes read non-locally over network or the bytes read locally from > the map output records after they are copied locally ? > > > FILE counters are from whatever is read off a local filesystem (file:///), > so would mean the latter. If you look again, you will notice another > counter named "Reduce shuffle bytes" that gives you the former count - > separately. >
Are the task run times displayed in Web GUI accurate ?
Harsh, I had a doubt regarding task runtimes displayed in Web GUI b'coz the Web GUI shows only task run times in seconds and not in milliseconds. Can i make it display nanotime or atleast milliseconds ? (or) Can i capture the time in some other way ? Arun On Sat, Dec 3, 2011 at 8:00 PM, arun k wrote: > Harsh, > > I wanted to conform about it b'coz in case if it doesn't i want to write > code to capture it. > > Does it make sense to classify a map/reduce task as I/O bound or cpu bound > based on its I/O rate ? > > Arun > > On Sat, Dec 3, 2011 at 2:43 PM, Harsh J wrote: > >> Arun, >> >> Inline again. >> >> On 03-Dec-2011, at 12:39 PM, arun k wrote: >> >> >> Q>Does the map/reduce task run time displayed in web GUI is >> decent/accurate enough ? >> >> >> Don't see why not. We only display what's been genuinely collected. What >> you get out of an API on the CLI is absolutely the same thing. Or perhaps I >> do not understand your question completely here - what's led you to ask >> this? >> >> Q>If i want to do find the IO rate of a task, will the task run time >> divided by total number of FIle bytes and HDFS bytes read/written give it >> approximately ? >> >> >> Yes, that should give you a stop-watch measure. Task start -> Task end, >> and the counters the task puts up for itself. >> >> Q>Does the FILE Bytes read for the reduce task include the map output >> record bytes read non-locally over network or the bytes read locally from >> the map output records after they are copied locally ? >> >> >> FILE counters are from whatever is read off a local filesystem (file:///), >> so would mean the latter. If you look again, you will notice another >> counter named "Reduce shuffle bytes" that gives you the former count - >> separately. >> > >
Which Hadoop version has Adaptive scheduler ?
Hi guys ! In which Hadoop Version can i find the adaptive scheduler of https://issues.apache.org/jira/browse/MAPREDUCE-1380 Can anyone tell me the difference between Dynamic scheduler and Adaptive scheduler ? Arun
Re: Which Hadoop version has Adaptive scheduler ?
Thanks Praveen ! Arun On Wed, Dec 7, 2011 at 10:57 PM, Praveen Sripati wrote: > The resolution of the JIRA says unresolved, so it's not yet in any of the > release. Best bet is to download the patch attached with the JIRA and see > the code changes if interested. > > Regards, > Praveen > > On Wed, Dec 7, 2011 at 8:06 PM, arun k wrote: > >> Hi guys ! >> >> In which Hadoop Version can i find the adaptive scheduler of >> https://issues.apache.org/jira/browse/MAPREDUCE-1380 >> Can anyone tell me the difference between Dynamic scheduler and Adaptive >> scheduler ? >> >> >> Arun >> >> >
Checksum error during trace generation using Rumen
Hi guys ! I was trying to generate job trace and topology trace. I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount program as hduser . I have mapreduce component set up in eclipse for user "arun". I set for a configuration : Class: org.apache.hadoop.tools.rumen.TraceBuilder Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json /usr/local/hadoop/logs/history When i run it i get the following error even when i have given : ${HADOOP_HOME} chmod -R 777 /logs 11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file: file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count. Ignoring exception: java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readFully(DataInputStream.java:152) at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:145) at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311) at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534) at org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.(PossiblyDecompressedInputStream.java:42) at org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42) at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142) Any help ? Arun
Re: Checksum error during trace generation using Rumen
Amar, I didn't modfiy Jobhistory file but i deleted hidden checksumfiels and it worked. I have a 3 node cluster set up. For generating a single job i am running say wordcount. Is there any way where i can generate good number of jobs(say 50,100,..) at a time ? How can i generate jobs trace for 50, 100, so on by keeping two nodes under one rack and the other in second rack in my cluster ? HOw do i control the placement of nodes in different rack and generate job and topology trace ? Arun On Thu, Dec 8, 2011 at 12:11 PM, Amar Kamat wrote: > Arun, > Did you modify the job history file manually? Looks like HDFS is not able > to match the job history file contents to its checksum. Try deleting the > checksum file. Note that the checksum file is a hidden file. > Amar > > > > On 12/8/11 12:03 PM, "arun k" wrote: > > Hi guys ! > > I was trying to generate job trace and topology trace. > I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount > program as hduser . > I have mapreduce component set up in eclipse for user "arun". > I set for a configuration : > Class: org.apache.hadoop.tools.rumen.TraceBuilder > Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json > /usr/local/hadoop/logs/history > > When i run it i get the following error even when i have given : > ${HADOOP_HOME} chmod -R 777 /logs > > 11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file: > file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count. > Ignoring exception: java.io.EOFException > at java.io.DataInputStream.readFully(DataInputStream.java:180) > at java.io.DataInputStream.readFully(DataInputStream.java:152) > at > org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:145) > at > org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311) > at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534) > at > org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.(PossiblyDecompressedInputStream.java:42) > at > org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42) > at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) > at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142) > > Any help ? > > Arun > > > >
Re: Checksum error during trace generation using Rumen
Amar, Job trace and topology trace are generated but the TraceBuilder program terminates after displaying 11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable It doesn't indicate whether it was a success and doesn't give any other info. Is it Ok ? I have used the generated trace with Mumak and it works fine. Arun On Thu, Dec 8, 2011 at 7:43 PM, arun k wrote: > Amar, > > I didn't modfiy Jobhistory file but i deleted hidden checksumfiels and it > worked. > I have a 3 node cluster set up. > For generating a single job i am running say wordcount. Is there any way > where i can generate good number of jobs(say 50,100,..) at a time ? > How can i generate jobs trace for 50, 100, so on by keeping two nodes > under one rack and the other in second rack in my cluster ? > HOw do i control the placement of nodes in different rack and generate job > and topology trace ? > > > Arun > > > On Thu, Dec 8, 2011 at 12:11 PM, Amar Kamat wrote: > >> Arun, >> Did you modify the job history file manually? Looks like HDFS is not able >> to match the job history file contents to its checksum. Try deleting the >> checksum file. Note that the checksum file is a hidden file. >> Amar >> >> >> >> On 12/8/11 12:03 PM, "arun k" wrote: >> >> Hi guys ! >> >> I was trying to generate job trace and topology trace. >> I have hadoop set up for hduser at /usr/local/hadoop and ran wordcount >> program as hduser . >> I have mapreduce component set up in eclipse for user "arun". >> I set for a configuration : >> Class: org.apache.hadoop.tools.rumen.TraceBuilder >> Args : /home/arun/Documents/jh.json /home/arun/Documents/top.json >> /usr/local/hadoop/logs/history >> >> When i run it i get the following error even when i have given : >> ${HADOOP_HOME} chmod -R 777 /logs >> >> 11/12/08 11:55:17 WARN util.NativeCodeLoader: Unable to load >> native-hadoop library for your platform... using builtin-java classes where >> applicable >> 11/12/08 11:55:18 WARN fs.FSInputChecker: Problem opening checksum file: >> file:/usr/local/hadoop/logs/history/job_201109191122_0002_1316411924369_hduser_word+count. >> Ignoring exception: java.io.EOFException >> at java.io.DataInputStream.readFully(DataInputStream.java:180) >> at java.io.DataInputStream.readFully(DataInputStream.java:152) >> at >> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:145) >> at >> org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:311) >> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:534) >> at >> org.apache.hadoop.tools.rumen.PossiblyDecompressedInputStream.(PossiblyDecompressedInputStream.java:42) >> at >> org.apache.hadoop.tools.rumen.DefaultInputDemuxer.bindTo(DefaultInputDemuxer.java:42) >> at org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:225) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) >> at org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142) >> >> Any help ? >> >> Arun >> >> >> >> >
Benchmarks for IO and CPU Intensive workload
Hi guys ! I want to see the behavior of a single node of Hadoop cluster when IO intensive / CPU intensive workload and mix of both is submitted to the single node alone. These workloads must stress the nodes. I see that TestDFSIO benchmark is good for IO intensive workload. Which benchmarks do i need to use for this ? What amount of input data will be fair enough for seeing the behavior under these workloads for each type of boxes if i have boxes with :- B1: 4 GB RAM, Dual core ,150-250 GB DISK , B2 : 1GB RAM, 50-80 GB Disk. Arun
Accessing Job counters displayed in WEB GUI in Hadoop Code
Hai guys ! Can i access the Job counters displayed in WEB GUI in Hadoop code when the job finished their execution ? If so, how can i access the values like "average task run time" and counters "FILE/HDFS BYTES READ/WRITTEN" immediately after the job has completed in JobQueueTaskScheduler Code or in some other code file ? Thanks, Arun
Where do i see Sysout statements after building example ?
HI guys ! I have a single node set up as per http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-single-node-cluster/ 1>I have put some sysout statements in Jobtracker and wordcount (src/examples/org/..) code 2>ant build 3>Ran example jar with wordcount Where do i find the sysout statements ? i have seen in logs/ datanode,tasktracker,*.out files. Can anyone help me out ? Arun
Analysing Completed Job info programmatically apart from Jobtracker GUI
Hi Guys ! I want to analyse the completed Job counters like FILE/HDFS BYTES READ/WRITTEN along with other values like average map/reduce task run time. I see that Jobtracker GUI has this info but i want to programmatically retrieve these values instead of manually noting down these values and do some analysis. Can i do it in a simple/easier way ? I also see that Cloudera's HUE is good for this but is there anything equivalent in Hadoop. Can anyone guide me in this regard ? Arun
Re: Generating job and topology traces from history folder of multinode cluster using Rumen
Ravi, Thanks for the info. Arun On Fri, Dec 16, 2011 at 12:27 PM, Ravi Gummadi wrote: > Amar is working on this issue MAPREDUCE-3349. The patch is not comiited to > trunk yet. Feel free to try it out while it gets reviewed and committed. > > -Ravi > ____ > From: arun k [arunk...@gmail.com] > Sent: Friday, December 16, 2011 12:22 PM > To: mapreduce-user@hadoop.apache.org > Subject: Generating job and topology traces from history folder of > multinode cluster using Rumen > > Hai guys ! > > I have set up 5 node cluster with each of them in different racks. > > I have hadoop-0.20.2 set up on my Eclipse Helios. So, i ran Tracebuilder > using > Main Class: org.apache.hadoop.tools.rumen.TraceBuilder > I ran some jobs on cluster and used copy of > /usr/local/hadoop/logs/history folder of master node as input to > TraceBuilder. > Arguments: /home/arun/job.json /home/arun/topology.json > /home/ubuntu/Documents/testlog > > But i get > 11/12/16 12:02:37 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 11/12/16 12:02:38 WARN rumen.TraceBuilder: TraceBuilder got an error while > processing the [possibly virtual] file > master_1324011575958_job_201112161029_0001_hduser_word+count within Path > file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0001_hduser_word+count > java.lang.NullPointerException >at > org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492) >at > org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilder.java:149) >at > org.apache.hadoop.tools.rumen.TraceBuilder.processJobHistory(TraceBuilder.java:310) >at > org.apache.hadoop.tools.rumen.TraceBuilder.run(TraceBuilder.java:264) >at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69) >at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:83) >at > org.apache.hadoop.tools.rumen.TraceBuilder.main(TraceBuilder.java:142) > 11/12/16 12:02:39 WARN rumen.TraceBuilder: TraceBuilder got an error while > processing the [possibly virtual] file > master_1324011575958_job_201112161029_0004_hduser_word+count within Path > file:/home/ubuntu/Documents/testlog/master_1324011575958_job_201112161029_0004_hduser_word+count > java.lang.NullPointerException >at > org.apache.hadoop.tools.rumen.JobBuilder.processTaskAttemptFinishedEvent(JobBuilder.java:492) >at org.apache.hadoop.tools.rumen.JobBuilder.process(JobBuilde > . > > > It generates job trace json file but the topology trace json file doesn't > have 5 node's info and is like this : > > { > "name" : "", > "children" : [ ] > } > > > Which history folder do i need to use ? masters or slaves ? how do i > use it ? > > I have a single node cluster from which i got logs and gave input > TraceBuilder and it works. > > How do i run tracebuilder from terminal and do i need to use only from > version 21 ? > > Can anyone help me out ? > > > Thanks, > Arun >
Can reduce phase be CPU/IO bound ?
Hi guys ! If we neglect the shuffle part, can reduce phase be CPU/IO bound ? Can anyone suggest some benchmark or example where we can see this ? Arun