It might be in safemode. turn off the safemode by using *bin/hadoop dfsadmin safemode -leave* to see the task tracker.
On Fri, Sep 16, 2011 at 2:09 PM, arun k <arunk...@gmail.com> wrote: > Hi ! > > I have setup hadoop0.20.2 on eclipse Helios and able to run the Example > wordcount using ExampleDriver class as mentioned by Faraz in > http://lucene.472066.n3.nabble.com/HELP-configuring-hadoop-on-ECLIPSE-td1086829.html#a2241534 > > Two questions ? > 1. I am unable to see the jobtracker & others in browser at http addr > mentioned in mapred- default.xml. i have not edited any site.xml files. > I have tried to edit site.xml files as per Michael noll site but that > didn't help. > > 2.Capacity Scheduler :I see the capacity -*- jar in lib folder. I have > modified mapred-site.xml and capacity-scheduler.xml as required. How do i > run some application jobs by submitting a job to a queue in this case ? > I have tried to run: > Program & Args as : wordcount -Dmapred.job.queue.name=myqueue1 > input_file_loc output_file_loc > But i get error : > Exception in thread "main" java.lang.Error: Unresolved compilation > problems: > ProgramDriver cannot be resolved to a type > ProgramDriver cannot be resolved to a type > DistributedPentomino cannot be resolved to a type > ................. > > Thanks, > Arun > > > On Fri, Sep 16, 2011 at 12:46 PM, Harsh J <ha...@cloudera.com> wrote: > >> Arun, >> >> Good to know. Happy Hadoopin'! >> >> On Fri, Sep 16, 2011 at 12:34 PM, arun k <arunk...@gmail.com> wrote: >> > Hi ! >> > Thanks Harsh ! >> > The problem was that i have set up queue info in mapred-site.xml instead >> of >> > capacity-scheduler.xml . >> > Arun >> > >> > On Fri, Sep 16, 2011 at 10:52 AM, Harsh J <ha...@cloudera.com> wrote: >> >> >> >> Arun, >> >> >> >> Please do not cross-post to multiple lists. Lets continue this on >> >> mapreduce-user@ alone. >> >> >> >> Your problem isn't the job submission here, but your Capacity >> >> Scheduler configuration. For every queue you configure, you need to >> >> add in capacities: Please see the queue properties documentation at >> >> >> >> >> http://hadoop.apache.org/common/docs/current/capacity_scheduler.html#Queue+properties >> >> for the vital configs required in additional to mapred.queue.names. >> >> Once done, you should have a fully functional JobTracker! >> >> >> >> On Fri, Sep 16, 2011 at 10:17 AM, arun k <arunk...@gmail.com> wrote: >> >> > Hi all ! >> >> > >> >> > Harsh ! Namenode appears to be out of safe mode : >> >> > In http://nn-host:50070 i see in time >> >> > >> >> > T1>Safe mode is ON. The ratio of reported blocks 0.0000 has not >> reached >> >> > the >> >> > threshold 0.9990. Safe mode will be turned off automatically. >> >> > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / >> >> > 966.69 >> >> > MB (1%) >> >> > >> >> > T2>Safe mode is ON. The ratio of reported blocks 1.0000 has reached >> the >> >> > threshold 0.9990. Safe mode will be turned off automatically in 17 >> >> > seconds. >> >> > 7 files and directories, 1 blocks = 8 total. Heap Size is 15.06 MB / >> >> > 966.69 >> >> > MB (1%) >> >> > >> >> > T3>9 files and directories, 3 blocks = 12 total. Heap Size is 15.06 >> MB / >> >> > 966.69 MB (1%) >> >> > >> >> > Added properties : >> >> > >> >> > mapred.jobtracker.taskScheduler org.apache.hadoopertiep.mapred.CTS >> >> > >> >> > mapred.queue.names myqueue1,myqueue2 >> >> > mapred.capacity-scheduler.queue.myqueue1.capacity 25 >> >> > mapred.capacity-scheduler.queue.myqueue1.capacity 75 >> >> > ${HADOOP_HOME}$ bin/hadoop jar hadoop*examples*.jar wordcount >> >> > -Dmapred.job.queue.name= >> >> > myqueue1 /user/hduser/wcinput /user/hduser/wcoutput >> >> > >> >> > I get the error: >> >> > java.io.IOException: Call to localhost/127.0.0.1:54311 failed on >> local >> >> > exception: java.io.IOException: Connection reset by peer >> >> > at org.apache.hadoop.ipc.Client.wrapException(Client.java:1065) >> >> > at org.apache.hadoop.ipc.Client.call(Client.java:1033) >> >> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:224) >> >> > ................... >> >> > >> >> > When i give >> >> > $jps >> >> > 32463 NameNode >> >> > 32763 SecondaryNameNode >> >> > 32611 DataNode >> >> > 931 Jps >> >> > >> >> > >> >> > The jobracker log gives info >> >> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> > 2011-09-16 00:21:42,012 INFO org.apache.hadoop.mapred.JobTracker: >> >> > Cleaning >> >> > up the system directory >> >> > 2011-09-16 00:21:42,014 INFO org.apache.hadoop.mapred.JobTracker: >> >> > problem >> >> > cleaning system directory: >> >> > hdfs://localhost:54310/app203/hadoop203/tmp/mapred/system >> >> > org.apache.hadoop.ipc.RemoteException: >> >> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot >> delete >> >> > /app203/hadoop203/tmp/mapred/system. Name node is in safe mode. >> >> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990. >> >> > Safe >> >> > mode will be turned off automatically in 6 seconds. >> >> > at >> >> > >> >> > >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1851) >> >> > at >> >> > >> >> > >> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1831) >> >> > 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: >> >> > java.io.IOException: Queue 'myqueue1' doesn't have configured >> capacity! >> >> > at >> >> > >> >> > >> org.apache.hadoop.mapred.CapacityTaskScheduler.parseQueues(CapacityTaskScheduler.java:905) >> >> > at >> >> > >> >> > >> org.apache.hadoop.mapred.CapacityTaskScheduler.start(CapacityTaskScheduler.java:822) >> >> > at >> >> > >> org.apache.hadoop.mapred.JobTracker.offerService(JobTracker.java:2563) >> >> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:4957) >> >> > >> >> > 2011-09-16 00:21:52,322 INFO org.apache.hadoop.mapred.JobTracker: >> >> > SHUTDOWN_MSG: >> >> > >> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ >> >> > >> >> > Even if i submit the job to "myqueue2" i see the same error of >> >> > "myqueue1" >> >> > 2011-09-16 00:21:52,321 FATAL org.apache.hadoop.mapred.JobTracker: >> >> > java.io.IOException: Queue 'myqueue1' doesn't have configured >> capacity! >> >> > >> >> > Thanks, >> >> > Arun >> >> > >> >> > >> >> > >> >> > On Thu, Sep 15, 2011 at 5:23 PM, Harsh J <ha...@cloudera.com> wrote: >> >> >> >> >> >> Hello Arun, >> >> >> >> >> >> To me it looks like your HDFS isn't setup properly, in this case. >> Can >> >> >> you ensure all DNs are properly up? Your NN appears to have gotten >> >> >> stuck somehow into a safemode. Check out your http://nn-host:50070 >> >> >> page for more details on why. >> >> >> >> >> >> Your JT won't come up until the NN is properly up and out of >> safemode >> >> >> (for which it needs the DNs). And once it comes up, I think you >> should >> >> >> be good to go, keeping in mind the changes Thomas mentioned earlier. >> >> >> >> >> >> On Thu, Sep 15, 2011 at 3:58 PM, arun k <arunk...@gmail.com> wrote: >> >> >> > Hi all ! >> >> >> > >> >> >> > Thanks Thomas ! it's working in terminal. >> >> >> > I saw the queues in web UI of JT. >> >> >> > when i try to run normally again (default) i get this error : >> >> >> > i tried formatting namenode and making safemode off and restart >> but >> >> >> > didn't >> >> >> > work. >> >> >> > >> >> >> > hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ >> bin/hadoop >> >> >> > jar >> >> >> > hadoop*examples*.jar wordcount /user/hduser/wcinput >> >> >> > /user/hduser/wcoutput6 >> >> >> > java.io.IOException: Call to localhost/127.0.0.1:54311 failed on >> >> >> > local >> >> >> > exception: java.io.IOException: Connection reset by peer >> >> >> > >> >> >> > The log of JobTracker shows : >> >> >> > 2011-09-15 12:46:13,346 INFO org.apache.hadoop.mapred.JobTracker: >> >> >> > JobTracker >> >> >> > up at: 54311 >> >> >> > 2011-09-15 12:46:13,347 INFO org.apache.hadoop.mapred.JobTracker: >> >> >> > JobTracker >> >> >> > webserver: 50030 >> >> >> > 2011-09-15 12:46:13,634 INFO org.apache.hadoop.mapred.JobTracker: >> >> >> > Cleaning >> >> >> > up the system directory >> >> >> > 2011-09-15 12:46:13,646 INFO org.apache.hadoop.mapred.JobTracker: >> >> >> > problem >> >> >> > cleaning system directory: >> >> >> > hdfs://localhost:54310/app/hadoop/tmp/mapred/system >> >> >> > org.apache.hadoop.ipc.RemoteException: >> >> >> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot >> >> >> > delete >> >> >> > /app/hadoop/tmp/mapred/system. Name node is in safe mode. >> >> >> > >> >> >> > Thanks, >> >> >> > Arun >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > >> >> >> > On Wed, Sep 14, 2011 at 7:46 PM, Thomas Graves >> >> >> > <tgra...@yahoo-inc.com> >> >> >> > wrote: >> >> >> >> >> >> >> >> I believe it defaults to submit a job to the default queue if you >> >> >> >> don't >> >> >> >> specify it. You don't have the default queue defined in your >> list >> >> >> >> of >> >> >> >> mapred.queue.names. So add -Dmapred.job.queue.name=myqueue1 (or >> >> >> >> another >> >> >> >> queue you have defined) to the wordcount command like: >> >> >> >> >> >> >> >> bin/hadoop jar >> >> >> >> > hadoop*examples*.jar wordcount -Dmapred.job.queue.name >> =myqueue1 >> >> >> >> /user/hduser/wcinput /user/hduser/wcoutput5 >> >> >> >> >> >> >> >> Tom >> >> >> >> >> >> >> >> >> >> >> >> On 9/14/11 5:57 AM, "arun k" <arunk...@gmail.com> wrote: >> >> >> >> >> >> >> >> > Hi ! >> >> >> >> > >> >> >> >> > I have set up single-node cluster using >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> http://www.google.co.in/url?sa=t&source=web&cd=1&ved=0CB0QFjAA&url=http%3A%2F% >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > 2Fwww.michael-noll.com >> %2Ftutorials%2Frunning-hadoop-on-ubuntu-linux-single-nod >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> e-cluster%2F&rct=j&q=michael%20noll%20single%20node&ei=b4ZwTvrCLsOrrAei-N32Bg& >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> usg=AFQjCNGhuvv0tNdvPj4u23bbj-qXJDlixg&sig2=7ij8Dy7aQZUkBwhTnS1rLw&cad=rja >> >> >> >> > and could run wordcount example application. >> >> >> >> > I was trying to run this application using capacity scheduler. >> >> >> >> > As per >> >> >> >> > >> >> >> >> > >> http://hadoop.apache.org/common/docs/current/capacity_scheduler.htmli >> >> >> >> > have done : >> >> >> >> > 1.Copied the hadoop-capacity-scheduler-*.jar from * >> >> >> >> > contrib/capacity-scheduler* directory to HADOOP_HOME/lib >> >> >> >> > 2.Set mapred.jobtracker.taskScheduler >> >> >> >> > 3.Set *mapred.queue.names to myqueue1,myqueue2. >> >> >> >> > 4.Set *mapred.capacity-scheduler.queue.<queue-name>.capacity to >> 30 >> >> >> >> > and >> >> >> >> > 70 >> >> >> >> > for two queues. >> >> >> >> > >> >> >> >> > When i run i get the error : >> >> >> >> > hduser@arun-Presario-C500-RU914PA-ACJ:/usr/local/hadoop$ >> >> >> >> > bin/hadoop >> >> >> >> > jar >> >> >> >> > hadoop*examples*.jar wordcount /user/hduser/wcinput >> >> >> >> > /user/hduser/wcoutput5 >> >> >> >> > 11/09/14 16:00:56 INFO input.FileInputFormat: Total input paths >> to >> >> >> >> > process : >> >> >> >> > 4 >> >> >> >> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: >> Queue >> >> >> >> > "default" >> >> >> >> > does not exist >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:2998) >> >> >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j >> >> >> >> > ava:25) >> >> >> >> > at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> >> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) >> >> >> >> > at >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959) >> >> >> >> > at >> org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955) >> >> >> >> > at java.security.AccessController.doPrivileged(Native >> Method) >> >> >> >> > at javax.security.auth.Subject.doAs(Subject.java:396) >> >> >> >> > at >> org.apache.hadoop.ipc.Server$Handler.run(Server.java:953) >> >> >> >> > >> >> >> >> > at org.apache.hadoop.ipc.Client.call(Client.java:740) >> >> >> >> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220) >> >> >> >> > at org.apache.hadoop.mapred.$Proxy0.submitJob(Unknown >> Source) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:800) >> >> >> >> > at org.apache.hadoop.mapreduce.Job.submit(Job.java:432) >> >> >> >> > at >> >> >> >> > org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:447) >> >> >> >> > at >> >> >> >> > org.apache.hadoop.examples.WordCount.main(WordCount.java:67) >> >> >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j >> >> >> >> > ava:25) >> >> >> >> > at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.j >> >> >> >> > ava:68) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64) >> >> >> >> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >> Method) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) >> >> >> >> > at >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.j >> >> >> >> > ava:25) >> >> >> >> > at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> >> > at org.apache.hadoop.util.RunJar.main(RunJar.java:156) >> >> >> >> > >> >> >> >> > I didn't submit jobs to a particular queue as such. Do i need >> to >> >> >> >> > do >> >> >> >> > it ? >> >> >> >> > How >> >> >> >> > can i do it ? >> >> >> >> > Any help ? >> >> >> >> > >> >> >> >> > Thanks, >> >> >> >> > Arun >> >> >> >> > >> >> >> >> > >> >> >> >> > * >> >> >> >> > * >> >> >> >> >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Harsh J >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Harsh J >> > >> > >> >> >> >> -- >> Harsh J >> > > -- Regards, Swathi.V.
