That looks fine but there is something in your setup that is off. For example, HBASE_HOME is being expanded in the below properly on all hosts? You are ssh forwarding this environment variable? (if you are using this variable name). St.Ack
On Tue, Jun 9, 2009 at 10:47 AM, llpind <sonny_h...@hotmail.com> wrote: > > Yeah that was my first thought. rowcounter and PerformanceEvaluation work. > here is my HADOOP_CLASSPATH: > =================================== > export > > HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:/sharedfs/hbase-config/conf > =================================== > slave boxes reference a shared conf: > export > > HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.19.3.jar:$HBASE_HOME/hbase-0.19.3-test.jar:$HBASE_HOME/conf > =================================== > where HBASE_HOME=/home/hadoop/hbase193. this is the same path on all > boxes. > > I run my .class path from the default package like so: > > hadoop19/bin/hadoop ColumnCountMapReduce inputTableName > > > > > stack-3 wrote: > > > > Thats a CLASSPATH issue. For sure the hbase jars are out on all nodes of > > your cluster? For sure the paths are correct? > > St.Ack > > > > On Tue, Jun 9, 2009 at 10:28 AM, llpind <sonny_h...@hotmail.com> wrote: > > > >> > >> Yep. Thanks, I didn't think to look there :). I do have another issue > >> when > >> running my own map/reduce. I've got the jar files in HADOOP_CLASSPATH > as > >> i > >> mentioned above, but when i run the job below is what i get. The > >> ClassDefNotFoundError happens on TableSplit multiple times. > >> ================================================ > >> > >> hadoop19/bin/hadoop ColumnCountMapReduce inputTableName > >> 09/06/09 10:26:08 WARN mapred.JobClient: No job jar file set. User > >> classes > >> may not be found. See JobConf(Class) or JobConf#setJar(String). > >> 09/06/09 10:26:11 INFO mapred.JobClient: Running job: > >> job_200906090918_0005 > >> 09/06/09 10:26:12 INFO mapred.JobClient: map 0% reduce 0% > >> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id : > >> attempt_200906090918_0005_m_000000_0, Status : FAILED > >> java.io.IOException: Split class > >> org.apache.hadoop.hbase.mapred.TableSplit > >> not found > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314) > >> at org.apache.hadoop.mapred.Child.main(Child.java:158) > >> Caused by: java.lang.ClassNotFoundException: > >> org.apache.hadoop.hbase.mapred.TableSplit > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > >> at java.lang.Class.forName0(Native Method) > >> at java.lang.Class.forName(Class.java:247) > >> at > >> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311) > >> ... 1 more > >> > >> 09/06/09 10:26:21 INFO mapred.JobClient: Task Id : > >> attempt_200906090918_0005_m_000004_0, Status : FAILED > >> java.io.IOException: Split class > >> org.apache.hadoop.hbase.mapred.TableSplit > >> not found > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:314) > >> at org.apache.hadoop.mapred.Child.main(Child.java:158) > >> Caused by: java.lang.ClassNotFoundException: > >> org.apache.hadoop.hbase.mapred.TableSplit > >> at java.net.URLClassLoader$1.run(URLClassLoader.java:200) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at java.net.URLClassLoader.findClass(URLClassLoader.java:188) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:307) > >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) > >> at java.lang.ClassLoader.loadClass(ClassLoader.java:252) > >> at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) > >> at java.lang.Class.forName0(Native Method) > >> at java.lang.Class.forName(Class.java:247) > >> at > >> > org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:673) > >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:311) > >> ... 1 more > >> > >> > >> stack-3 wrote: > >> > > >> > It looks pretty obvious what your issue is. > >> > > >> > My guess is that your child process needs its heap size bumped up. > >> > > >> > Look for the child parameters configuration in your hadoop-default.xml > >> (or > >> > your mapred-default.xml if 0.20.x hadoop). > >> > > >> > St.Ack > >> > > >> > On Tue, Jun 9, 2009 at 10:05 AM, llpind <sonny_h...@hotmail.com> > wrote: > >> > > >> >> > >> >> Hmm...jobtracker UI has the following: > >> >> > >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > >> >> contact > >> >> region server 192.168.0.195:60020 for region > TestTable,,1244499094604, > >> >> row > >> >> '0000010477', but failed after 10 attempts. > >> >> Exceptions: > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951) > >> >> at > >> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397) > >> >> at > >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341) > >> >> at > >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182) > >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > >> >> at org.apache.hadoop.mapred.Child.main(Child.java:158) > >> >> > >> >> ================================================= > >> >> > >> >> org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > >> >> contact > >> >> region server 192.168.0.195:60020 for region > TestTable,,1244499094604, > >> >> row > >> >> '0002107629', but failed after 10 attempts. > >> >> Exceptions: > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> java.lang.OutOfMemoryError: Java heap space > >> >> > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:859) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:951) > >> >> at > >> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:1397) > >> >> at > >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1341) > >> >> at > >> org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1321) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$SequentialWriteTest.testRow(PerformanceEvaluation.java:515) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$Test.test(PerformanceEvaluation.java:386) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.runOneClient(PerformanceEvaluation.java:583) > >> >> at > >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation$EvaluationMapTask.map(PerformanceEvaluation.java:182) > >> >> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > >> >> at org.apache.hadoop.mapred.Child.main(Child.java:158) > >> >> > >> >> > >> >> > >> >> > >> >> stack-3 wrote: > >> >> > > >> >> > Look at why the job failed in the jobtracker UI -- usually on port > >> >> 50030. > >> >> > Looks like your job launched fine. You have the conf dir on the > >> >> > HADOOP_CLASSPATH so the MR job can find hbase? > >> >> > St.Ack > >> >> > > >> >> > On Tue, Jun 9, 2009 at 9:25 AM, llpind <sonny_h...@hotmail.com> > >> wrote: > >> >> > > >> >> >> > >> >> >> Hey all, > >> >> >> > >> >> >> getting started with map/reduce jobs. Figured I'd try the > >> >> >> PerformanceEvaluation test program first. > >> >> >> > >> >> >> After adding HBase jar files to HADOOP_CLASSPATH in hadoop-env.sh, > >> I > >> >> >> issue > >> >> >> the following command: > >> >> >> > >> >> >> hadoop19/bin/hadoop org.apache.hadoop.hbase.PerformanceEvaluation > >> >> >> sequentialWrite 4 > >> >> >> > >> >> >> and get the following output: > >> >> >> > >> >> >> 09/06/09 09:20:54 WARN mapred.JobClient: Use GenericOptionsParser > >> for > >> >> >> parsing the arguments. Applications should implement Tool for the > >> >> same. > >> >> >> 09/06/09 09:20:55 INFO mapred.FileInputFormat: Total input paths > to > >> >> >> process > >> >> >> : 1 > >> >> >> 09/06/09 09:20:56 INFO mapred.JobClient: Running job: > >> >> >> job_200906090918_0001 > >> >> >> 09/06/09 09:20:57 INFO mapred.JobClient: map 0% reduce 0% > >> >> >> java.io.IOException: Job failed! > >> >> >> at > >> >> org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) > >> >> >> at > >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.doMapReduce(PerformanceEvaluation.java:293) > >> >> >> at > >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.runNIsMoreThanOne(PerformanceEvaluation.java:221) > >> >> >> at > >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.runTest(PerformanceEvaluation.java:639) > >> >> >> at > >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.doCommandLine(PerformanceEvaluation.java:748) > >> >> >> at > >> >> >> > >> >> >> > >> >> > >> > org.apache.hadoop.hbase.PerformanceEvaluation.main(PerformanceEvaluation.java:768) > >> >> >> > >> >> >> > >> >> >> What am I doing wrong...? > >> >> >> > >> >> >> > >> >> >> -- > >> >> >> View this message in context: > >> >> >> > >> >> > >> > http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23946437.html > >> >> >> Sent from the HBase User mailing list archive at Nabble.com. > >> >> >> > >> >> >> > >> >> > > >> >> > > >> >> > >> >> -- > >> >> View this message in context: > >> >> > >> > http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947225.html > >> >> Sent from the HBase User mailing list archive at Nabble.com. > >> >> > >> >> > >> > > >> > > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947629.html > >> Sent from the HBase User mailing list archive at Nabble.com. > >> > >> > > > > > > -- > View this message in context: > http://www.nabble.com/PerformanceEvaluation-test-tp23946437p23947980.html > Sent from the HBase User mailing list archive at Nabble.com. > >
