There is also a blog that I recently wrote, if it helps http://www.cloudera.com/blog/2011/01/how-to-include-third-party-libraries-in-your-map-reduce-job
On Tue, Jan 11, 2011 at 12:33 PM, Alex Kozlov <ale...@cloudera.com> wrote: > Have you implemented GenericOptionsParser? Do you see your jar in the * > mapred.cache.files* or *tmpjars* parameter in your job.xml file (can view > via a JT Web UI)? > > -- > Alex Kozlov > Solutions Architect > Cloudera, Inc > twitter: alexvk2009 > <http://www.cloudera.com/company/press-center/hadoop-world-nyc/> > > > On Tue, Jan 11, 2011 at 11:49 AM, C.V.Krishnakumar Iyer < > f2004...@gmail.com> wrote: > >> Hi, >> >> I have tried that as well, using -files <jar file> But it still gives the >> exact same error. Any other thing that I could try? >> >> Thanks, >> Krishna. >> >> On Jan 11, 2011, at 10:23 AM, Ted Yu wrote: >> >> > Refer to Alex Kozlov's answer on 12/11/10 >> > >> > On Tue, Jan 11, 2011 at 10:10 AM, C.V.Krishnakumar Iyer >> > <f2004...@gmail.com>wrote: >> > >> >> Hi, >> >> >> >> Could anyone please guide me as to how to use the -libjars option in >> HDFS? >> >> >> >> I have added the necessary jar file (the hbase jar - to be precise) to >> the >> >> classpath of the node where I am starting the job. >> >> >> >> The following is the format that i am invoking: >> >> bin/hadoop jar <Our Jar> <MainClass> -libjars <Dependent jars >> (separated by >> >> Commas)> <Arguments to our main class> >> >> >> >> bin/hadoop jar /Users/hdp/cvk/myjob.jar mr2.mr2a.MR2ADriver -libjars >> >> /Users/hdp/hadoop/lib/hbase-0.20.6.jar inputmr2a outputmr2a >> >> >> >> Despite this, I find that I get the java.lang.ClassNotFoundException >> >> error! :( >> >> java.lang.RuntimeException: java.lang.RuntimeException: >> >> java.lang.ClassNotFoundException: >> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable >> >> at >> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:841) >> >> at >> >> >> org.apache.hadoop.mapred.JobConf.getMapOutputValueClass(JobConf.java:551) >> >> at >> >> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.(MapTask.java:793) >> >> at >> >> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:524) >> >> at >> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:613) >> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) >> >> at org.apache.hadoop.mapred.Child.main(Child.java:170) >> >> Caused by: java.lang.RuntimeException: >> java.lang.ClassNotFoundException: >> >> org.apache.hadoop.hbase.io.ImmutableBytesWritable >> >> at >> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:809) >> >> at >> >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:833) >> >> >> >> The strange thing is that there is another MR job I have that runs >> >> perfectly with the libjars option! Could anybody tell me what I am >> doing >> >> wrong? One more thing - not sure if it is relevant : I am using the new >> >> Hadoop MapReduce API. >> >> >> >> Thanks in advance! >> >> >> >> Regards, >> >> Krishnakumar. >> >> >
