RE: Having trouble adding external JAR to MapReduce Program

[Gaurav Gupta]

Jonathan,

 

You have to make sure that the jar is available on the nodes where the map
reduce job is running. Setting the HADOOP_CLASSPATH on the single node
doesn't work.

You can use -libjars to the hadoop command line.

 

Thanks

Gaurav

 

From: Jonathan Poon [mailto:jkp...@ucdavis.edu] 
Sent: Friday, February 21, 2014 5:12 PM
To: user@hadoop.apache.org
Subject: Having trouble adding external JAR to MapReduce Program

 

Hi Everyone,

I'm running into trouble adding the Avro JAR into my MapReduce program.  I
do the following to try to add the Avro JAR:

export
HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avr
o-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro
-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar"

export
LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/a
vro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/av
ro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar"

hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS}
/user/jonathanpoon/avro /user/jonathanpoon/output

However, I get the following error:

14/02/21 17:01:17 INFO mapred.JobClient: Task Id :
attempt_201402191318_0014_m_000001_2, Status : FAILED
java.lang.RuntimeException: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857)
        at
org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:1
87)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
        at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.mapreduce.AvroKeyInputFormat
        at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
        at
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
        ... 8 more




Am I placing the Avro JAR files in the improper place?  

Thanks for your help!

Jonathan