Thanks for the suggestions! I will give them a look and try to see it it will work!
Jonathan On Fri, Feb 21, 2014 at 6:35 PM, Azuryy Yu <azury...@gmail.com> wrote: > Hi, > > you cannot add jar like this way. > > please look at DistributeCache in the Hadoop Java Doc. > > please call DistributeCache.addArchive() in your main Class before submit > the MR job. > > > On Sat, Feb 22, 2014 at 9:30 AM, Gaurav Gupta <gau...@datatorrent.com>wrote: > >> Jonathan, >> >> >> >> You have to make sure that the jar is available on the nodes where the >> map reduce job is running. Setting the HADOOP_CLASSPATH on the single node >> doesn't work. >> >> You can use -libjars to the hadoop command line. >> >> >> >> Thanks >> >> Gaurav >> >> >> >> *From:* Jonathan Poon [mailto:jkp...@ucdavis.edu] >> *Sent:* Friday, February 21, 2014 5:12 PM >> *To:* user@hadoop.apache.org >> *Subject:* Having trouble adding external JAR to MapReduce Program >> >> >> >> Hi Everyone, >> >> I'm running into trouble adding the Avro JAR into my MapReduce program. >> I do the following to try to add the Avro JAR: >> >> export >> HADOOP_CLASSPATH="/tmp/singleEvent.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar:/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar:/usr/local/hadoop/hadoop-core-1.2.1.jar" >> >> export >> LIBJARS="/tmp/singleEvent.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-mapred-1.7.6-hadoop1.jar,/home/jonathanpoon/local/lib/java/avro-1.7.6/avro-tools-1.7.6.jar,/usr/local/hadoop/hadoop-core-1.2.1.jar" >> >> hadoop jar AvroReader.jar org.avro.AvroReader -libjars ${LIBJARS} >> /user/jonathanpoon/avro /user/jonathanpoon/output >> >> However, I get the following error: >> >> 14/02/21 17:01:17 INFO mapred.JobClient: Task Id : >> attempt_201402191318_0014_m_000001_2, Status : FAILED >> java.lang.RuntimeException: java.lang.ClassNotFoundException: >> org.apache.avro.mapreduce.AvroKeyInputFormat >> at >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:857) >> at >> org.apache.hadoop.mapreduce.JobContext.getInputFormatClass(JobContext.java:187) >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:722) >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364) >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:415) >> at >> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) >> at org.apache.hadoop.mapred.Child.main(Child.java:249) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.avro.mapreduce.AvroKeyInputFormat >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at java.lang.Class.forName0(Native Method) >> at java.lang.Class.forName(Class.java:270) >> at >> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810) >> at >> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855) >> ... 8 more >> >> >> Am I placing the Avro JAR files in the improper place? >> >> Thanks for your help! >> >> Jonathan >> >> >> > >
