The -libjars feature is needed if you use the classes in your remote code (map and reduce functions). Is it possible you only use it in your main() method?
-Joey Sent from my iPhone On Feb 1, 2012, at 3:38, Daniel Quach <danqu...@cs.ucla.edu> wrote: > Thanks, this seems to make it work. In fact, I did not even need to specify > lib jars in the command line…should I be worried that it doesn't work that > way? > > On Jan 31, 2012, at 4:09 PM, Joey Echeverria wrote: > >> You also need to add the jar to the classpath so it's available in >> your main. You can do soemthing like this: >> >> HADOOP_CLASSPATH=/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar >> hadoop jar ... >> >> -Joey >> >> On Tue, Jan 31, 2012 at 1:38 PM, Daniel Quach <danqu...@cs.ucla.edu> wrote: >>> For Hadoop 0.20.203 (the latest stable), is it sufficient to do this to >>> parse the lib jars from the command line? >>> >>> public static main (String args[]) { >>> Configuration conf = new Configuration(); >>> String[] otherArgs = new GenericOptionsParser(conf, >>> args).getRemainingArgs(); >>> >>> Job job = new Job(conf, "makevector"); >>> job.setJarByClass(MakeVector.class); >>> // etc other code for mappers/reducers >>> } >>> >>> I'm thinking I am missing a step here which is why it won't load the mahout >>> jar while running the map reduce. >>> >>> On Jan 30, 2012, at 11:10 PM, Daniel Quach wrote: >>> >>>> I compiled using javac: >>>> >>>> javac -classpath >>>> :/usr/local/hadoop/hadoop-core-0.20.203.0.jar:/usr/local/hadoop/lib/commons-cli-1.2.jar:/usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar >>>> -d makevector_classes/ MakeVector.java; >>>> >>>> If I don't include the mahout-math jar, it gives me a compile error >>>> because of DenseVector. >>>> >>>> >>>> On Jan 30, 2012, at 10:42 PM, Prashant Kommireddi wrote: >>>> >>>>> How are you building the mapreduce jar? Try not to include the Mahout dist >>>>> while building MR jar, and include it only on "-libjars" option. >>>>> >>>>> On Mon, Jan 30, 2012 at 10:33 PM, Daniel Quach <danqu...@cs.ucla.edu> >>>>> wrote: >>>>> >>>>>> I have been compiling my mapreduce with the jars in the classpath, and I >>>>>> believe I need to also add the jars as an option to -libjars to hadoop. >>>>>> However, even when I do this, I still get an error complaining about >>>>>> missing classes at runtime. (Compilation works fine). >>>>>> >>>>>> Here is my command: >>>>>> hadoop jar makevector.jar org.myorg.MakeVector -libjars >>>>>> /usr/local/mahout/math/target/mahout-math-0.6-SNAPSHOT.jar input/ output/ >>>>>> >>>>>> This is the error I receive: >>>>>> Exception in thread "main" java.lang.NoClassDefFoundError: >>>>>> org/apache/mahout/math/DenseVector >>>>>> >>>>>> I wonder if I am using the GenericOptionsParser incorrectly? I'm not sure >>>>>> if there is a deeper problem here. >>>>>> >>>> >>> >> >> >> >> -- >> Joseph Echeverria >> Cloudera, Inc. >> 443.305.9434 >