If I submit a jar that has a lib directory that contains a bunch of
jars, shouldn't those jars be in the classpath and available to all nodes?
The reason I ask this is because I am trying to submit a jar myjar.jar
that has the following structure
--src
\.... (My source classes)
-- lib
\
-- mahout-collections-0.3.jar
-- mahout-core-0.3.jar
-- mahout-math-0.3.jar
-- hbase-0.20.0.jar
-- commons-cli-2.0-mahout.jar
Now the job I am trying to run is actually part of mahout-core-0.3...
not src. The job starts but then fails with the following error
10/09/10 03:15:13 INFO mapred.JobClient: Task Id :
attempt_201009100306_0003_r_000000_0, Status : FAILED
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:115)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:550)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:113)
... 3 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/mahout/math/map/OpenObjectIntHashMap
at
org.apache.mahout.fpm.pfpgrowth.ParallelFPGrowthReducer.<init>(ParallelFPGrowthReducer.java:56)
... 8 more
Apparently it cant find the
org/apache/mahout/math/map/OpenObjectIntHashMap.class although that
class is definitely in mahout-collections-0.3.jar. Is this a problem
because the mahout-core-0.3.jar doesn't have a lib directory?
What is an easy way around this?
Thanks
