Trying to upgrade from Spark 1.0.1 to 1.1.0. Can’t imagine the upgrade is the problem but anyway...
I get a NoClassDefFoundError for RandomGenerator when running a driver from the CLI. But only when using a named master, even a standalone master. If I run using master = local[4] the job executes correctly but if I set the master to spark://Maclaurin.local:7077 though they are the same machine I get the NoClassDefFoundError. The classpath seems correct on the CLI and the jars do indeed contain the offending class (see below). There must be some difference in how classes are loaded between local[4] and spark://Maclaurin.local:7077? Any ideas? =============== The driver is in mahout-spark_2.10-1.0-SNAPSHOT-job.jar so it’s execution means it must be in the classpath. When I look at what’s in the jar I see RandomGenerator. Maclaurin:target pat$ jar tf mahout-spark_2.10-1.0-SNAPSHOT-job.jar | grep RandomGenerator cern/jet/random/engine/RandomGenerator.class org/apache/commons/math3/random/GaussianRandomGenerator.class org/apache/commons/math3/random/JDKRandomGenerator.class org/apache/commons/math3/random/UniformRandomGenerator.class org/apache/commons/math3/random/RandomGenerator.class <==========! org/apache/commons/math3/random/NormalizedRandomGenerator.class org/apache/commons/math3/random/AbstractRandomGenerator.class org/apache/commons/math3/random/StableRandomGenerator.class But get the following error executing the job: 14/10/19 15:39:00 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 6.9 (TID 84, 192.168.0.2): java.lang.NoClassDefFoundError: org/apache/commons/math3/random/RandomGenerator org.apache.mahout.common.RandomUtils.getRandom(RandomUtils.java:65) org.apache.mahout.math.cf.SimilarityAnalysis$$anonfun$5.apply(SimilarityAnalysis.scala:272) org.apache.mahout.math.cf.SimilarityAnalysis$$anonfun$5.apply(SimilarityAnalysis.scala:267) org.apache.mahout.sparkbindings.blas.MapBlock$$anonfun$1.apply(MapBlock.scala:33) org.apache.mahout.sparkbindings.blas.MapBlock$$anonfun$1.apply(MapBlock.scala:32) scala.collection.Iterator$$anon$11.next(Iterator.scala:328) scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:371) org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:235) org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:163) org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:70) org.apache.spark.rdd.RDD.iterator(RDD.scala:227) org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31) org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) org.apache.spark.rdd.RDD.iterator(RDD.scala:229) org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35) org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262) org.apache.spark.rdd.RDD.iterator(RDD.scala:229) org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:62) org.apache.spark.scheduler.Task.run(Task.scala:54) org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) java.lang.Thread.run(Thread.java:695) --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org