I'm trying to submit a simple test code through spark-submit. first portion
of the code works fine, but some calls to breeze vector library fails:

14/10/22 17:36:02 INFO CacheManager: Partition rdd_1_0 not found, computing
it
14/10/22 17:36:02 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.NoSuchMethodError:
breeze.linalg.SparseVector$mcD$sp.<init>([I[DILbreeze/storage/DefaultArrayValue;)V
at spark.kmeans.SparseVector.toBreeze(Vectors.scala:182)
at spark.kmeans.MyTest$$anonfun$1.apply(MyTest.scala:56)
at spark.kmeans.MyTest$$anonfun$1.apply(MyTest.scala:56)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:236)


I ran this code with
~/tools/spark-1.1.0-bin-hadoop2.4/bin/spark-submit  --class
spark.kmeans.MyTest --master local target/reco-v1-jar-with-dependencies.jar



my system scala version is 2.10.4

the relevant sections in my pom is: (I did not build with sbt, since it
gave me some problems)
                <dependency>
                        <groupId>org.apache.spark</groupId>
                        <artifactId>spark-core_2.10</artifactId>
                        <version>1.0.2</version>
                        <exclusions>
                                <exclusion>
                                        <groupId>org.apache.hadoop</groupId>

<artifactId>hadoop-client</artifactId>
                                </exclusion>
                        </exclusions>
                </dependency>

                <dependency>
                    <groupId>org.apache.spark</groupId>
                    <artifactId>spark-mllib_2.10</artifactId>
                    <version>1.0.2</version>
                </dependency>

             <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>2.10.4</version>
        </dependency>



Thanks a lot
Yang

Reply via email to