Hi, DB i've add breeze jars to workers using sc.addJar() breeze jars include : breeze-natives_2.10-0.7.jar breeze-macros_2.10-0.3.jar breeze-macros_2.10-0.3.1.jar breeze_2.10-0.8-SNAPSHOT.jar breeze_2.10-0.7.jar
almost all the jars about breeze i can find, but still NoSuchMethodError: breeze.linalg.DenseMatrix from the executor stderr, you can see the executor successsully fetches these jars, what's wrong about my method? thank you! 14/05/14 20:36:02 INFO Executor: Fetching http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with timestamp 1400070957376 14/05/14 20:36:02 INFO Utils: Fetching http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to /tmp/fetchFileTemp7468892065227766972.tmp 14/05/14 20:36:02 INFO Executor: Adding file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar to class loader 14/05/14 20:36:02 INFO Executor: Fetching http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp 1400070957441 14/05/14 20:36:02 INFO Utils: Fetching http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to /tmp/fetchFileTemp2324565598765584917.tmp 14/05/14 20:36:02 INFO Executor: Adding file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar to class loader 14/05/14 20:36:02 INFO Executor: Fetching http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with timestamp 1400070957358 14/05/14 20:36:02 INFO Utils: Fetching http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to /tmp/fetchFileTemp8730123100104850193.tmp 14/05/14 20:36:02 INFO Executor: Adding file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar to class loader 14/05/14 20:36:02 INFO Executor: Fetching http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with timestamp 1400070957414 14/05/14 20:36:02 INFO Utils: Fetching http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to /tmp/fetchFileTemp3473404556989515218.tmp 14/05/14 20:36:02 INFO Executor: Adding file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar to class loader 14/05/14 20:36:02 INFO Executor: Fetching http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp 1400070956753 14/05/14 20:36:02 INFO Utils: Fetching http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to /tmp/fetchFileTemp1289055585501269156.tmp 14/05/14 20:36:02 INFO Executor: Adding file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar to class loader 14/05/14 20:36:02 INFO Executor: Fetching http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp 1400070957228 14/05/14 20:36:02 INFO Utils: Fetching http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to /tmp/fetchFileTemp1287317286108432726.tmp 14/05/14 20:36:02 INFO Executor: Adding file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar to class loader DB Tsai-2 wrote > Since the breeze jar is brought into spark by mllib package, you may want > to add mllib as your dependency in spark 1.0. For bring it from your > application yourself, you can either use sbt assembly in ur build project > to generate a flat myApp-assembly.jar which contains breeze jar, or use > spark add jar api like Yadid said. > > > Sincerely, > > DB Tsai > ------------------------------------------------------- > My Blog: https://www.dbtsai.com > LinkedIn: https://www.linkedin.com/in/dbtsai > > > On Sun, May 4, 2014 at 10:24 PM, wxhsdp < > wxhsdp@ > > wrote: > >> Hi, DB, i think it's something related to "sbt publishLocal" >> >> if i remove the breeze dependency in my sbt file, breeze can not be found >> >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: not >> found: object breeze >> [error] import breeze.linalg._ >> [error] ^ >> >> here's my sbt file: >> >> name := "Build Project" >> >> version := "1.0" >> >> scalaVersion := "2.10.4" >> >> libraryDependencies += "org.apache.spark" %% "spark-core" % >> "1.0.0-SNAPSHOT" >> >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/"; >> >> i run "sbt publishLocal" on the Spark tree. >> >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in >> /lib >> directory, sbt package is >> ok, i can run my app in workers without addJar >> >> what's the difference between add dependency in sbt after "sbt >> publishLocal" >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib >> directory? >> >> why can i run my app in worker without addJar this time? >> >> >> DB Tsai-2 wrote >> > If you add the breeze dependency in your build.sbt project, it will not >> be >> > available to all the workers. >> > >> > There are couple options, 1) use sbt assembly to package breeze into >> your >> > application jar. 2) manually copy breeze jar into all the nodes, and >> have >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat >> > assembly jar, so you don't need to add breeze dependency yourself. >> > >> > >> > Sincerely, >> > >> > DB Tsai >> > ------------------------------------------------------- >> > My Blog: https://www.dbtsai.com >> > LinkedIn: https://www.linkedin.com/in/dbtsai >> > >> > >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp < >> >> > wxhsdp@ >> >> > > wrote: >> > >> >> Hi, >> >> i'am trying to use breeze linalg library for matrix operation in my >> >> spark >> >> code. i already add dependency >> >> on breeze in my build.sbt, and package my code sucessfully. >> >> >> >> when i run on local mode, sbt "run local...", everything is ok >> >> >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077 >> ...", >> >> error occurs >> >> >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to >> >> java.lang.NoSuchMethodError >> >> java.lang.NoSuchMethodError: >> >> >> >> >> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$; >> >> >> >> in my opinion, everything needed is packaged to the jar file, isn't >> it? >> >> and does anyone used breeze before? is it good for matrix operation? >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html >> >> Sent from the Apache Spark User List mailing list archive at >> Nabble.com. >> >> >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
