Hi, DB i tried including breeze library by using spark 1.0, it works. but how can i call the native library in standalone cluster mode.
in local mode 1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in sbt build file 2. i install openblas it works in standalone mode 1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in sbt build file 2. install openblas in workers 3. add sepatate jars using sc.addJar(). jars: breeze-natives_2.10-0.7.jar, netlib-native_ref-linux- x86_64-1.1-natives.jar, netlib-native_system-linux-x86_64-1.1-natives.jar 4. i also include classpath of the above jars but does not work:( DB Tsai-2 wrote > Hi Wxhsdp, > > I also have some difficulties witth "sc.addJar()". Since we include the > breeze library by using Spark 1.0, we don't have the problem you ran into. > However, when we add external jars via sc.addJar(), I found that the > executors actually fetch the jars but the classloader still doesn't honor > it. I'm trying to figure out the problem now. > > > Sincerely, > > DB Tsai > ------------------------------------------------------- > My Blog: https://www.dbtsai.com > LinkedIn: https://www.linkedin.com/in/dbtsai > > > On Wed, May 14, 2014 at 5:46 AM, wxhsdp < > wxhsdp@ > > wrote: > >> Hi, DB >> i've add breeze jars to workers using sc.addJar() >> breeze jars include : >> breeze-natives_2.10-0.7.jar >> breeze-macros_2.10-0.3.jar >> breeze-macros_2.10-0.3.1.jar >> breeze_2.10-0.8-SNAPSHOT.jar >> breeze_2.10-0.7.jar >> >> almost all the jars about breeze i can find, but still >> NoSuchMethodError: >> breeze.linalg.DenseMatrix >> >> from the executor stderr, you can see the executor successsully fetches >> these jars, what's wrong >> about my method? thank you! >> >> 14/05/14 20:36:02 INFO Executor: Fetching >> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with >> timestamp >> 1400070957376 >> 14/05/14 20:36:02 INFO Utils: Fetching >> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to >> /tmp/fetchFileTemp7468892065227766972.tmp >> 14/05/14 20:36:02 INFO Executor: Adding >> >> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar >> to class loader >> 14/05/14 20:36:02 INFO Executor: Fetching >> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp >> 1400070957441 >> 14/05/14 20:36:02 INFO Utils: Fetching >> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to >> /tmp/fetchFileTemp2324565598765584917.tmp >> 14/05/14 20:36:02 INFO Executor: Adding >> >> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar >> to class loader >> 14/05/14 20:36:02 INFO Executor: Fetching >> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with >> timestamp >> 1400070957358 >> 14/05/14 20:36:02 INFO Utils: Fetching >> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to >> /tmp/fetchFileTemp8730123100104850193.tmp >> 14/05/14 20:36:02 INFO Executor: Adding >> >> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar >> to class loader >> 14/05/14 20:36:02 INFO Executor: Fetching >> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with >> timestamp >> 1400070957414 >> 14/05/14 20:36:02 INFO Utils: Fetching >> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to >> /tmp/fetchFileTemp3473404556989515218.tmp >> 14/05/14 20:36:02 INFO Executor: Adding >> >> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar >> to class loader >> 14/05/14 20:36:02 INFO Executor: Fetching >> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp >> 1400070956753 >> 14/05/14 20:36:02 INFO Utils: Fetching >> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to >> /tmp/fetchFileTemp1289055585501269156.tmp >> 14/05/14 20:36:02 INFO Executor: Adding >> >> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar >> to class loader >> 14/05/14 20:36:02 INFO Executor: Fetching >> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp >> 1400070957228 >> 14/05/14 20:36:02 INFO Utils: Fetching >> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to >> /tmp/fetchFileTemp1287317286108432726.tmp >> 14/05/14 20:36:02 INFO Executor: Adding >> >> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar >> to class loader >> >> >> DB Tsai-2 wrote >> > Since the breeze jar is brought into spark by mllib package, you may >> want >> > to add mllib as your dependency in spark 1.0. For bring it from your >> > application yourself, you can either use sbt assembly in ur build >> project >> > to generate a flat myApp-assembly.jar which contains breeze jar, or use >> > spark add jar api like Yadid said. >> > >> > >> > Sincerely, >> > >> > DB Tsai >> > ------------------------------------------------------- >> > My Blog: https://www.dbtsai.com >> > LinkedIn: https://www.linkedin.com/in/dbtsai >> > >> > >> > On Sun, May 4, 2014 at 10:24 PM, wxhsdp < >> >> > wxhsdp@ >> >> > > wrote: >> > >> >> Hi, DB, i think it's something related to "sbt publishLocal" >> >> >> >> if i remove the breeze dependency in my sbt file, breeze can not be >> found >> >> >> >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5: >> not >> >> found: object breeze >> >> [error] import breeze.linalg._ >> >> [error] ^ >> >> >> >> here's my sbt file: >> >> >> >> name := "Build Project" >> >> >> >> version := "1.0" >> >> >> >> scalaVersion := "2.10.4" >> >> >> >> libraryDependencies += "org.apache.spark" %% "spark-core" % >> >> "1.0.0-SNAPSHOT" >> >> >> >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/" >> >> >> >> i run "sbt publishLocal" on the Spark tree. >> >> >> >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in >> >> /lib >> >> directory, sbt package is >> >> ok, i can run my app in workers without addJar >> >> >> >> what's the difference between add dependency in sbt after "sbt >> >> publishLocal" >> >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib >> >> directory? >> >> >> >> why can i run my app in worker without addJar this time? >> >> >> >> >> >> DB Tsai-2 wrote >> >> > If you add the breeze dependency in your build.sbt project, it will >> not >> >> be >> >> > available to all the workers. >> >> > >> >> > There are couple options, 1) use sbt assembly to package breeze into >> >> your >> >> > application jar. 2) manually copy breeze jar into all the nodes, and >> >> have >> >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat >> >> > assembly jar, so you don't need to add breeze dependency yourself. >> >> > >> >> > >> >> > Sincerely, >> >> > >> >> > DB Tsai >> >> > ------------------------------------------------------- >> >> > My Blog: https://www.dbtsai.com >> >> > LinkedIn: https://www.linkedin.com/in/dbtsai >> >> > >> >> > >> >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp < >> >> >> >> > wxhsdp@ >> >> >> >> > > wrote: >> >> > >> >> >> Hi, >> >> >> i'am trying to use breeze linalg library for matrix operation in >> my >> >> >> spark >> >> >> code. i already add dependency >> >> >> on breeze in my build.sbt, and package my code sucessfully. >> >> >> >> >> >> when i run on local mode, sbt "run local...", everything is ok >> >> >> >> >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077 >> >> ...", >> >> >> error occurs >> >> >> >> >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to >> >> >> java.lang.NoSuchMethodError >> >> >> java.lang.NoSuchMethodError: >> >> >> >> >> >> >> >> >> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$; >> >> >> >> >> >> in my opinion, everything needed is packaged to the jar file, >> isn't >> >> it? >> >> >> and does anyone used breeze before? is it good for matrix >> operation? >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> View this message in context: >> >> >> >> >> >> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html >> >> >> Sent from the Apache Spark User List mailing list archive at >> >> Nabble.com. >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html >> >> Sent from the Apache Spark User List mailing list archive at >> Nabble.com. >> >> >> >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5730.html Sent from the Apache Spark User List mailing list archive at Nabble.com.