Hi, DB
i tried including breeze library by using spark 1.0, it works. but how can
i call
the native library in standalone cluster mode.
in local mode
1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in
sbt build file
2. i install openblas
it works
in standalone mode
1. i include "org.scalanlp" % "breeze-natives_2.10" % "0.7" dependency in
sbt build file
2. install openblas in workers
3. add sepatate jars using sc.addJar(). jars: breeze-natives_2.10-0.7.jar,
netlib-native_ref-linux-
x86_64-1.1-natives.jar,
netlib-native_system-linux-x86_64-1.1-natives.jar
4. i also include classpath of the above jars
but does not work:(
DB Tsai-2 wrote
> Hi Wxhsdp,
>
> I also have some difficulties witth "sc.addJar()". Since we include the
> breeze library by using Spark 1.0, we don't have the problem you ran into.
> However, when we add external jars via sc.addJar(), I found that the
> executors actually fetch the jars but the classloader still doesn't honor
> it. I'm trying to figure out the problem now.
>
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Wed, May 14, 2014 at 5:46 AM, wxhsdp <
> wxhsdp@
> > wrote:
>
>> Hi, DB
>> i've add breeze jars to workers using sc.addJar()
>> breeze jars include :
>> breeze-natives_2.10-0.7.jar
>> breeze-macros_2.10-0.3.jar
>> breeze-macros_2.10-0.3.1.jar
>> breeze_2.10-0.8-SNAPSHOT.jar
>> breeze_2.10-0.7.jar
>>
>> almost all the jars about breeze i can find, but still
>> NoSuchMethodError:
>> breeze.linalg.DenseMatrix
>>
>> from the executor stderr, you can see the executor successsully fetches
>> these jars, what's wrong
>> about my method? thank you!
>>
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar with
>> timestamp
>> 1400070957376
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze-natives_2.10-0.7.jar to
>> /tmp/fetchFileTemp7468892065227766972.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-natives_2.10-0.7.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar with timestamp
>> 1400070957441
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.jar to
>> /tmp/fetchFileTemp2324565598765584917.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar with
>> timestamp
>> 1400070957358
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.8-SNAPSHOT.jar to
>> /tmp/fetchFileTemp8730123100104850193.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.8-SNAPSHOT.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar with
>> timestamp
>> 1400070957414
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze-macros_2.10-0.3.1.jar to
>> /tmp/fetchFileTemp3473404556989515218.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze-macros_2.10-0.3.1.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar with timestamp
>> 1400070956753
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/build-project_2.10-1.0.jar to
>> /tmp/fetchFileTemp1289055585501269156.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./build-project_2.10-1.0.jar
>> to class loader
>> 14/05/14 20:36:02 INFO Executor: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar with timestamp
>> 1400070957228
>> 14/05/14 20:36:02 INFO Utils: Fetching
>> http://192.168.0.106:42883/jars/breeze_2.10-0.7.jar to
>> /tmp/fetchFileTemp1287317286108432726.tmp
>> 14/05/14 20:36:02 INFO Executor: Adding
>>
>> file:/home/wxhsdp/spark/spark/tags/v1.0.0-rc3/work/app-20140514203557-0000/0/./breeze_2.10-0.7.jar
>> to class loader
>>
>>
>> DB Tsai-2 wrote
>> > Since the breeze jar is brought into spark by mllib package, you may
>> want
>> > to add mllib as your dependency in spark 1.0. For bring it from your
>> > application yourself, you can either use sbt assembly in ur build
>> project
>> > to generate a flat myApp-assembly.jar which contains breeze jar, or use
>> > spark add jar api like Yadid said.
>> >
>> >
>> > Sincerely,
>> >
>> > DB Tsai
>> > -------------------------------------------------------
>> > My Blog: https://www.dbtsai.com
>> > LinkedIn: https://www.linkedin.com/in/dbtsai
>> >
>> >
>> > On Sun, May 4, 2014 at 10:24 PM, wxhsdp <
>>
>> > wxhsdp@
>>
>> > > wrote:
>> >
>> >> Hi, DB, i think it's something related to "sbt publishLocal"
>> >>
>> >> if i remove the breeze dependency in my sbt file, breeze can not be
>> found
>> >>
>> >> [error] /home/wxhsdp/spark/example/test/src/main/scala/test.scala:5:
>> not
>> >> found: object breeze
>> >> [error] import breeze.linalg._
>> >> [error] ^
>> >>
>> >> here's my sbt file:
>> >>
>> >> name := "Build Project"
>> >>
>> >> version := "1.0"
>> >>
>> >> scalaVersion := "2.10.4"
>> >>
>> >> libraryDependencies += "org.apache.spark" %% "spark-core" %
>> >> "1.0.0-SNAPSHOT"
>> >>
>> >> resolvers += "Akka Repository" at "http://repo.akka.io/releases/";
>> >>
>> >> i run "sbt publishLocal" on the Spark tree.
>> >>
>> >> but if i manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in
>> >> /lib
>> >> directory, sbt package is
>> >> ok, i can run my app in workers without addJar
>> >>
>> >> what's the difference between add dependency in sbt after "sbt
>> >> publishLocal"
>> >> and manully put spark-assembly-1.0.0-SNAPSHOT-hadoop1.0.4.jar in /lib
>> >> directory?
>> >>
>> >> why can i run my app in worker without addJar this time?
>> >>
>> >>
>> >> DB Tsai-2 wrote
>> >> > If you add the breeze dependency in your build.sbt project, it will
>> not
>> >> be
>> >> > available to all the workers.
>> >> >
>> >> > There are couple options, 1) use sbt assembly to package breeze into
>> >> your
>> >> > application jar. 2) manually copy breeze jar into all the nodes, and
>> >> have
>> >> > them in the classpath. 3) spark 1.0 has breeze jar in the spark flat
>> >> > assembly jar, so you don't need to add breeze dependency yourself.
>> >> >
>> >> >
>> >> > Sincerely,
>> >> >
>> >> > DB Tsai
>> >> > -------------------------------------------------------
>> >> > My Blog: https://www.dbtsai.com
>> >> > LinkedIn: https://www.linkedin.com/in/dbtsai
>> >> >
>> >> >
>> >> > On Sun, May 4, 2014 at 4:07 AM, wxhsdp <
>> >>
>> >> > wxhsdp@
>> >>
>> >> > > wrote:
>> >> >
>> >> >> Hi,
>> >> >> i'am trying to use breeze linalg library for matrix operation in
>> my
>> >> >> spark
>> >> >> code. i already add dependency
>> >> >> on breeze in my build.sbt, and package my code sucessfully.
>> >> >>
>> >> >> when i run on local mode, sbt "run local...", everything is ok
>> >> >>
>> >> >> but when turn to standalone mode, sbt "run spark://127.0.0.1:7077
>> >> ...",
>> >> >> error occurs
>> >> >>
>> >> >> 14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
>> >> >> java.lang.NoSuchMethodError
>> >> >> java.lang.NoSuchMethodError:
>> >> >>
>> >> >>
>> >>
>> breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
>> >> >>
>> >> >> in my opinion, everything needed is packaged to the jar file,
>> isn't
>> >> it?
>> >> >> and does anyone used breeze before? is it good for matrix
>> operation?
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> View this message in context:
>> >> >>
>> >>
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
>> >> >> Sent from the Apache Spark User List mailing list archive at
>> >> Nabble.com.
>> >> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5355.html
>> >> Sent from the Apache Spark User List mailing list archive at
>> Nabble.com.
>> >>
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5704.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310p5730.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
