An additional option 4) Use SparkContext.addJar() and have the
application ship your jar to all the nodes.
Yadid
On 5/4/14, 4:07 PM, DB Tsai wrote:
If you add the breeze dependency in your build.sbt project, it will
not be available to all the workers.
There are couple options, 1) use sbt assembly to package breeze into
your application jar. 2) manually copy breeze jar into all the nodes,
and have them in the classpath. 3) spark 1.0 has breeze jar in the
spark flat assembly jar, so you don't need to add breeze dependency
yourself.
Sincerely,
DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai
On Sun, May 4, 2014 at 4:07 AM, wxhsdp <wxh...@gmail.com
<mailto:wxh...@gmail.com>> wrote:
Hi,
i'am trying to use breeze linalg library for matrix operation in
my spark
code. i already add dependency
on breeze in my build.sbt, and package my code sucessfully.
when i run on local mode, sbt "run local...", everything is ok
but when turn to standalone mode, sbt "run
spark://127.0.0.1:7077...",
error occurs
14/05/04 18:56:29 WARN scheduler.TaskSetManager: Loss was due to
java.lang.NoSuchMethodError
java.lang.NoSuchMethodError:
breeze.linalg.DenseMatrix$.implOpMulMatrix_DMD_DMD_eq_DMD()Lbreeze/linalg/operators/DenseMatrixMultiplyStuff$implOpMulMatrix_DMD_DMD_eq_DMD$;
in my opinion, everything needed is packaged to the jar file,
isn't it?
and does anyone used breeze before? is it good for matrix operation?
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/NoSuchMethodError-breeze-linalg-DenseMatrix-tp5310.html
Sent from the Apache Spark User List mailing list archive at
Nabble.com.
